The competition is open to everyone all around the world and is led by CATEC (Advanced Center for Aerospace Technologies, Seville, Spain), one of the reference research centres in Europe devoted to aerial robotic technologies.

In this edition, the competition will take place in two subsequent stages. First, the cascade evaluation campaign will run from January to April 2023, where teams will develop and test their algorithms on datasets provided by CATEC. Then, the teams with the best performances will be allowed to participate in the field evaluation campaign, which will take place at the ICUAS conference venue in June 2023 in Warsaw, Poland.

It is important to remind that teams must first participate in the cascade evaluation campaign, and register through the ICUAS’23 UAV Competition website.

CASCADE EVALUATION CAMPAIGN

The cascade competition will be evaluated fully virtually using data generated by the competition organizers in previous RAMI campaigns.

There will be two different challenges (Functionality Benchmarks or FBM):

• FBM1: precise navigation without GNSS, since I&M activities may take place in environments with poor GNSS coverage, or even indoors.
• FBM2: automatic detection of defects using advanced AI algorithms, which is important for inspectors when they face considerable amounts of data to review.

FBM1

The execution of this FBM consists in assessing the accuracy of a localization system for the autonomous navigation of aerial robots using only onboard sensors. The evaluation will be based on the comparison of the team’s localization solution with respect to a precise motion capture system. Teams will be provided with several sample flight datasets for testing their solutions, with the aerial robot performing specific trajectories. The scoring will be based on the Root Mean Squared Error (RMSE) of the provided trajectory with respect to the ground truth trajectory that the aerial robot followed during an evaluation dataset that was not previously released to the teams. In order to obtain this metric, a .txt file with the estimation of the pose at minimum 1 Hz (at least 10 Hz recommended) must be provided by the solution proposed by teams, containing one line by pose. The poses will be defined by the timestamp, translation (x, y, z) and the orientation in quaternion (x, y, z, w) as the example below:

FBM2

The execution of this FBM consists of assessing the performance of a surface defect detection system. Teams will be provided with a publicly available dataset for training their algorithms. The assessment will be based on an offline analysis of images obtained by the aerial robot, which will show several surface defects artificially placed along with the testing scenario. Critical Success Index or Threat Score metric will be used to assess the performance of each team, comparing the results with ground truth labels. A new .txt file must be provided containing one line per detection performed, containing the image name and detection coordinates in image plane as in the following example of a single detection:

To ensure equal conditions for every team, a Docker image will be provided with the datasets, a detailed description of how the results must be delivered, and an example of participation for each FBM. This participation example will contain a sample solution, its mandatory running script, and a sample result file.

Each team must install their own dependencies, their solutions with every single file needed to run them, and it must output the results into the indicated path with the format mentioned above. Then, this Docker images must be uploaded to run the competitor's solution with the new final evaluation data.

For more information about the evaluation, please refer to RAMI's Evaluation Plan (section 4.2). Regarding the how to provide the data and the general use of the Docker image, please refer the corresponding documents in the “More Information” box.

IMPORTANT NOTE FOR CASCADE COMPETITION

Since this is a joint event with ICUAS’23 UAV Competition, the team performance and hence their access to the field evaluation campaign is not only measured from the results of the cascade evaluation campaign. There is also an exploration benchmark based on a simulation environment (Gazebo) which is not fully part of RAMI. Teams are strongly encouraged to access the ICUAS’23 UAV Competition website to check additional information. RAMI’s FBMs correspond to benchmarks 2 and 3 in the ICUAS competition.