METRICS concept and methodology
METRICS aims at establishing a metrology-grade methodological framework for the evaluation of robotic systems in the four priority areas.
The consortium defines:
- An evaluation plan template for the METRICS competitions (including a methodology for the evaluation protocol, the evaluation tasks, the metrics used, the test facilities, test databases, participation rules and operating procedures) capitalising on many campaigns organised by METRICS partners in the past;
- A good practice guide for the organisation of evaluation campaigns, including the organisational process based on a dry-run followed by official evaluation campaigns, instructions for the fair treatment of participants, communication and awareness-raising actions to be carried out;
- Ethical guidelines for the definition of the evaluation plan and the implementation of the competition.
METRICS common framework focuses in particular on providing methods to maximise the reproducibility of evaluations, particularly in terms of identifying and controlling influencing factors, measuring the environmental conditions (brightness, temperature, wind, ground conditions, etc.) and describing the test environments (objects to be recognised and handled, furniture and wall layout, test database, etc.).