Robotics competitions trends and needs

For more than twenty years, competitions have been pushing forward research and development in the robotic community:

  • Fostered by the development of Information and Communication Technologies (ICT), the evolution of AI and the democratisation of robotics components, competitions gather each year hundreds of participants and sponsors, and millions of spectators;
  • Competitions are mainly organised by networks of expert roboticists (industrials, academics) who set the objectives of the contest and coordinate the definition of the rules and the evaluation process;
  • Several competitions are explicitly dedicated to scientific and technological emulation and public dissemination of robotics, and rely on the organisation of worldwide events (e.g. RoboCup), while others are dedicated to a more specialised audience of researchers and industrials in robotics (e.g. the DARPA Robotics Challenge);
  • Competitions are encouraged to be more than “one-off demonstrations”1 and adopt the form of repeatable and reproducible scientific experiments so as to assess their real benefit on robotics development and control (e.g. the European Commission (EC) recommends that autonomous systems be subject to rigorous, quantified and objective scrutiny): indeed, several European projects (RoCKIn, euRathlon, RockEU2, SciRoc) pursued this approach2, currently embodied by the ERL competition;
  • Recently, reports issued by public and independent scientific committees (MAR3, Delvaux report4, EGE5, AI HLEG6, etc.) stressed the need to urgently structure and mobilise the robotics and AI community;
  • The reliability and validity of AI algorithms are at the heart of European concerns (2019 Council of Europe conference7 and reports cited above). This requires that competitions address both the behaviour of the robot in a physical environment, and the behaviour of its AI algorithms when confronted with sets of properly qualified and controlled data. To our knowledge, this has not been addressed in previous European robotics competitions.

In order to assess the reliability of the different competing robots in a rigorous and unbiased way, European robotics competitions must rely on an evaluation framework that applies metrological practices for intelligent systems.