EURON SiG workshop at ICRA

The 26th flagship conference of the IEEE Robotics and Automation Society (RAS)—the ICRA 2010—received over 2000 submissions from 47 countries this year. The number of conference attendees surpassed every expectation, as more than 1300 people wanted to join the scientific discussions and learn from the lectures. The 150 technical sessions just could not fit all the topics, therefore quite a large number (over 40) tutorials and workshops took place right before and after the conference. A full-day event was the latest meeting of the European Robotics Research Network (EURON) Special Interest Group on Good Experimental Methodology for Robotics (SIG GEM).
The Workshop on The Role of Experiments in Robotics Research included nine 30-minute-long talks, each addressing a major area of robotics from aerial swarms to surgical manipulators and algorithm benchmarking. While certain speakers were more focusing on the key aspects of their field of specialty, others preferred to present their views through case studies. I was happy to deliver a talk on the necessary need of accuracy standards in CIS.
"It is crucial to well define the experiments evaluating the usability of a system. There are three different types of accuracies that can be specified for CIS systems:

• Intrinsic (technical) accuracy (0.1–0.6 mm)
• Registration accuracy (0.2–3 mm)
• Application accuracy (0.6–10 mm).

Intrinsic accuracy applies to certain elements, such as the robot or the localizer. It describes the average error of the given component in operational use. Random errors (e.g., mechanical compliance, friction, loose hardware), resolution of the imaging device, inadequate control and noise can all result in low intrinsic accuracy. On the user interface side, discretized input and modeling errors may further decrease precision.
All registration methods involve some kind of errors, as it is only possible to compute a normalized (e.g., least squares) solution for a mathematical fitting problem. In IGS, a major source of error can be the markers (different types, forms and materials), displacement of the fiducials and determination of the center of the fiducials. Even though the distribution of errors for one fiducial in a particular set of measurements is usually Gaussian, the aggregated error for many fiducials is chi square at best estimate.
Application accuracy refers to the overall targeting error of the integrated system while used in a clinical procedure or a mock setup. This measures most realistically the effectiveness of a system and is commonly used for validation. The application accuracy is dependent on all other sources of errors in a complex, non-linear way, therefore typically phantom, cadaver and clinical trials are required to determine it.
International bodies are making efforts to standardize CIS procedures similarly to industrial robotics, but currently, there are no accepted regulations. The ISO 10360 standard “Acceptance and reverification tests for coordinate measuring machines (CMMs)” from 1994 (current version: 2001) can be applied in many cases.
In 2004, the American Society for Testing and Materials (ASTM) initiated a new standards committee (ASTM F04.05) under the title “Standard Practice for Measurement of Positional Accuracy of Computer Assisted Surgical Systems (CAOS)“. The goal was to develop an international standard for metrology, validation and performance of CAOS systems.
Supporting the ASTM group, a multi-institution technical committee presented a white paper more recently, calling for the standardization of many areas of CIS. Based on technological and economical analysis, metrology and standards should be applied especially to the following categories of medical devices:

• Computer assisted navigation and surgery
• Surgical robots (mostly in manual control mode)
• Surgical robots and phantom (artifact) devices
• Stimulation devices
• Drug-delivery and physiologic monitoring devices."
  

Read and see more: access the original presentation.

Note: the picture was taken during the IEEE GOLD lunch.