Systematic benchmarking for reproducibility of computer vision algorithms for real-time systems: The example of optic flow estimation
Paper i proceeding, 2019

Until now there have been few formalized methods for conducting systematic benchmarking aiming at reproducible results when it comes to computer vision algorithms. This is evident from lists of algorithms submitted to prominent datasets, authors of a novel method in many cases primarily state the performance of their algorithms in relation to a shallow description of the hardware system where it was evaluated. There are significant problems linked to this non-systematic approach of reporting performance, especially when comparing different approaches and when it comes to the reproducibility of claimed results. Furthermore how to conduct retrospective performance analysis such as an algorithm's suitability for embedded real-time systems over time with underlying hardware and software changes in place. This paper proposes and demonstrates a systematic way of addressing such challenges by adopting containerization of software aiming at formalization and reproducibility of benchmarks. Our results show maintainers of broadly accepted datasets in the computer vision community to strive for systematic comparison and reproducibility of submissions to increase the value and adoption of computer vision algorithms in the future.

benchmarking

performance

computer vision

optic flow

virtualization

run-time

reproducibility

containerization

embedded systems

real-time computing

Författare

Björnborg Nguyen

Chalmers, Mekanik och maritima vetenskaper, Fordonsteknik och autonoma system

Christian Berger

Chalmers, Data- och informationsteknik, Software Engineering, Software Engineering for People, Architecture, Requirements and Traceability

Ola Benderius

Chalmers, Mekanik och maritima vetenskaper, Fordonsteknik och autonoma system

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

2153-0858 (ISSN) 2153-0866 (eISSN)

5264-5269-5269

International Conference on Intelligent Robots and Systems (IROS)
Macau, China,

Ämneskategorier

Datorteknik

Programvaruteknik

Datavetenskap (datalogi)

Datorsystem

Datorseende och robotik (autonoma system)

Styrkeområden

Informations- och kommunikationsteknik

Drivkrafter

Innovation och entreprenörskap

Infrastruktur

ReVeRe (Research Vehicle Resource)

DOI

10.1109/IROS40897.2019.8968066

Mer information

Senast uppdaterat

2020-02-17