A hull reconstruction-reprojection method for pose estimation of free-flying fruit flies

Journal article, 2023

Understanding the mechanisms of insect flight requires high quality data of free-flight kinematics, e.g., for comparative studies or genetic screens. While recent improvements in high-speed videography allow us to acquire large amounts of free-flight data, a significant bottleneck is automatically extracting accurate body and wing kinematics. Here, we present an experimental system and a hull reconstruction-reprojection algorithm for measuring the flight kinematics of fruit flies. The experimental system can automatically record hundreds of flight events per day. Our algorithm resolves a significant portion of the occlusions in this system by a reconstruction-reprojection scheme that integrates information from all cameras. Wing and body kinematics, including wing deformation, are then extracted from the hulls of the wing boundaries and body. This model-free method is fully automatic, accurate and open-source, and can be readily adjusted for different camera configurations or insect species.