Object manipulation with a variable-stiffness robotic mechanism using deep neural networks for visual semantics and load estimation
Artikel i vetenskaplig tidskrift, 2020

In recent years, the computer vision applications in the robotics have been improved to approach human-like visual
perception and scene/context understanding. Following this aspiration, in this study, we explored the possibility of better
object manipulation performance by connecting the visual recognition of objects to their physical attributes, such as weight
and center of gravity (CoG). To develop and test this idea, an object manipulation platform is built comprising a robotic
arm, a depth camera fixed at the top center of the workspace, embedded encoders in the robotic arm mechanism, and
microcontrollers for position and force control. Since both the visual recognition and force estimation algorithms use deep
learning principles, the test set-up was named as Deep-Table. The objects in the manipulation tests are selected from
everyday life and are common to be seen on modern office desktops. The visual object localization and recognition
processes are performed from two distinct branches by deep convolutional neural network architectures. We present five of
the possible cases, having different levels of information availability on the object weight and CoG in the experiments. The
results confirm that using our algorithm, the robotic arm can move different types of objects successfully varying from
several grams (empty bottle) to around 250 g (ceramic cup) without failure or tipping. The proposed method also shows
that connecting the object recognition with load estimation and contact point further improves the performance characterized
by a smoother motion.

Robotic Manipulation

Object Recognition

Deep Neural Networks

Context Awareness

Force Estimation


Ertugrul Bayraktar

Istanbul Teknik Universitesi (ITÜ)

Duzce University

Cihat Bora Yigit

Istanbul Teknik Universitesi (ITÜ)

Siemens AS

Pinar Boyraz Baykas

Istanbul Teknik Universitesi (ITÜ)

Chalmers, Mekanik och maritima vetenskaper, Fordonssäkerhet, Olycksanalys och prevention

Neural Computing and Applications

0941-0643 (ISSN) 1433-3058 (eISSN)

Vol. 32 13 9029-9045



Annan maskinteknik

Medicinsk apparatteknik

Robotteknik och automation


Informations- och kommunikationsteknik


Livsvetenskaper och teknik (2010-2018)


Innovation och entreprenörskap



Mer information

Senast uppdaterat