Théo Morales

Title: Hand-object manipulation tracking using computer vision

Supervision Team: Gerard Lacey, TCD / Alistair Sutherland, DCU

Description: Current hand tracking for VR/AR interfaces focuses on the manipulation of virtual objects such as buttons, sliders and knobs. Such tracking is most often based on tracking each hand independently and when hands become partially occluded or are grasping a real object the hand tracking often fails. Tracking the hands during the manipulation of real-world objects opens up AR/VR to much richer forms of interaction and would provide the basis for activity recognition and the display of detailed contextual information related to the task at hand. This PhD project involves researching the tracking of unmodified hands with an ego-centric camera (2D and 3D) in the presence of partial occlusions. Technologies will include the use of deep learning models in combination with 3D models to determine hand pose in the presence of occlusion. Our approach will also exploit high level knowledge about object affordances and common hand grasp configurations which is commonly used in Robotic grasping.