Manipulation action tree bank: A knowledge resource for humanoids

Yezhou Yang, Anupam Guha, Cornelia Fermüller, Yiannis Aloimonos

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Scopus citations


Our premise is that actions of manipulation are represented at multiple levels of abstraction. At the high level a grammatical structure represents symbolic information (objects, actions, tools, body parts) and their interaction in a temporal sequence, and at lower levels the symbolic quantities are grounded in perception. In this paper we create symbolic high-level representations in the form of manipulation action tree banks, which are parsed from annotated action corpora. A context free grammar provides the grammatical description for the creation of the semantic trees. Experiments conducted on the tree banks show that they allow to 1) generate so-called visual semantic graphs (VSGs), 2) compare the semantic distance between steps of activities and 3) discover the underlying semantic space of an activity. We believe that tree banks are an effective and practical way to organize semantic structures of manipulation actions for humanoids applications. They could be used as basis for 1) automatic manipulation action understanding and execution and 2) reasoning and prediction during both observation and execution. The knowledge resource follows the widely used Penn Tree Bank format.

Original languageEnglish (US)
Title of host publication2014 IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014
PublisherIEEE Computer Society
Number of pages6
ISBN (Electronic)9781479971749
StatePublished - Feb 12 2015
Externally publishedYes
Event2014 14th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014 - Madrid, Spain
Duration: Nov 18 2014Nov 20 2014

Publication series

NameIEEE-RAS International Conference on Humanoid Robots
ISSN (Print)2164-0572
ISSN (Electronic)2164-0580


Other2014 14th IEEE-RAS International Conference on Humanoid Robots, Humanoids 2014

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition
  • Hardware and Architecture
  • Human-Computer Interaction
  • Electrical and Electronic Engineering


Dive into the research topics of 'Manipulation action tree bank: A knowledge resource for humanoids'. Together they form a unique fingerprint.

Cite this