Cooking activities are complex activities consisting of multiple steps or tasks. These tasks can be associated with one another based on two properties - the temporal structure that defines the sequence of occurrence of tasks and the objects that are used in the activity. This paper develops cooking activity models for the purpose of task prediction based on these two properties. The temporal structure of the sequence of tasks is captured by the novel hierarchical state space markov chain (HMC) and the object usage is represented using the object based task group (OTG) models. A probabilistic task prediction algorithm that fuses the HMC and OTG models has been developed to predict the next most probable task, given that a sequence of tasks has been completed. The proposed models and algorithms have been evaluated on two complex cooking activities - making brownies and making eggs, achieving a subject independent accuracy of 68.5% for predicting tasks, which is an improvement by an average of 6% in comparison to a Markov chain. The work done in the paper is first of its kind as it focuses on task prediction rather than task recognition. The task prediction framework described in the paper can be easily adapted to any complex activity supporting various annotation schemes and activity models.

Original languageEnglish (US)
Title of host publicationProceedings - 2010 IEEE International Symposium on Multimedia, ISM 2010
Number of pages6
StatePublished - 2010
Event2010 IEEE International Symposium on Multimedia, ISM 2010 - Taichung, Taiwan, Province of China
Duration: Dec 13 2010Dec 15 2010

Publication series

NameProceedings - 2010 IEEE International Symposium on Multimedia, ISM 2010


Other2010 IEEE International Symposium on Multimedia, ISM 2010
Country/TerritoryTaiwan, Province of China


  • Activity prediction
  • Cooking activties
  • Hierarchical state space
  • Markov Chain
  • Task prediction

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Human-Computer Interaction


Dive into the research topics of 'Task prediction in cooking activities using hierarchical state space Markov chain and object based task grouping'. Together they form a unique fingerprint.

Cite this