Graph based multi-modality learning

Hanghang Tong, Jingrui He, Mingjing Li, Changshui Zhang, Wei Ying Ma

Research output: Chapter in Book/Report/Conference proceedingConference contribution

95 Scopus citations


To better understand the content of multimedia, a lot of research efforts have been made on how to learn from multi-modal feature. In this paper, it is studied from a graph point of view: each kind of feature from one modality is represented as one independent graph; and the learning task is formulated as inferring from the constraints in every graph as well as supervision information (if available). For semi-supervised learning, two different fusion schemes, namely linear form and sequential form, are proposed. For each scheme, it is derived from optimization point of view; and further justified from two sides: similarity propagation and Bayesian interpretation. By doing so, we reveal the regular optimization nature, transductive learning nature as well as prior fusion nature of the proposed schemes, respectively. Moreover, the proposed method can be easily extended to unsupervised learning, including clustering and embedding. Systematic experimental results validate the effectiveness of the proposed method.

Original languageEnglish (US)
Title of host publicationProceedings of the 13th ACM International Conference on Multimedia, MM 2005
Number of pages10
StatePublished - 2005
Externally publishedYes
Event13th ACM International Conference on Multimedia, MM 2005 - Singapore, Singapore
Duration: Nov 6 2005Nov 11 2005

Publication series

NameProceedings of the 13th ACM International Conference on Multimedia, MM 2005


Conference13th ACM International Conference on Multimedia, MM 2005


  • Bayesian interpretation
  • Graph model
  • Multi-modality analysis
  • Regularized optimization
  • Similarity propagation

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction
  • Software


Dive into the research topics of 'Graph based multi-modality learning'. Together they form a unique fingerprint.

Cite this