Unsupervised Learning of Optical Flow with CNN-Based Non-Local Filtering

Long Tian, Zhigang Tu, Dejun Zhang, Jun Liu, Baoxin Li, Junsong Yuan

Research output: Contribution to journalArticlepeer-review

33 Scopus citations


Estimating optical flow from successive video frames is one of the fundamental problems in computer vision and image processing. In the era of deep learning, many methods have been proposed to use convolutional neural networks (CNNs) for optical flow estimation in an unsupervised manner. However, the performance of unsupervised optical flow approaches is still unsatisfactory and often lagging far behind their supervised counterparts, primarily due to over-smoothing across motion boundaries and occlusion. To address these issues, in this paper, we propose a novel method with a new post-processing term and an effective loss function to estimate optical flow in an unsupervised, end-to-end learning manner. Specifically, we first exploit a CNN-based non-local term to refine the estimated optical flow by removing noise and decreasing blur around motion boundaries. This is implemented via automatically learning weights of dependencies over a large spatial neighborhood. Because of its learning ability, the method is effective for various complicated image sequences. Secondly, to reduce the influence of occlusion, a symmetrical energy formulation is introduced to detect the occlusion map from refined bi-directional optical flows. Then the occlusion map is integrated to the loss function. Extensive experiments are conducted on challenging datasets, i.e. FlyingChairs, MPI-Sintel and KITTI to evaluate the performance of the proposed method. The state-of-the-art results demonstrate the effectiveness of our proposed method.

Original languageEnglish (US)
Article number9159910
Pages (from-to)8429-8442
Number of pages14
JournalIEEE Transactions on Image Processing
StatePublished - 2020


  • Optical flow
  • loss function
  • non-local term
  • occlusion map
  • unsupervised learning

ASJC Scopus subject areas

  • Software
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Unsupervised Learning of Optical Flow with CNN-Based Non-Local Filtering'. Together they form a unique fingerprint.

Cite this