Computer Vision Aided Beam Tracking in A Real-World Millimeter Wave Deployment

Shuaifeng Jiang, Ahmed Alkhateeb

Research output: Chapter in Book/Report/Conference proceedingConference contribution

13 Scopus citations


Millimeter-wave (mmWave) and terahertz (THz) communications require beamforming to acquire adequate receive signal-to-noise ratio (SNR). To find the optimal beam, current beam management solutions perform beam training over a large number of beams in pre-defined codebooks. The beam training overhead increases the access latency and can become infeasible for high-mobility applications. To reduce or even eliminate this beam training overhead, we propose to utilize the visual data, captured for example by cameras at the base stations, to guide the beam tracking/refining process. We propose a machine learning (ML) framework, based on an encoder-decoder architecture, that can predict the future beams using the previously obtained visual sensing information. Our proposed approach is evaluated on a large-scale real-world dataset, where it achieves an accuracy of 64.47% (and a normalized receive power of 97.66%) in predicting the future beam. This is achieved while requiring less than 1% of the beam training overhead of a corresponding baseline solution that uses a sequence of previous beams to predict the future one. This high performance and low overhead obtained on the real-world dataset demonstrate the potential of the proposed vision-aided beam tracking approach in real-world applications.

Original languageEnglish (US)
Title of host publication2022 IEEE GLOBECOM Workshops, GC Wkshps 2022 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages6
ISBN (Electronic)9781665459754
StatePublished - 2022
Event2022 IEEE GLOBECOM Workshops, GC Wkshps 2022 - Virtual, Online, Brazil
Duration: Dec 4 2022Dec 8 2022

Publication series

Name2022 IEEE GLOBECOM Workshops, GC Wkshps 2022 - Proceedings


Conference2022 IEEE GLOBECOM Workshops, GC Wkshps 2022
CityVirtual, Online


  • beam tracking
  • DeepSense 6G
  • machine learning
  • real-world data
  • sensing
  • vision

ASJC Scopus subject areas

  • Instrumentation
  • Computer Networks and Communications
  • Control and Optimization


Dive into the research topics of 'Computer Vision Aided Beam Tracking in A Real-World Millimeter Wave Deployment'. Together they form a unique fingerprint.

Cite this