TY - GEN
T1 - Improving the Efficiency of CMOS Image Sensors through In-Sensor Selective Attention
AU - Zhang, Tianyi
AU - Kasichainula, Kishore
AU - Jee, Dong Woo
AU - Yeo, Injune
AU - Zhuo, Yaoxin
AU - Li, Baoxin
AU - Seo, Jae Sun
AU - Cao, Yu
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - Inspired by the selective attention mechanism in human vision, we propose to introduce a saliency-based processing step in the CMOS image sensor, to continuously select pixels corresponding to salient objects and feedback such information to the sensor, instead of blindly passing all pixels to the sensor output. To minimize the overhead of saliency detection in this feedback loop, we propose two techniques: (1) saliency detection with low-precision, down-sampled grayscale images, and (2) Optimization of the loss function and model structure. Finally, we pad the minimum number of pixels around the selected pixels to maintain the accuracy of object detection (OD). Our method is experimented with two types of OD algorithms on three representative datasets. At the similar OD accuracy with the full image, our proposed selective feedback method successfully achieves 70.5% reduction in the volume of output pixels for BDD100K, which translates to 4.3× and 3.4× reduction in power consumption and latency, respectively.
AB - Inspired by the selective attention mechanism in human vision, we propose to introduce a saliency-based processing step in the CMOS image sensor, to continuously select pixels corresponding to salient objects and feedback such information to the sensor, instead of blindly passing all pixels to the sensor output. To minimize the overhead of saliency detection in this feedback loop, we propose two techniques: (1) saliency detection with low-precision, down-sampled grayscale images, and (2) Optimization of the loss function and model structure. Finally, we pad the minimum number of pixels around the selected pixels to maintain the accuracy of object detection (OD). Our method is experimented with two types of OD algorithms on three representative datasets. At the similar OD accuracy with the full image, our proposed selective feedback method successfully achieves 70.5% reduction in the volume of output pixels for BDD100K, which translates to 4.3× and 3.4× reduction in power consumption and latency, respectively.
KW - image sensor
KW - latency
KW - object detection
KW - power consumption
KW - saliency detection
KW - selective attention
UR - http://www.scopus.com/inward/record.url?scp=85167658524&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85167658524&partnerID=8YFLogxK
U2 - 10.1109/ISCAS46773.2023.10181835
DO - 10.1109/ISCAS46773.2023.10181835
M3 - Conference contribution
AN - SCOPUS:85167658524
T3 - Proceedings - IEEE International Symposium on Circuits and Systems
BT - ISCAS 2023 - 56th IEEE International Symposium on Circuits and Systems, Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 56th IEEE International Symposium on Circuits and Systems, ISCAS 2023
Y2 - 21 May 2023 through 25 May 2023
ER -