A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware

Zhiyuan Xu, Dejun Yang, Chengxiang Yin, Jian Tang, Yanzhi Wang, Guoliang Xue

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

With the emergence of more and more powerful chipsets and hardware and the rise of Artificial Intelligence of Things (AIoT), there is a growing trend for bringing Deep Neural Network (DNN) models to empower mobile and edge devices with intelligence such that they can support attractive AI applications on the edge in a real-time or near real-time manner. To leverage heterogeneous computational resources (such as CPU, GPU, DSP, etc) to effectively and efficiently support concurrent inference of multiple DNN models on a mobile or edge device, we propose a novel online Co-Scheduling framework based on deep REinforcement Learning (DRL), which we call COSREL. COSREL has the following desirable features: 1) it achieves significant speedup over commonly-used methods by efficiently utilizing all the computational resources on heterogeneous hardware; 2) it leverages emerging Deep Reinforcement Learning (DRL) to make dynamic and wise online scheduling decisions based on system runtime state; 3) it is capable of making a good tradeoff among inference latency, throughput and energy efficiency; and 4) it makes no changes to given DNN models, thus preserves their accuracies. To validate and evaluate COSREL, we conduct extensive experiments on an off-the-shelf Android smartphone with widely-used DNN models to compare it with three commonly-used baselines. Our experimental results show that 1) COSREL consistently and significantly outperforms all the baselines in terms of both throughput and latency; and 2) COSREL is generally superior to all the baselines in terms of energy efficiency.

Original languageEnglish (US)
Pages (from-to)1275-1288
Number of pages14
JournalIEEE Transactions on Mobile Computing
Volume22
Issue number3
DOIs
StatePublished - Mar 1 2023
Externally publishedYes

Keywords

  • Mobile computing
  • deep learning
  • deep reinforcement learning
  • edge computing
  • on-device DNN inference

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices With Heterogeneous Hardware'. Together they form a unique fingerprint.

Cite this