A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving - Network and Parallel Computing Access content directly
Conference Papers Year : 2021

A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving


Convolutional neural networks (CNNs) are widely used in vision-based autonomous driving, i.e., detecting and localizing objects captured in live video streams. Although CNNs demonstrate the state-of-the-art detection accuracy, processing multiple video streams using such models in real-time imposes a serious challenge to the on-car computing systems. The lack of optimized system support, for example, could lead to a significant frame loss due to the high processing latency, which is unacceptable for safety-critical applications. To alleviate this problem, several optimization strategies such as batching, GPU parallelism, and data transfer modes between CPU/GPU have been proposed, in addition to a variety of deep learning frameworks and GPUs. It is, however, unclear how these techniques interact with each other, which particular combination performs better, and under what settings. In this paper, we set out to answer these questions. We design and develop a Multi-Tenant Parallel CNN Inference Framework, MPInfer, to carefully evaluate the performance of various parallel execution modes with different data transfer modes between CPU/GPU and GPU platforms. We find that on more powerful GPUs such as GTX 1660, it achieves the best performance when we adopt parallelism across CUDA contexts enhanced by NVIDIA Multi-Process Service (MPS), with 147.06 FPS throughput and 14.50 ms latency. Meanwhile, on embedded GPUs such as Jetson AGX Xavier, pipelining is a better choice, with 46.63 FPS throughput and 35.09 ms latency.
Fichier principal
Vignette du fichier
511910_1_En_8_Chapter.pdf (1.05 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03768759 , version 1 (04-09-2022)





Yitong Huang, Yu Zhang, Boyuan Feng, Xing Guo, Yanyong Zhang, et al.. A Close Look at Multi-tenant Parallel CNN Inference for Autonomous Driving. 17th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2020, Zhengzhou, China. pp.92-104, ⟨10.1007/978-3-030-79478-1_8⟩. ⟨hal-03768759⟩
19 View
4 Download



Gmail Facebook X LinkedIn More