14:30
17:30

The rise of deep neural networks has revolutionised many fields, including computer vision and natural language processing. However, the increasing size of these models means that the computational demands are becoming ever greater, putting current computing infrastructures to the test. Graphics Processing Units (GPUs), although optimised for massively parallel computing, can be overloaded by a high throughput of inference requests, leading to congestion and exploding latency. At the same time, central processing units (CPUs), often associated with GPUs for management tasks, are not fully exploited despite their potential for handling certain less computationally intensive tasks. It therefore seems appropriate to use available CPUs to assist GPUs in their inference calculations. In this thesis, we propose StarONNX, a solution aimed at accelerating inference computations on heterogeneous computing systems combining CPUs and GPUs, particularly in the case of heavy inference loads.
StarONNX is based on the integration of DNN model partitions obtained using the METIS partitioning tool. This enables asymmetric partitioning of the computing volume. StarONNX is based on StarPU, an execution system optimised for dynamic task scheduling on heterogeneous multicore architectures, and on ONNX Runtime, a high-performance engine for model inference in ONNX format. Thanks to this combination, StarONNX exploits the specific capabilities of each processor to improve inference throughput and latency while maintaining efficient resource management. Our approach is based on a combination of methods from the field of high-performance computing aimed at maximising the use of available resources. Firstly, we use the possibility of superimposing calculations and communications between processors in time.
Secondly, we optimise the use of computing resources by grouping CPU cores to execute tasks. Thirdly, we introduce pipelining of inference tasks, enabling different processing steps to be executed simultaneously on GPUs and CPUs, thereby increasing parallelism and throughput. When we compare our solution with NVIDIA's Triton inference server, we see significant improvements in terms of minimising latency. Our solution pushes the system congestion threshold to higher throughputs and fully exploits previously under-utilised CPU capacities. Despite the additional latency introduced by partitioning, this remains acceptable for partitioning into two or three segments.

Salle Ada Lovelace (Inria)