BATCH-DNN: Adaptive and Dynamic Batching for Multi-DNN Accelerators

Paper i proceeding, 2026

Multi-DNN accelerators enable the simultaneous execution of multiple DNN workloads which improves performance by overlapping computations and memory accesses of multiple DNN workloads. However, on-chip memory must accommodate the footprint of all workloads. Batching allows DNN inferences using the same model to share weights which improves weight reuse and reducing off-chip access costs over a batch. Batching determines the batch size statically, leading to stalls when there is not enough on-chip memory available at runtime. This paper introduces BATCH-DNN, a dynamic method for adapting batch size on a layer-by-layer basis to available on-chip memory. It employs two techniques: adaptive cascaded sub-batching and adaptive sub-batch merging. Offline profiling establishes the footprint, while run-time adjustment establishes the maximum batch size on a layer-by-layer basis based on available on-chip memory. BATCH-DNN can improve the utilization of accelerator compute fabrics by 60%, which increases throughput by up to 27% and by 6%, on average, for batched multi-DNN workloads.