DNN Partitioning for Cooperative Inference in Edge Intelligence: Modeling, Solutions, Toolchains

100%

DNN Partitioning for Cooperative Inference in Edge Intelligence: Modeling, Solutions, Toolchains

With rapid advancements in artificial intelligence and Internet of Things technologies, the deployment of deep neural network models on the edge nodes and the end nodes has become an essential trend. However, the limited computational power, storage capacity, and resource constraints of these devices present significant challenges for deep learning inference. Traditional acceleration methods, such as model compression and hardware optimization, often struggle to balance real-time performance, accuracy, and cost-effectiveness. To address these challenges, collaborative inference through DNN partitioning has emerged as a promising solution. This article provides a comprehensive overview of architectural frameworks for DNN partitioning in collaborative inference. We establish a unified mathematical framework to describe various architectures, DNN models, and their associated optimization problems. In addition, we systematically classify and analyze existing partitioning strategies based on partition count and granularity. Furthermore, we summarize commonly used experimental setups and tools, offering practical insight into implementation. Finally, we discuss key challenges and open issues in DNN partitioning for collaborative inference, such as ensuring data security and privacy and efficiently partitioning large-scale models, providing valuable guidance for future research.

One Introduction

With rapid advances in artificial intelligence and Internet of Things, deep neural networks have been widely applied in fields such as image recognition, natural language processing, autonomous driving, and intelligent healthcare. Deep neural networks excel in classification, prediction, and decision-making due to their high accuracy. However, increasing task complexity demands larger models with more parameters and deeper architectures, leading to greater computational and storage requirements. For example, AlexNet achieved improved image recognition accuracy with eight layers and sixty million parameters; VGG-sixteen employed three by three convolutions, extending to sixteen layers and one hundred forty million parameters; GPT-three has one hundred seventy-five billion parameters, requiring over seven hundred gigabytes of memory and three thousand five hundred eighty teraflops per inference.

Despite improvements in accuracy and efficiency, deep neural networks' computational and storage demands continue to grow. As deep neural network applications expand, inference bottlenecks intensify, especially in resource-limited edge environments with constrained computation, storage, and energy. In real-time applications like autonomous driving and video analysis, the heavy computation often exceeds device capacities, causing latency and performance degradation. Therefore, optimizing inference efficiency while preserving accuracy is crucial. Various acceleration techniques have been proposed in academia and industry; these methods are summarized as follows:

One. Model Optimization: Various techniques have been proposed to reduce deep neural network complexity while maintaining accuracy, such as quantization, pruning, knowledge distillation, and early-exit mechanisms.

Two. Hardware Acceleration: Inference efficiency is further boosted through specialized hardware including graphics processing units, tensor processing units, field-programmable gate arrays, and application-specific integrated circuits.

Three. DNN Partitioning: Deep neural network partitioning reduces inference latency by leveraging the architecture's inter-layer connectivity and output size reduction. For instance, Tiny YOLOv-two's input is zero point ninety-five megabytes, while its max five layer output is only zero point zero eight megabytes-a ninety-three percent decrease. This size reduction minimizes data transmission and eases network bandwidth requirements. Partitioning distributes layers or sub-layers across nodes, balancing computation and communication through pipelined execution. As shown in Figure one, Node one (resource-constrained, e.g., end or edge device) and Node two (more powerful, e.g., edge server or cloud) are linked via wired or wireless connections. The initial layers run on Node one, with subsequent, heavier layers offloaded to Node two. Intermediate feature maps are transmitted between nodes following the model's layerwise dependencies. By placing the partition where feature maps shrink significantly, transmission overhead is minimized without sacrificing accuracy, effectively balancing computation and communication costs.

While both model optimization and deep neural network partitioning target neural network processing, they differ fundamentally. Model optimization reduces model complexity by modifying the network structure through pruning, quantization, or knowledge distillation, often requiring extensive retraining and risking performance loss in complex tasks. Such compressed models are typically platform- or dataset-specific, limiting generalizability. Hardware acceleration improves speed and energy efficiency via specialized hardware but faces deployment constraints in edge environments due to cost and power. Conversely, deep neural network partitioning retains the original model architecture, optimizing execution by distributing computation and communication across nodes (cloud, edge, end devices) based on their capabilities, making it well-suited for distributed, resource-limited scenarios that demand accuracy preservation.

One point one Comparison and Our Contribution

One point two The Organization of the Survey

Two DNN Collaborative Inference: Model Structure, Scenario Architecture, and Optimization Issues

Two point two Collaborative Inference Architecture in Edge Computing

Two point three Optimization Problems

Three. Method

Three point one. One-Dimensional Solution Space: Single-Layer Solution Space

Three point two. Two-Dimensional Solution Space: Single-Layer and Model Optimization Solution Space

Three point three. Two-Dimensional Solution Space: Single-Layer and Resource Allocation Solution Space

Three point four. Two-Dimensional Solution Space: Multiple-Layer/Sub-Layer and Offloading Decision Solution Space

Three point six. One-Dimensional Solution Space: Tensor Parallelism Based on DNN Sub-Layers

Four. Experimental Setup and Commonly Used Tools

Four point one. Summary of DNNs, Associated Tools, and Datasets

Four point two. Summary of Computing Nodes and Communication Tools

Four point three. Summary of Resource Control Tools

Four point four. Summary of Parameter Analysis and Measurement Tools

Five. Challenges and Open Issues

Five point two. Robustness

Five point three. Partitioning of Large-Scale Models

Six. Conclusion

Overview

The article provides an in-depth overview of DNN partitioning for collaborative inference in edge intelligence, addressing challenges in deployment due to limited computational resources. It reviews existing approaches and proposes a unified framework for optimizing collaborative inference strategies.

Key Points

1DNN partitioning can improve efficiency in edge intelligence applications
2The study presents a unified mathematical framework for various DNN architectures
3It classifies existing partitioning strategies according to granularity and count
4Key challenges include data security, privacy, and model partitioning efficiency
5The work also reviews Transformer-specific partitioning methods and their unique considerations.

Details

Authors: Yuntao Hao, Nan Ding, Weiguo Xia, Hongwei Ge, Li Xu
Category: Technology and Engineering

PDF
TBI-603 Network Security & Cyber Law Unit Notes
This document serves as comprehensive unit notes for a course on Network Security and Cyber Law, covering topics like network security practices, types of security attacks, and legal aspects of cybersecurity.
PDF
ESTI IT2409 Expressions and Assignment Statements In Python
This document provides an in-depth overview of expressions and assignment statements in Python, explaining their fundamental role in programming logic, calculations, and data management. It includes various types of expressions, examples, and the use of assignment statements for variable management.
PDF
Molecular deep learning at the edge of chemical space
This article discusses a novel joint modeling approach to molecular machine learning that enables better predictions for structurally novel molecules by introducing a metric called unfamiliarity to assess generalizability beyond the training chemical space.
PDF
EC8702 - Ad Hoc and Wireless Sensor Networks: All Units - 16 Marks Questions with Answers
This document contains a comprehensive set of 16 marks questions and answers regarding Ad Hoc and Wireless Sensor Networks for the EC8702 subject, specifically tailored for the B.E. ECE 7th Semester under the Anna University Regulation 2017. It covers various important topics and previous year questions from 2020 to 2024.
PDF
Deception Abilities Emerged in Large Language Models
This study investigates the emergence of deception capabilities in state-of-the-art large language models (LLMs), showcasing their potential to understand and induce false beliefs in others, which has significant implications for AI safety and ethics.