Multi-Agent DRL for Queue-Aware Task Offloading in Hierarchical MEC-Enabled Air-Ground Networks

Mobile edge computing (MEC)-enabled air-ground networks advance 6G wireless networks by utilizing aerial base stations (ABSs) such as unmanned aerial vehicles (UAVs) and high altitude platform stations (HAPS) to provide dynamic services to ground IoT devices (IoTDs). These IoTDs support real-time applications like multimedia and Metaverse services, which demand high computational resources and strict quality of service (QoS) guarantees, specifically in terms of latency and efficient task queue management. However, IoTDs often face constraints in energy and computational power, requiring efficient queue management and task scheduling to maintain QoS. To address these challenges, UAVs and HAPS are deployed to supplement the computational limitations of IoTDs by offloading tasks for distributed processing. Due to UAVs' resource limitations, particularly in terms of power and coverage area, HAPS are used to enhance their capabilities and extend coverage. Overloaded UAVs may relay tasks to HAPS, creating a multi-tier computing system. This paper addresses the overall energy minimization problem in the MEC-enabled air-ground integrated network (MAGIN) by optimizing UAV trajectories, computing resource allocation, and queue-aware task offloading decisions. The optimization problem is highly complex due to the nonconvex and nonlinear nature of this hierarchical system, which traditional methods cannot effectively solve. To tackle this, we reformulate the problem as a multi-agent Markov decision process (MDP) with continuous action spaces and heterogeneous agents. We propose a novel variant of multi-agent proximal policy optimization (MAPPO) with Beta distribution (MAPPO-BD) to solve this problem. Extensive simulations show that MAPPO-BD significantly outperforms other baselines, achieving superior energy savings and efficient resource management in MAGIN, while adhering to constraints related to queue delays and edge computing capabilities.