HAP-Based Underlay Non-Terrestrial Networks with Imperfect CSI: Joint RSMA Transmission and Beyond-Diagonal RIS Beamforming

Non-terrestrial networks (NTNs) have emerged as a promising technology for providing ubiquitous connectivity in remote, underserved, and disaster-stricken regions. In particular, high-altitude platforms (HAPs) can offer wide-area coverage; however, their performance is often limited by severe path loss, multi-user interference, and spectrum-sharing constraints. To address these challenges, this paper investigates a multi-antenna HAP-based underlay aerial-to-ground communication network employing rate-splitting multiple access (RSMA) and assisted by a terrestrial beyond-diagonal reconfigurable intelligent surface (BD-RIS). The objective is to maximize the system sum rate while satisfying user rate requirements, HAP transmit-power constraints, and interference-temperature constraints imposed to protect the primary network. The design further accounts for imperfect channel state information (CSI) through a worst-case robust optimization framework. The resulting problem is highly non-convex due to the coupled optimization of RSMA precoding and BD-RIS beamforming. To address this challenge, the transmit precoding design is reformulated as a convex semidefinite program and solved using successive convex approximation and the MOSEK solver, while the BD-RIS scattering matrix is optimized over the unitary manifold using Riemannian manifold optimization. Simulation results demonstrate the effectiveness of the proposed framework and show that the BD-RIS-assisted system achieves up to a 45.2% sum-rate improvement compared with conventional single-connected RIS (SC-RIS) architectures while maintaining robustness against CSI uncertainty and satisfying all system constraints.