AI-Driven RSMA Beamforming for Integrated Sensing and Communication in Terrestrial-Satellite Networks

The increasing demand for high-capacity and globally accessible wireless services has intensified the need for spectrum-efficient and wide-coverage solutions. Integrated terrestrial-satellite networks (ITSNs) offer a promising architecture to address these challenges, while with the deployment of integrated sensing and communication (ISAC) and rate-splitting multiple access (RSMA) further enhances system functionality by supporting joint communication and sensing while effectively managing interference. This paper investigates RSMA for a multi-antenna LEO satellite that shares the licensed spectrum of terrestrial distributed MIMO systems to simultaneously perform target sensing and provide communication services. A weighted sum-rate maximization problem is formulated, subject to power, sensing, and interference constraints. To solve the resulting non-convex problem, we develop a hybrid solution that combines a deep convolutional neural network (CNN) for power allocation with a semidefinite relaxation (SDR)-based method for precoding and rate optimization. Simulation results demonstrate that the proposed scheme satisfies all constraints and achieves performance close to a successive convex approximation (SCA)based benchmark, while significantly reducing computational time, which makes it suitable for real-time deployment in resource-constrained satellite systems. Additionally, the RSMA-based approach outperforms conventional baseline methods.