Multi-Head PPO for Joint Resource Allocation in FSO-RF Space-Air-Ground Networks

Non-terrestrial networks extend connectivity to re-mote and disaster-affected areas and provide high-capacity backhaul, making them attractive for resilient wide-area communications. This paper studies joint resource allocation in a space-air-ground system where a free-space optical (FSO) satellite backhaul serves multiple multi-antenna unmanned aerial vehicle (UAV) relays that in turn serve ground users over RF links. The resulting optimization mixes continuous power-splitting decisions with discrete beam-selection choices from a discrete Fourier transform codebook, producing a large nonconvex problem. We propose a hybrid Proximal Policy Optimization (PPO) actor-critic with Dirichlet heads for continuous power fractions and categorical heads for beam selection, implemented as a shared-backbone, multi-head policy. Simulation results show reliable convergence, consistent gains over a random-beam baseline, and strong performance across varying power regimes.