UAV Trajectory, User Association, and Power Control for Multi-UAV-Enabled Energy-Harvesting Communications: Offline Design and Online Reinforcement Learning

In this article, we consider multiple solar-powered wireless nodes (WNs) which utilize the harvested solar energy to transmit collected data to multiple unmanned aerial vehicles (UAVs) in the uplink. In this context, we jointly design UAV flight trajectories, UAV-node user association, and uplink power control to effectively utilize the harvested energy and manage co-channel interference within a finite time horizon. The design goal is to ensure the fairness of WNs by maximizing the worst user rate. The joint design problem is highly nonconvex and requires causal (future) knowledge of the instantaneous energy state information (ESI) and channel state information (CSI), which are difficult to predict in reality. To overcome these challenges, we propose an offline method based on convex optimization that only utilizes the average ESI and CSI, where line-of-sight (LOS) and non-LOS (NLOS) channels are considered. The problem is solved by three convex subproblems with successive convex approximation (SCA) and alternative optimization. We further design an online convex-assisted reinforcement learning (CARL) method based on real-time environmental information. An idea of multi-UAV regulated flight corridors, based on the optimal offline UAV trajectories, is proposed to avoid unnecessary flight exploration by UAVs and enables us to improve the learning efficiency and system performance, as compared with the conventional reinforcement learning (RL) method. Computer simulations are used to verify the effectiveness of the proposed methods. The proposed CARL method provides 25% and 12% improvement on the worst user rate over the offline and conventional RL methods.