Characteristics-based data-driven reinforcement learning for optimal control of nonlinear hyperbolic PDE systems

This paper addresses the optimal control of a class of nonlinear hyperbolic partial differential equation (PDE) systems with multiple transport velocities and unknown dynamics. Such systems arise naturally in transport–reaction processes, including fixed-bed chemical reactors, where interacting temperature and concentration waves propagate at different speeds. To overcome the limitations of model-based approaches, a data-driven optimal control framework based on integral reinforcement learning (IRL) is developed in an infinite-dimensional setting. The proposed approach integrates the method of characteristics with the Hamilton–Jacobi–Bellman formulation, transforming the hyperbolic PDE system into characteristic-based ordinary differential equations for trajectory and cost evaluation. This enables exact computation of state trajectories and accumulated costs without spatial discretization or explicit knowledge of the nonlinear system operators. An iterative policy evaluation and improvement scheme is derived using smooth value function approximations and least-squares-based updates, yielding admissible distributed control laws. Convergence properties of the proposed policy iteration scheme are established under standard assumptions. The effectiveness of the method is demonstrated through an application to a nonlinear fixed-bed reactor model, where the proposed controller achieves improved closed-loop performance and maintains robust regulation in the presence of inlet disturbances and moderate plant-parameter variations under actuator constraints.