In this paper a new reinforcement learning strategy is used for on-line tuning the control system of the aerodynamic missile. Aerodynamics missile automatic control system's mission is to overcome the missile's flight various disturbances encountered in the process of precise and real-time control of missiles attitude. Reinforcement learning algorithm (RL) is used to tune a PID controller to replace "gain schedule" Technique usually used. The result shows that RL with the new reward function is able to optimize the PID parameters with advantage over old method in terms of convergence speed and smaller overshoot