This paper is devoted to investigate the nonlinear vibration control of a horizontally supported rotor system governed by two coupled second-order differential equations having both quadratic and cubic nonlinearities. A combination of the proportional controller and the integral resonant controller is employed to suppress the undesired lateral vibration of the considered rotor system. In addition, an 8-pole active-magnetic-bearing is utilized as an active actuator through which the control signal will be applied on the rotor system in the form of electromagnetic attractive forces. Based on the proposed control strategy, the whole system mathematical model has been established as a nonlinear two-degree-of-freedom system coupled to two first-order differential equations. Then, the multiple scales perturbation method has been applied to obtain an analytical solution for the obtained mathematical model up to the second-order approximation. Using the derived analytical solution, the influence of the different control gains on the system vibration amplitudes has been explored in terms of different bifurcation diagrams. The obtained results showed that the proposed control method can mitigate the rotor undesired vibrations close to zero regardless of the excitation force amplitude. Moreover, numerical validation for all obtained analytical results has been performed using the Runge-Kutta algorithm, where an excellent agreement between the analytical and numerical results is demonstrated.