Reliability assessment of a digital dynamic system using traditional Fault Tree Analysis (FTA) is difficult. This paper addresses the dynamic modeling of safety-critical complex systems such as the digital Reactor Protection System (RPS) in Nuclear Power Plants (NPPs). The digital RPS is a safety system utilized in the NPPs for safe operation and shut-down of the reactor in emergency events. A quantitative evaluation reliability analysis for the digital RPS with 2-out-of-4 architecture using the state transition diagram is presented in this paper. The study assesses the effects of independent hardware failures, Common Cause Failures (CCFs), and software failures on the failure of the RPS through calculating Probability of Failure on Demand (PFD). The results prove the validity of the proposed method in analyzing and evaluating reliability of the digital RPS and also show that the CCFs and longer detection time are the main contributions to the PFD of digital RPS.