Discrete Cosine Transform (DCT) and Discrete Hartley Transform (DHT) based on Orthogonal Frequency Division Multiplexing (OFDM) systems are proposed in this project that aims to enhance the performance of the standard Discrete Fourier Transform (DFT) based on OFDM system. The investigation of the Bit Error Rate (BER) performance and the reduction of the Peak-to-Average Power Ratio (PAPR) of different OFDM systems over doubly dispersive fading channel is illustrated. This paper first analyzes the characteristics of the DFT-, DCT- and DHT-based OFDM systems and then use the algorithms for signal processing to reduce the PAPR by using Selected Mapping technique (SLM) and Partial Transmit Sequence technique (PTS). It has been verified mathematically that these approaches will enhance the BER performance and reduce the PAPR. Finally, the simulation results show that using DCT- and DHT-based OFDM system instead of DFT-based OFDM system enhance the BER performance by (4) dB to achieve BER (10-3). The suggested PAPR threshold reduction scheme is statistically feasible. Statistically, through significant improvement in PAPR, preventing nonlinear fluctuations in OFDM systems.