Filter bank multi-carrier (FBMC) is a popular researched technique for wireless communications that offers several advantages over traditional orthogonal frequency-division multiplexing (OFDM) systems. In this paper, we propose a FBMC system using offset Triangle phase shift keying (TPSK) modulation (FBMC/OTPSK), analyze its bit error rate (BER)
performance via additive white Gaussian noise (AWGN) channel, and compare it to that of conventional FBMC using offset quadrature amplitude modulation (FBMC/OQAM) system. Simulation results show that the proposed FBMC/OTPSK system considerably improves BER performance over FBMC/OQAM system. Additionally, we explore the impact of using ternary convolutional coding (TCC) with TPSK and binary convolutional coding (BCC) with QAM. The presented TPSK modulation encoded by TCC offers better error rate performance and minimum amplitude fluctuation compared to QAM encoded by BCC; therefore, TCC-coded OTPSK is proposed to be used with FBMC instead of BCC-coded OQAM to benefit from its lower error rates without bandwidth expansion, as well increase information carried by symbol, hence increases throughput.