This paper aims to identify security risks associated with the use of 5G technology in IoT networks. It compares current security approaches for IoT communication based on 5G networks and analyzes key performance indicators concerning security threats. The paper focuses on differentiating security protocols for IoT applications with reference to 5G and aims to establish how various security issues can be addressed by each protocol. Future research challenges include improving encryption technologies, better authentication modes, prevention mechanisms, and security models. The objective of this systematic literature review is to contribute to the improved understanding of conventional security protocols for 5G-IoT environments, extending research in the domain and assisting academics in creating new ways to enhance the resilience and security of 5G-based IoT systems, This research paper delves into the security threats emerging from the application of 5G technology in IoT networks. It explores how the enhanced capabilities of 5G, while beneficial, also present new security challenges