Human Activity Recognition (HAR) is considered a multidisciplinary field that different branches of science contribute to its advancements. Vision-Based HAR is one of the means to use Computer Vision (CV) and its techniques to study and analyze the behavior of humans within the context of videos. Recently, Video Anomaly detection (VAD) has gained vast attention and becomes a popular research topic in recent years. This is due to their enormous potential in many fields such as healthcare monitoring, surveillance/crowd analysis, sports, Ambient Assistive Living (AAL), event analysis, and security. Manually detecting and analyzing inappropriate behavior was a very challenging task, especially in real-time scenarios which led to a great demand for smart surveillance systems. In recent work, deep learning approaches have been dominated in this field as they outperform the performance of other traditional methods. This literature provides the latest algorithms for anomalous human activities, the challenges facing this field, and a comprehensive review of the State-Of-The-Art (SOTA) approaches including the feature extractor, the method, and the loss function. In addition, we propose the effect of applying swarm optimization algorithms in the anomaly detection field in recent years. Moreover, it presents a chronological background to the subject with an emphasis on the recent advancements in the VAD field.