Billions of active online users are continuously feeding the world with multimedia Big Data through their smart phones and PCs. These heterogenous productions are existing in different social media platforms, such as Facebook and Twitter, delivering a composite message in the form of audio, visual and textual signals. Analyzing multimedia Big Data to understand the intended delivered message, had been a challenge to audio, image, video and text processing researchers. Thanks to the recent advances in deep learning algorithms, researchers had been able to improve the performance of multimedia Big Data analytics and understanding techniques This paper presents a survey on how a multimedia file is analyzed, key challenges facing multimedia analysis, and how deep learning is helping conquer and advance beyond those challenges. Future directions of multimedia analysis are also addressed. The aim is to stay objective all through this study, bringing both empowering enhancements and in addition inescapable shortcomings, wishing to bring up fresh questions and stimulating new research frontiers for the reader.