Sentiment analysis is essential for comprehending public views on various issues. While Sentiment Analysis (SA) techniques have been widely adopted, the lack of transparency in conventional machine learning models inhibits a comprehensive understanding of the reasoning behind sentiment predictions, where this opacity hinders the trustworthiness of sentiment analysis models, limiting their applicability in real-world scenarios where interpretability is crucial. It is even worse when sentiment analysis is applied to Arabic text since the language's intricacy and cultural quirks produce particular difficulties, so the problem addressed in this paper revolves around the need for accurate sentiment analysis in Arabic textual data and providing the feature of interpreting the results of sentiment analysis and the predictions reached and making them more understandable. Furthermore, to achieve the vision and goal of the research, scientific steps have been implemented on several data sets, the most prominent of which is the Covid-19 data set which has produced a vast amount of public sentiment views regarding the virus and vaccination so we have used it in this work as a case study. This paper attempts to solve this issue by utilizing machine learning methods and explainable artificial intelligence (XAI) techniques to create a sentiment analysis framework, that not only achieves high sentiment prediction accuracy but also offers clear and understandable explanations for the underlying factors influencing sentiment classifications. The goal of the paper is to close the gap between human interpretability and accurate sentiment analysis by incorporating XAI approaches into machine-learning models.