This paper evaluates the quality of the Translation Memory embedded in MateCat, a computer-aided translation tool. MateCat's built-in Translation Memory, MyMemory, is constructed using a crowdsourcing system and is in constant development by the post-edited translations that are uploaded to the server from the huge number of MateCat's users. The current study carries out a translation of a literary text where the translation is implemented in two phases: Initial Translation and Follow-up Translation. Next, a comparison between the outputs of the two phases was drawn to observe the error rate after the two trials. Homage of Switzerland, a short story written by Ernest Hemmingway, was imported into MateCat to evaluate the Arabic output translation. The study concludes that despite the technological revolution in computer-aided translation tools, there is a considerable sum of linguistic errors on the Arabic language's lexical, grammatical, and morphological levels. The study also asserts that, despite the enormous efforts paid to develop computer programs to facilitate the translation tasks, human intervention is still a must.