Brain cancer, a perilous disease, underscores the critical need for brain tumor classification to enhance treatment outcomes and increase patient survival rates. Nevertheless, the challenging task of classifying brain tumors in their initial stages is compounded by variations in size, shape, and appearance. Deep learning (DL) gained prominence as a promising solution, particularly in the healthcare sector, utilizing brain magnetic resonance (MR) images for effective detection and classification. The prevalent use of transfer learning via fine-tuning addresses this challenge, where specific layers of a pre-trained architecture are adapted for a related target task. Despite its efficacy, selecting the optimal fine-tuning layers remains a key issue. This study presents a novel system employing a fine-tuning approach with manually chosen layers across five diverse architectures (EfficientNetV2s, Inception ResNetV2, MobileNetV2, RegNetY-320, and ConvNeXt-Large). A Global Average Pooling (GAP) layer was implemented at the output to address overfitting and vanishing gradient challenges, while a dropout layer was added to improve generalization. A comprehensive evaluation of multiple models on the BT-Large-4C dataset, which consists of 3,264 brain MRI images, shows that the fine-tuned EfficientNetV2s architecture outperforms other models. It achieved an impressive test accuracy of 97.86% while using only image resizing as the preprocessing step. Additionally, EfficientNetV2s outperforms state-of-the-art methods, making it a highly efficient and effective choice for classification of brain tumors. This study underscores the effectiveness of tailored fine-tuning in improving brain tumor classification.