The need for an automatic speech to text conversion is continuously increasing, especially for people with special needs. Thus, automatic speech recognition techniques have been proposed to tackle such needs. The automatic recognition allows computers to identify words from lip movement, regardless of the visual source. It is known that visual speech recognition techniques improve the accuracy of word identification and shield the recognition system against acoustic noise. In this paper, we propose a hybrid voting model for automatic Arabic figures recognition based on visual perception of mouth-lip movement. The proposed model has been built for Arabic language, such that, it is able to extract Arabic figures from predefined Arabic lexicon. The predefined lexicon mainly contains the Arabic figures from zero to nine with different shapes. The model takes a video (or sequence of images) as an input, and outputs the corresponding Arabic figure for the frames extracted from the input video. Here, three techniques have been employed to extract effective visual features from mouth-lip movement. Such techniques are SURF (speeded up robust features), HoG (histogram of oriented gradient) and Haar feature extractor. The resultant features in each technique are fed separately into a classification model, namely, the hidden Markov model (HMM). The HMM identifies corresponding Arabic figure from a predefined lexicon based on input features. The final classification models that are produced from the three techniques have been grouped in a voting scheme to produce the final classification result (i.e. classification by voting). The proposed model in this paper has been tested on handcrafted data set of lip movement, and it has shown a promising result with improved accuracy of Arabic figures recognition.