Target tracking is an important field that attracted a lot of interest in computer vision. This paper describes an end-to-end technique for extracting moving targets from a real-time video stream and real sequence of live camera when the appearance of the object changes. Targets are classified into predefined categories according to image based properties,
and then robustly tracking them. However, since the scale of the targets often varied irregularly, systems with fixed-size tracking window usually could not accommodate to these scenarios. In present paper, a modified approach of tracking algorithm with Self-Updating Tracking was introduced. The proposed algorithm divided into two stages, the first stage
depending on detecting the sharp corner of moving target based on Lucas-kanade techniques. The second stage detecting the color of target being detected in the previous stage and able to track the detected color belonged to moving target with adaptive window size. Experimental results demonstrated that the improved algorithms could select the proper size of tracking window not only when the object scale increases but the scale decreases as well with minor extra computational overhead.