In this paper we discuss a system for finding and segmenting a human hand, which is holding an object. This is a difficult computer vision problem, because of the many degrees of freedom, the flexibility, and the clutter problem of the hand. In order to find the correspondences we have used a lateral similarity function. In the lateral excitation we have not only compared different features at one point but also the features at that point and its neighbors ( for example : Gobar feature , and skin color feature ). The system is an extension of the Elastic Graph Matching (EGM) algorithm [1]. EGM has been shown to be successful in numerous object vision tasks, its best performance being in face recognition.