The Generalized Hough Transform (GHT) has been applied successfully on the area of hand posture recognition. However, for a typical hand posture recognition application, the position of user’s hand can be detected in advance of the matching process. Moreover, the variations on posture’s size and angle between training and test images of identical posture are within limited ranges. Under these hypotheses, so many redundant votes are cast into the Hough accumulator and this leads to a high probability of recognition error. A method called exclusion-before-voting is proposed in this paper to deal with this problem. We exclude these redundant votes according to the absolute object reference point position (AORPP), the scale, and orientation information of the interest points (keypoints). Experimental results show that our method not only improves the recognition rate, but also saves the recognition time. Moreover, with this method we demonstrate that a 2D Hough accumulator is sufficient to achieve similar performances of a 4D Hough Accumulator. Thus the memory cost can be saved too.