As web cameras cost is much lower and easier to get nowadays, the ways to develop an interesting applications and interaction methods are hot research topics now. In this paper, an effective real-time virtual control mechanism based on hand gesture recognition is proposed. Through developed algorithms, hand gesture can be recognized efficiently to further control the application without any additional equipments or devices. Furthermore, the disturbance of human face and arm that have the same feature, skin color, with hand is excluded to enhance the accuracy of hand gesture recognition. A ball game is designed to evaluate the performance of the proposed system. Experimental results that the proposed system provides high accuracy in virtual control and has high potential for various kinds of applications.