In recent years, plenty of object tracking methods have been put forward for better tracking accuracies. However, few of them can be applied to the real-time applications due to high computational cost. In this paper, aiming at achieving better real-time tracking performance, we propose an adaptive robust framework for object tracking based on the CamShift approach, which is notable for its simplicity and high efficiency. An adaptive local search method, which is composed of two key steps, i.e. iterative extension and iterative split search, is put forward to search for the optimal object candidate to avoid that the CamShift tracker may get confused by the distracters from the surrounding background and hence erroneously incorporate them into the object region. A Kalman filter is also incorporated into our framework for prediction of the object's movement, so as to reduce the search effort and possible tracking failure caused by fast object motion. The experimental results demonstrate that the proposed tracking framework is robust and computationally effective.