In this study, a real-time tongue tracking system is developed. The general goal of this system is to track a user’s tongue in a safe, non-contact manner using a webcam and image processing algorithms. This system first detects the approximate location of the mouth. Then, the exact mouth state and tongue direction are determined by image classification and post-processing. A rapid object detection algorithm that uses multi-scale block local binary patterns (MB-LBPs) is applied in this system. Instead of pixel intensities, this algorithm employs MB-LBP features for computations in processing digital images, which significantly reduces computational requirements and increases the frame rate. The gentle adaptive boosting (AdaBoost) meta-algorithm is used to obtain one classifier for each mouth/tongue state. Six-state classification accuracy is measured using a 6-fold cross-validation test with six subjects of various ethnicities recorded in two dissimilar lighting conditions. A 6-way classification accuracy of 89.01% is achieved, which is comparable to that of our previous prototype but more robust to variations in ambient lighting and head pose, and more effective in distinguishing faces from the background. In addition, two sets of leave-one-out tests are performed in a third lighting condition (accuracy of 95.8% observed), demonstrating the system’s ability to generalize to new environments, a key requirement for eventual deployment. The system can be conveniently extended to cover even more diverse mouth/tongue shapes, skin tones, and lighting conditions with further training. This system may enable the gamification of speech and language therapy by permitting users to control an engaging application with their tongues. Ultimately, such a system may be useful as part of speech therapy or stroke-recovery regimes.