The proposed efficient human detection system is based on an adaptive neural fuzzy network (ANFN). In the preprocessing process, we apply a background subtraction algorithm with Gaussian mixture model (GMM) background model to extract moving objects, and adopt a shadow elimination process to eliminate some noise and irregular moving objects. The modified independent component analysis (mICA) based conditional entropy is presented to extract and select the efficient features (independent components). Furthermore, we use an adaptive neural fuzzy network as a human detection system to recognize human objects. The ANFN model uses a functional link neural network (FLNN) to create the consequent part of the fuzzy rules. The orthogonal polynomial is applied as a functional expansion of the FLNN. The learning process of ANFN consists of structure learning and parameter learning. The structure learning depends on the entropy measure to determine the number of fuzzy rules. The parameter learning, based on the back propagation method, is used to adjust the membership function and corresponding weights of the FLNN. Finally, the proposed human detection system is applied in various circumstances. The results of this study demonstrate the accuracy of the proposed method.