In order to fulfill the real-time pedestrian detection system, it is difficult to balance the higher detection rate and greater detection speed at the same time. Therefore, this paper proposed an Adaptive Neighborhood Selection-Deformable Part Model (ANS-DPM) as a novel pedestrian feature operator to solve this problem. ANS-DPM adopts the adaptive feature zones to extract the pedestrian features based on the relationship between features ex-traction score with the experience threshold to decrease the whole calculation and upgrate. Meanwhile, the inner-layer and inter-layer constraint are introduced into the ANS-DPM to deal with the model of robustness. Finally, ANS-DPM with Latent-SVM constructs the pedestrian detection system and experiment result shows that the ANS-DPM can improve the feature rate, while the pedestrian detection system can detect 30 fps to satisfy the real-time detection system.