The algorithm used is based on the Faster R-CNN network. It uses resnet101_vd as the backbone network to extract navel orange image features. It minimizes the error between the inferred bounding box and the actual labeled bounding box through the RPN network and the ROI Pooling layer and non-maximum suppression method. The AP during model training reached 92.34% on sunny days, 96.84% on cloudy days, and 90.05% on foggy days. When evaluating the model, it reached 92.34% on sunny days, 96.89% on cloudy days, and 89.2% on foggy days. The processing time of this model is 63.84fps on sunny days, 68.6fps on cloudy days, and 56.29fps on foggy days. It meets the requirements of rapid and accurate identification in actual picking. In addition, it compares this model with the Faster RCNN with vgg16 as the backbone network and YOLO-v4 models. It effectively improves detection accuracy and speed. Moreover, it reduces the number of false detection and missed detection of navel orange detection. It improves position accuracy. This paper realizes the efficient detection of ripe navel orange fruits on trees in various weather.