Indonesia is one of the countries with the largest industry of crude palm oil (CPO) in the world. During 2013-2017, the growth of the area of oil palm plantations in Indonesia decreased -0.52%, the decline is expected not to affect the amount of CPO production. One of the things that affect CPO production is the primary raw material availability of palm oil fresh fruit bunches (FFB). Raw material requirements can be predicted by several forecasting methods, but the methods only predict the raw material requirements FFB, not the availability. The development of deep learning eases humans in doing things. Deep learning can be used to calculate FFB automatically using the faster R-CNN algorithm. This study presented a system of automatic detection and calculation of FFB. The evaluation is carried out by comparing 4 network architectures; resnet inception V2, inception V2, resnet 50, and resnet 101. The results of this study indicate success in calculating FFB. The success is indicated by the results of evaluating the four network models with the average F1 scores above 80%.