Vehicle detection is one of the key techniques of intelligent transportation system with high requirements for accuracy and real-time. However, the existing algorithms suffer from the contradiction between detection speed and detection accuracy, and weak generalization ability. To address these issues, an improved vehicle detection algorithm is presented based on the You Only Look Once (YOLO). On the one hand, an efficient feature extraction network is restructured to speed up the feature transfer of the object, and reuse the feature information extracted from the input image. On the other hand, considering that the fewer pixels are occupied for the smaller objects, a novel feature fusion network is designed to fuse the semantic information and representation information extracted by different depth feature extraction layers, and ultimately improve the detection accuracy of small and medium objects. Experiment results indicate that the mean Average Precision (mAP) of the proposed algorithm is up to 93.87%, which is 11.51%, 18.56% and 20.42% higher than that of YOLOv3, CornerNet, and Faster R-CNN, respectively. Furthermore, its detection speed can meet the real-time requirement of practical application basically with 49.45 frames per second.