Smoke detection technology is of great significance for early fire warning. Aiming at the problems of low precision and slow speed in complex scenes and the inability to frame the smoke area quickly, a video smoke detection method based on lightweight YOLOv4 is proposed. Firstly, efficient channel attention (ECA)-bneck is introduced into the backbone network to extract image features, avoid the interference of redundant background, and enhance the detection performance. Then a 1 × 1 convolution is added to form the backward residual structure and strengthen the ability of learning features to improve the detection precision. Finally, standard convolution is replaced by depthwise separable convolution to compress the amount of network parameters, and reduces the number of CSP modules in the backbone to improve the detection speed; Experiments show that the proposed algorithm has strong adaptability in complex scenes, the model detection accuracy reaches 98.2%, and the detection frame rate is increased by 9.25 frames per second on average.