The traditional video encryption algorithm only encrypts video images, which has the problems of an extended time-consuming algorithm and poor format retention. To improve the efficiency of video encryption, this paper proposes a multi-link selective video encryption algorithm based on the Cross Coupled Map Lattices system by combining H.264/AVC video coding structure. The algorithm reduces the amount of encrypted data while ensuring encryption security to satisfy the needs of video encryption security and real-time performance. The encryption algorithm’s security and visual encryption effect are analyzed subjectively and objectively. The experimental results show that the encryption scheme has an excellent visual encryption effect and strong attack resistance, the encryption time consumption is low, and the video format remains unchanged. It can be applied to real-time video encryption occasions such as video conferences.