With the improvement of people’s living standards, the increase in car ownership has caused more urban traffic accidents, so how to reduce the incidence of traffic accidents has become a current hot issue. In this paper, based on the problem of traffic accidents caused by drivers’ poor emotional state while driving, we investigate the research of driver emotional state recognition based on visual evoked potentials. Firstly, a reasonable experimental paradigm is designed to collect EEG signals by visual evocation; After that, a ReliefF-PSO algorithm is proposed, and the parameters of each channel feature subset are optimized by using the Particle Swarm Optimization (PSO) algorithm. The Broad Learning System (BLS) was used to evaluate the approximate entropy and wavelet packet feature extraction algorithms. The results demonstrate that the BLS incremental learning algorithm can well recognize the EEG signals in different emotional states. The EEG signal processing method proposed in this paper is expected to provide a new technical means for cognitive science research on emotional states and the Emotional Brain–Computer Interface (E-BCI) system.