The rise of convolutional neural network (CNN) has greatly improved the disadvantage of the traditional video classification method. However, in the pre-training process of classification network, often due to over-fitting, gradient disappearance and other factors lead to training data convergence performance is poor, thus would affect the accuracy of the classifier. Aiming at the problem of network optimization, this paper proposes an algorithm to combine improved adaptive genetic algorithm (IAGA) with deep convolution neural network (DCNN) classifier. The weighting of the network is initialized by the IAGA algorithm, and the weight is corrected by combining the gradient descent (GD) algorithm. Finally, the fusion of global feature extracted by the network is input into the extreme learning machine (ELM) for classification. The results of the news video classification show that the algorithm can combine the global search ability of IAGA with the local optimization ability of gradient descent algorithm to improve the accuracy of the training network with less parameters, and the average classification accuracy rate can reach 90.03%. Compared with the three existing algorithms, the algorithm has higher classification accuracy. Compared with the four kinds of network pretraining methods, the algorithm presented in this article is more dominant.