To solve the problem of poor face recognition performance in unrestricted environments. A face recognition algorithm based on improved residual IR-ResNet-SE is designed. Firstly, the IR structure is added to the 34-layer residual network to reduce the variability of different features; Secondly, we add the channel attention module to increase the weight of important channel features; Finally, the Arcface loss function is used to improve the classification ability of the model. The LFW, AgeDB, and AR datasets reflect unrestricted factors such as pose, age, expression, occlusion, and illumination. The algorithm proposed in this paper is experimented on these three datasets. The experimental results show that the IR-ResNet-SE algorithm proposed in this paper can achieve 99.74% accuracy in the dataset LFW. And it has excellent robustness in face recognition under unrestricted conditions.