This paper proposes a small sample dataset, regarded as the specific-task dataset in deep transfer learning, in order to improve the performance of transfer learning. Each single-frame image is divided into three top-down sub-images in the dataset. Object features, such as sharp and texture information, are enhanced to capture the features of each class in the target domain, to reduce the loss of the network function caused by one-way transfer. Therefore, the network can efficiently learn more accurate information, and help to reduce softmax cross-entropy loss and generalization error. In addition, we explore the knowledge transfer among different attributes, such as photos to paintings, and proposes a two-phase training method to improve the loss function and its generalization error. From the experimental results, transfer learning between different attributes is not as effective as the proposed two-phase training used in knowledge transfer. Especially in VGG-11 with batch normalization (BN), our method can effectively improve the accuracy of 11.78 % and reduce softmax cross-entropy loss by 1.283 and generalization error by 1.496, respectively. Therefore, the multi-scale small sample dataset can improve the information loss caused by one-way transfer, thereby improving the overall network performance, and making its prediction closer to human recognition results.