Facial expression recognition from a single image is challenging due to the subtle differences between the various expression types. Considering that facial expressions are not only expressed by the appearance of individual facial organs, but also highly correlated with the local features of these organs, we propose a new network architecture, namely the Pooling First Bilinear Convolutional Neural Network (PF-BCNN). The network uses two simplified CNNs to extract facial appearance features, then integrates them using a modified bilinear approach, which encodes the interaction of local features. In addition, we use a multi-task learning model to simultaneously train and predict two typical outputs for emotion recognition, namely action units (AU) and emotion types (ET). Experimental results show that the method achieves the most advanced performance, especially the recognition of emotion types.