A back-propagation (BP) neural network consists of an input layer, one or more hidden layers and an output layer. An input vector is presented to the network, it is propagated forward through the network, layer by layer, until it reaches the output layer. The output of the network is then compared to the desired output, using a loss function, The BP neural network easily falls into a local extreme values and the slow convergence, during the Classification of Speech using it. This experiment selected four types of speech: Guzheng, folk songs, rock and pop, the categories of speech were converted to the matrix Z1×4. A BP neural network structure of 24-7-4 was established. The BP neural network with 24 dimensional feature data as input and four categories as output was used. A new method is put forward to optimize weights and threshold of BP neural network using PSO. results of which were analysed and compared with that BP neural network. The PSO-BP neural network not only enhances the accuracy, but also reduces the computation.