In the human nervous system, sensory inputs are coded in a sparse manner where only small numbers of neurons are active at a given time, thus the sparse coding is reasonable to be as a plausible model of the auditory cortex. In this paper, we propose a biologically inspired feature extraction method for speaker verification based on sparse coding. When encoding the speech data using sparse coding model, the learned dictionary has the similar characteristics with simple cell receptive fields of auditory neurons and the sparse coding coefficients simulate the response of the auditory cortex neuron. Moreover, every dictionary is learned from every speaker training sample, so that it has more individual information of the speaker and is useful for discriminating different speakers with less dictionary atoms. And based on human auditory masking effect, a neuron which performs a Max Pooling operation on the pooled inputs responds to the strongest one of its inputs and inhibits other weaker inputs. The robustness of the proposed method is better in terms of a strategy to represent natural sounds. The experimental results show that the proposed method outperforms the baseline system on two typical corpuses.