Clustering analysis plays an important role in finding natural structures of datasets. It is widely used in many areas, such as data mining, pattern recognition and image processing. By generating a set of nested partitions of datasets, hierarchical clustering algorithms provide more information than partitional clustering algorithms. However, due to the generated clustering hierarchies are too complex to analyze, many existing hierarchical clustering algorithms cannot properly process many non-spherical and overlapping datasets. Clustering validity index is the key technique for forming the optimal clustering partitions and evaluating the clustering results generated by clustering algorithms. However, many existing clustering validity indexes suffer from instability and narrow range of applications. Aiming at these problems, the traditional Average-Linkage hierarchical clustering algorithm is firstly improved for better processing the above irregular datasets. Then, a new clustering validity index (MSTI) is defined to stably and effectively evaluate the clustering results of the improved algorithm. Finally, the new algorithm for determining the optimal clustering number is designed by leveraging the improved Average-Linkage hierarchical clustering algorithm and the new MSTI. Experimental results have shown that our new clustering method is stable, accurate and efficient in processing many kinds of datasets.