Although many computer vision-based object detection techniques have evolved in the past decade, it suffers from inconsistent detection accuracy, especially for multiclass classification problems. This paper proposed an approach using the Single Stage Deep Transfer Learning model (SS-DTLM) for multiclass apparel detection using a customized YoloV3 algorithm by adapting 3-level Spatial pyramid pooling (SPP), a multi-scale image feature extractor for faster and reasonable apparel detection and classification. This approach produced a reasonable Mean Average Precision (mAP), reliable object detection, and classification. Our model trained and tested on Open Images Dataset (OIDV4) with six object classes and Custom built Apparel Dataset with five object classes of apparels. Finally, experimental results compared with baseline Yolov3 and Yolov3-Tiny algorithms. Further, this paper also emphasized the detected image's various color spaces using SS-DTLM by applying the K-Means clustering algorithm for further analysis.