Although a Differentiated Service (DiffServ) network provides the Class of Service (CoS) delivery quality to real-time video data, the received picture quality may still be seriously degraded if it lacks an effective significance classification scheme for video packets. Moreover, even with a significance classification scheme, the performance could still be limited due to the use of a fixed set of parameters for videos with various coding characteristics. To solve above problems, this paper proposes an Adaptive Significance Classification mechanism in Temporal and Spatial domains (ASC-TS) for video data over DiffServ networks. ASC-TS determines the video packet significance simultaneously in temporal and spatial domains. From the temporal domain, ASC-TS evaluates the packet significance based on the estimated error propagation if a packet is lost. From the spatial domain, ASC-TS computes the packet significance based on the content complexity belonging to a packet. Moreover, ASC-TS is adapted to various video sequences with a self-learning mechanism. Compared with traditional significance classification schemes, simulation results show that the proposed mechanism can significantly improve the accuracy of signification determination up to 15% and effectively improve the received video quality up to 0.7dB in PSNR.