The video streaming applications are full of potentials in the IP dual stack network that supports IPv4 and IPv6 protocols simultaneously. However, the significances of video packets belonged to various video sequences are different. An equal error protection to all video packets in the IP network will degrade the video quality significantly. This paper proposes an Adaptive Significance Determination Mechanism in Temporal and Spatial domains (ASDM-TS) for H.264 videos over IP dual stack network with DiffServ model. ASDM-TS determines the video packet significance simultaneously in temporal and spatial domains. From the temporal domain, ASDM-TS evaluates the packet significance based on the estimated error propagation if a packet is lost. From the spatial domain, ASDM-TS computes the packet significance based on the content complexity belonging to a packet. Moreover, ASDM-TS is adaptive to various video sequences with a self-learning method. Compared with traditional schemes, simulation results show that the proposed scheme significantly improves the accuracy of signification determination up to 15% and effectively improves the received video quality up to 0.7dB in PSNR.