In this study, we presented a novel summarization method for generating sports video abstracts, which utilized motion entropy analysis and mutual information. Both of them are based on an attentive model. In order to capture and detect significant segments among a video, we exploited saliency maps by calculating color contrast, intensity contrast, and orientation contrast of frames. In the next step, motion vectors between maps were computed and converted into salient motion entropy. Meanwhile, a new algorithm based on mutual information was proposed to improve the smoothness problem when we selected boundaries of segments. The experiments showed that our proposed algorithm could not only detect highlights effectively but also generate smooth playable clips. Compared with the traditional approaches, our system improved the precision by 7.6% and enhanced smoothness by 1.2, which also verified feasibility of our system.