In this study, we evaluated the effect of push notifications on mobile app engagement. Over the duration of a month, we collected data from 12,071 online users of an English learning app named 'English Practice' which can be freely installed from the Google Play store. Four criteria of app engagement used by us were as follows: session count, session length, time consumption and retention. In addition, we examined the frequency of sending notifications, how to display notifications and the use of notifications on wearable devices. The results show that push notifications with regards to incoming messages can increase learner engagement most. On the other hand, abusing push notifications (e.g. if they are too frequent) result in the opposite effect. Based on the results, we suggest that educators or developers focus on an increased integration of social communication tools inside apps.