Previous statistical studies showed that there was a correlation between the ultralow frequency (ULF) seismo-magnetic phenomena and local seismicity in the Kakioka region, Japan. In this study, utilizing Molchan's error diagram, we evaluate whether these phenomena contain precursory information and discuss how they can be used in short-term forecasting of sizable earthquakes. In practice, for given series of precursory signals and related earthquake events, each prediction strategy is characterized by the leading time of alarms (Δ) and the length of alarm window (L). The leading time is the time length between a detected anomaly and its following alarm, and the alarm window is the duration that an alarm lasts. A modified area skill score measuring the area between actual prediction curve and random prediction line in Molchan's error diagram is used to assess the efficiency of different prediction strategies. The results indicate that predictions based on ULF magnetic data in Kakioka, Japan perform better than random prediction when Δ is around 1 week and L is less than 4 d or Δ is 13-14 d and L is less than 1 week. The optimal strategy of short-term forecasts has been established by setting Δ at 8 d and L at 1 d. The methodology proposed in this study could also be useful in evaluating the prediction policy and optimizing other kinds of measurements for short-term earthquake forecasting.