In this work, an accurate solar irradiance now-casting system is presented. The proposed system utilizes information from all-sky images to complement insufficient temporal and spatial resolution of satellite images and help improve prediction accuracy. Based on the history irradiance and the all-sky image features, irradiance are predicted ten to fifteen minutes ahead to allow Photovoltaics operators to schedule and allocate energy resources more effectively. To capture the relationship between clouds and the sun, an enhanced cloud detection algorithm using multi-scale neighborhood features is proposed. Then, feature points on the clouds are tracked to predict if the sun will be occluded by the clouds. According to the cloud tracking results, ramp-down events are forecasted and the predicted solar irradiance is refined. The proposed system is validated using a challenging dataset and exhibits superior performance compared with existing works.