Rain streak in an image can degrade the human vision, as well as the image's quality. However, the rain removal of a single image is a challenging problem, because the rain is moving fast and may become torrential. In this paper, a single image rain removal process based on the non-negative matrix factorization is proposed. In the proposed method, the rain image is decomposed into a low-frequency part and a high-frequency part by a Gaussian filter. Therefore, the rain component, which is usually in the middle frequency, could be discarded in high and low frequency domains. In this paper, the non-negative matrix factorization (NMF) method is applied to deal with the rain streak in the low frequency; while in the high frequency part, the concept of Canny edge detection and block copy strategy are utilized separately to remove the rain hidden in high frequency and improve the image quality. By comparing with the state-of-the-art approaches, our proposed method does not need the extra image database to train the desirable dictionary, but still reaches similar results.