In the IC design process, the test process is the main factor of production cost. Existing tests rely on additional analysis of testing result data by the engineer to determine the status of the process. Thus it could take an additional amount of time and cannot make adjustments of the process immediately. Wafer map defect recognition is an import part of semiconductor. There is lots of information in wafer maps which can quickly help engineers to identify what failure type it is. The location of the error point is graphical represented and the relationship of these points contains the feature of this map. In this paper, we proposed a classifier with reduced-weight architecture based on depthwise separable convolutions. The entire work is verified by using the real-world wafer map dataset (WM-811K). The accuracy is 96.63% in test set.