Lumber is a primary material for the production of various types of wood products. However, many industries still carry out the lumber quality inspection process manually, relying on human sight and instinct to compare many similar objects. To streamline the inspection process, this study developed a deep learning-based surface defect detection system with a proposed 'lightning YOLOv4' model. Specifically, to improve the model's performance speed, we simplify CSPDarknet53 and path aggregation network (PANet) for the feature extraction stage of YOLOv4 by reducing the convolution layers. Moreover, we introduce the simplification technique to reduce the number of channels in CSPDarknet53 by multiplying it with the scaling coefficient. In addition, we add spatial attention module (SAM) to the structures, which can improve whole system performance on two types of lumber datasets (pine and rubber lumber). According to the experimental results, the proposed detection system improves the average precision of defect localization with the highest gap of 1.3%, as well as improves the frames per second (FPS) by 10.8 points over the baseline.