Deep learning technologies based on Convolution Neural Networks (CNN) have been widely used in fabric defect detection. On-site CNN model training and defect detection offer several desirable properties for the fabric manufactures, such as better data security and less connectivity requirements, when compared with the on-cloud training approach. However, computers installed at the manufacturing site are usually industrial computers with limited computing power, which are not able to run many effective CNN models. A lightweight CNN model should be used in this scenario, in order to find a balance point among defect detection, efficiency, memory consumption and model training time. This paper presents a lightweight CNN-based architecture for fabric defect detection. Compared with VGG16, MobileNetV2, EfficientNet, and DenseNet as state-of-the-art architectures, the proposed architecture, namely FN-Net, can perform training 3 to 33 times as fast as these architectures with less graphics processing unit and memory consumption. With adaptive class determination, FN-Net has an average F1 score 0.86, while VGG16 and EfficientNet as the best and the worst among the baseline models have 0.81 and 0.50, respectively.