With the continuous improvement in the accuracy of steganalysis based on convolutional neural networks (CNNs), the network scale has shown explosive growth. Consequently, CNNs have a high demand for hardware resources and time-consuming training. To reduce the number of CNN parameters and improve the efficiency of steganalysis, we propose a lightweight steganalysis CNN called W-Net. The proposed W-Net first uses grouped convolution and channel shuffling units to extract noise residuals, strengthen the information exchange between groups, and improve feature extraction. In addition, the depth-wise separable convolution is applied to obtain different channels and spaces. The fusion of position information achieves the effect of conventional convolution while reducing the number of network parameters. We verified the effect of activation functions on steganalysis accuracy through experiments. In addition, the proposed W-Net can detect the steganographic data from the S-UNIWARAD spatial steganography algorithm with an embedding rate of 0.4 bpp. Compared with Xu-Net and Zhu-Net, the proposed W-Net improves the detection accuracy by 12.70% and 6.38%, respectively.