The advent of convolutional neural networks has been driving the rapid development of image super-resolution (SR) tasks. Existing works, however, tend to devise deeper and wider networks to boost accuracy, leading to huge model sizes and computation costs. In addition, they also ignore the effect of frequency domain information on image restoration. To address these challenges, we propose a simple and effective frequency-based attention network, comprising a series of frequency-domain enhancement modules (FDEMs), for accurate image SR. Each FDEM integrates a two-dimensional discrete wavelet transform, progressive frequency enhancement module (PFEM), and frequency aggregation module (FAM). Specifically, DWT first decomposes the features into high-frequency and low-frequency parts, and then the low-frequency branch is processed and gradually added to the high-frequency branch to realize the interaction of frequency information in PFEM. Finally, frequency representations from the PFEM are upsampled to the FAM to further achieve information enhancement in a specific space. Extensive experiments indicate that our method is efficient in reconstruction accuracy with less model capacity, exceeding most existing lightweight SR networks.