Image composition constructs a new image by cutting out a part of one image and then pasting it on another image. However, the quality of the new image is generally low due to inconsistency between the two parts. To overcome this, image harmonization aims to obtain realistic composite images by adjusting the color, illumination, and visual style of the foreground, to make it compatible with the background. Nevertheless, previous image harmonization techniques mostly concentrate on learning a mapping network from composite image to real image, and they ignore the significant role of background visual styles as a mapping guide. In this task, we consider image harmonization as a style transfer problem. Specifically, we take foreground as the content image and background as the style image, to transform the foreground’s style through background’s style features. To do so, we propose a unique self-attention-based module to learn the mapping between foreground features and background features using a modified self-attention mechanism. The proposed module can calculate the degree of correlation between the foreground and background according to the semantic distribution and also aligns the channel-wise mean and variance of the foreground features with that of the background features in the meantime. To comprehensively investigate the effectiveness of proposed module, we perform multiple experiments and ablation studies on the existing benchmark dataset iHarmony4. The experimental results prove that our module is more effective compared to the baseline in our task, and our harmonized image looks much closer to the real image.