Volume Bragg grating (VBG) can be used in laser resonator to control the transverse distribution due to its excellent Bragg selectivity. The coupled-wave theory is used to analyze the angular selectivity of VBG, and the output modes of the volume Bragg resonator are simulated with the fast Fourier transform (FFT) method and the coupled-wave theory. In this paper, the volume Bragg grating is inserted into a plane-parallel resonator, the intensity distribution and diffraction losses for the mode of TEM
00, TEM
10 and TEM
20 are simulated, and the loss difference for different modes at different angular selectivity of VBGs are discussed. At the VBG angular selectivity of 3 mrad, the diffraction loss for fundamental mode is of 6.3%, while the diffraction loss for TEM
10 and TEM
20 mode are 19.8% and 32.7%, respectively. Therefore, TEM
10 and TEM
20 can be easily suppressed if the gain is between 6.3% and 19.8%, and a fundamental transverse mode can be obtained.
Besides, the simulation results show that the intensity distribution profile of the transverse modes become smooth with the insertion of VBG, but the diffraction losses of transverse modes are increasing, and the diffraction loss increases with the order number of transverse modes increasing. Moreover, the loss difference between modes is getting large under the effect of VBG. The high loss difference between different modes is good for transverse mode selection, and VBG with reasonable angular selectivity in laser resonator will force the multi-mode to operate in a single transverse mode, which may has potential applications in lasers.