We propose a method to reduce the speckle noise associated with phase, retrieved using off-axis digital holography with transport of intensity. Multiple decorrelated holograms are obtained by simultaneously varying the linear polarization state of the object and the reference beams, which are applied in the combined technique to retrieve the phase. Reduction in speckle noise is verified through statistical analysis of the intensity and phase contrasts.