Channeled spectropolarimetry (CSP) is a spectral modulation technique that furnishes snapshot polarimetric measurements. Among the different spectropolarimetric methods, fiber-based techniques are fundamental requirements in specific biomedical procedures, such as endoscopy. However, the optical fiber exhibits high sensitivity to temperature changes, which subsequently incurs errors in the polarimetric reconstructions. We present a calibration technique for fiber-based CSP that leverages phase-shifting interferometry (PSI) to acquire and demodulate the fiber’s retardation phase accurately. This technique provides high robustness against temperature variations and improved polarimetric reconstruction performance. Experimental results demonstrated that the PSI calibrations offer higher stability and accuracy, at different optical fiber temperatures, as compared to reference-beam calibrations.