Significance: In multiphoton microscopy, two-photon excited fluorescence (TPEF) spectra carry valuable information on morphological and functional biological features. For measuring these biomarkers, separation of different parts of the fluorescence spectrum into channels is typically achieved by the use of optical band pass filters. However, spectra from different biomarkers can be unknown or overlapping, creating a crosstalk in between the channels. Previously, establishing these channels relied on prior knowledge or heuristic testing.

Aim: The presented method aims to provide spectral bands with optimal separation between groups of specimens expressing different biomarkers.

Approach: We have developed a system capable of resolving TPEF with high spectral resolution for the characterization of biomarkers. In addition, an algorithm is created to simulate and optimize optical band pass filters for fluorescence detection channels. To demonstrate the potential improvements in cell and tissue classification using these optimized channels, we recorded spectrally resolved images of cancerous (HT29) and normal epithelial colon cells (FHC), cultivated in 2D layers and in 3D to form spheroids. To provide an example of an application, we relate the results with the widely used redox ratio.

Results: We show that in the case of two detection channels, our system and algorithm enable the selection of optimized band pass filters without the need of knowing involved fluorophores. An improvement of 31,5% in separating different 2D cell cultures is achieved, compared to using established spectral bands that assume NAD(P)H and FAD as main contributors of autofluorescence. The compromise is a reduced SNR in the images.

Conclusions: We show that the presented method has the ability to improve imaging contrast and can be used to tailor a given label-free optical imaging system using optical band pass filters targeting a specific biomarker or application.