Significance: Quantifying subject-specific optical properties (OPs) including absorption and transport scattering coefficients of tissues in the human head could improve the modeling of photon propagation for the analysis of functional near-infrared spectroscopy (fNIRS) data and dosage quantification in therapeutic applications. Current methods employ diffuse approximation, which excludes a low-scattering cerebrospinal fluid compartment and causes errors.

Aim: This work aims to quantify OPs of the scalp, skull, and gray matter in vivo based on accurate Monte Carlo (MC) modeling.

Approach: Iterative curve fitting was applied to quantify tissue OPs from multidistance continuous-wave NIR reflectance spectra. An artificial neural network (ANN) was trained using MC-simulated reflectance values based on subject-specific voxel-based tissue models to replace MC simulations as the forward model in curve fitting. To efficiently generate sufficient data for training the ANN, the efficiency of MC simulations was greatly improved by white MC simulations, increasing the detectors’ acceptance angle, and building a lookup table for interpolation.

Results: The trained ANN was six orders of magnitude faster than the original MC simulations. OPs of the three tissue compartments were quantified from NIR reflectance spectra measured at the forehead of five healthy subjects and their uncertainties were estimated.

Conclusions: This work demonstrated an MC-based iterative curve fitting method to quantify subject-specific tissue OPs in-vivo, with all OPs except for scattering coefficients of scalp within the ranges reported in the literature, which could aid the modeling of photon propagation in human heads.