2.1.

Framework of Wavelength Multiplexed Fourier Ptychographic Microscopy

The comparison between the WMFPM strategy and the original FPM is shown in Fig. 1. An objective lens can be viewed as a low-pass filter in Fourier domain due to its limited NA. The original FPM method expands the NA of an objective lens using multiangle illumination. As such, the optical resolution is improved. Since an incident beam from a specific LED makes a certain lateral shift of the sample’s Fourier plane, the image sequence captured under various angular illuminations represents a temporal scanning in Fourier domain, as shown in Fig. 1(a).

Fig. 1

Extension from the original FPM to WMFPM. Specifically, (a) is the schematic of the original FPM and (b) is the schematic of WMFPM.

Distinguished from most multiplexed FPM techniques in Fourier domain,^14,21,22 we explore the potential of information multiplexing for FPM in the spectral domain. The WMFPM framework uses multicolor LEDs, each of which has a narrowband wavelength coverage to simultaneously illuminate different circular regions in a sample’s Fourier plane, as shown in Fig. 1(b). It introduces a superposition of several shifts of the sample’s Fourier plane in each measurement, resulting in a reduction of the acquisition time. For multiplexed FPM techniques in Fourier domain,^14,21,22 the inverse process to separate the multiplexed information aliased in a single image limits the multiplexing ability and requires more computation time for the optimization of the underdetermined inverse problem. To the contrary, our WMFPM strategy labels the spatial frequency information in different circular regions by separated narrowband spectral channels (or wavelengths). As the captured data are intrinsically not aliased in these nonoverlapping spectral channels, they can be easily separated through optical or imaging devices, such as diffraction gratings and RGB cameras, without the aforementioned limitations. More comparisons between previous multiplexed FPM techniques in Fourier domain^14,21,22 and our WMFPM strategy will be mentioned in Sec. 4.3.

The WMFPM framework includes the following steps. First, the number of wavelengths in use is determined, and a sequence of illumination patterns with wavelength multiplexing is designed. An illumination pattern is a combination of several angular incident lights labeled by different wavelengths using a programmable multicolor LED array. The sequence of illumination patterns should ensure that a large area in Fourier domain of the sample can be covered completely. Second, the illumination patterns are lit up one by one, and an image sensor is used to capture corresponding multicolor data. Finally, the multicolor data are separated based on different wavelengths.

An assumption of our WMFPM strategy is that the responses of many biological samples are nearly the same across different spectral channels, except for the wavelength-dependent resolution and the chromatic defocus aberration. Otherwise, the correspondence between spectral domain and Fourier domain will be no longer valid. Fortunately, a thin label-free phase object often satisfies this assumption. Previous studies^33–36 have demonstrated phase retrieval techniques by tuning multiwavelength illumination for nondispersive objects. Even a phase imaging method under a white-light source has been proposed.³⁷ These methods^33–37 assume that the specimen maintains similar transmission properties across a large bandwidth. In our strategy, the differences of wavelength-dependent resolution vary the sizes of the circular regions in the Fourier plane. The chromatic defocus aberration, caused by the different propagation speeds of different wavelengths through the optical system, can be calibrated by introducing a phase-shift factor.

2.2.

Representative Realization of Wavelength Multiplexed Fourier Ptychographic Microscopy

To validate the capability of the WMFPM, we build a proof-of-concept prototype system, called red, green, and blue (RGB) multiplexed FPM (RGB-MFPM). The schematic of the system is shown in Fig. 2(a). It is a simplified but representative realization of the aforementioned WMFPM strategy, which accomplishes threefold reduction of the acquisition time. We utilize an LED array with three colors (RBG) to realize the “line pattern” illumination shown in Fig. 2(b). An RGB camera is used to collect the data labeled by three colors simultaneously. Figure 2(c) is a photograph showing the prototype system. The three-color (RGB) illumination enables the use of a common RGB camera instead of three monochromatic cameras with narrowband color filters. The RGB-camera system just requires relatively minor modification of the off-the-shelf FPM system¹¹ without adding extra optical devices. To the contrary, a monochromatic-camera system needs more cameras and additional light paths for beam splitting and wavelength filtering. What needs to be noted here is that the intrinsic drawbacks (such as spectral cross talk, chromatic defocus aberration, and equivalent downsampling by a Bayer filter) introduced by the RGB camera can all be eliminated in our strategy. These solutions will be specifically discussed in Secs. 2.2.1, 2.2.4, and 4.1, respectively.

Fig. 2

The RGB-MFPM system: (a) is the schematic of RGB-MFPM and (b) is the $15\times 5$ line pattern illumination. Three color LEDs are placed side-by-side in each line unit; (c) is the photograph showing the setup of RGB-MFPM. The close-up is one of the line unit, each LED unit is set to a certain color (red, green, or blue); (d) is the relative luminous intensity curve of LEDs and the quantum efficiency curve of the RGB camera (AVT Prosilica GT1290C).

Each LED unit is set to a certain color, as shown in the close-up of Fig. 2(c). According to the relative luminous intensity curve of LEDs and the quantum efficiency curve of the RGB camera shown in Fig. 2(d), we find that the LEDs have no spectral overlap while the RGB camera has almost independent responses to these different color illuminations. However, the green channel relatively has a high response ($\sim 0.12$ quantum efficiency) to blue LEDs, which may introduce the cross talk and therefore decrease the quality of reconstruction. The problem can be solved by a preliminary calibration step, which will be specifically discussed in Sec. 2.2.1. Alternatively, using blue LEDs with shorter wavelengths (e.g., with 420-nm centered wavelength) might avoid the spectral cross talk.

The data acquisition process of RGB-MFPM works as follows: first, we light up the adjacent three-color (RGB) LEDs in a “line unit” simultaneously, as shown in the left top of Fig. 2(b). Second, the $15\times 5$ line patterns are sequentially lit up from left top to right bottom, as shown in Fig. 2(b). The corresponding Fourier space coverage is shown in the middle right of Fig. 2(a). Third, we use an RGB camera to capture a sequence of RGB images of the sample under each line unit illumination. An improved algorithm is designed for reconstruction. Before reconstruction, some preprocessing steps are conducted and wavelength-dependent parameters are also introduced as follows.

2.2.5.

Phase retrieval algorithm

Generally, the sample’s phase $\mathrm{\Delta }\varphi$ is defined as $2\pi ·n·d/\lambda$, where $n$ is the refractive index, $d$ is the thickness of the sample, and $\lambda$ is the wavelength. We convert the phase distribution into a thickness profile (or optical path length, $\mathrm{OPL}=n·d$) to avoid the wavelength-dependent problem.³⁵ During reconstruction, several modifications are made based on the embedded pupil function recovery algorithm.¹⁶ As shown in Fig. 3, the flowchart demonstrates the main recovery procedures of RGB-MFPM. After the multiwavelength separation and the contrast balance, the high-resolution complex field of the sample can be recovered by incorporating the wavelength-dependent parameters and phase-shift factors into the recovery algorithm. Compared with traditional FPM, we can achieve about 3 times reduction in acquisition time.

Fig. 3

The recovery procedures of RGB-MFPM. The flowchart illustrates the algorithm of RGB-MFPM based on line pattern illumination.

3.1.

Experimental Setup

For the prototype system, we use a three-color LED array with the layout of the line pattern in Fig. 2(b) to illuminate the sample, an RGB camera (AVT Prosilica GT1290C, which has the pixel size of $3.75\mu \mathrm{m}$, the pixel number of $1280(H)\times 960(V)$, and the bit depth of 12 bit for color model) to capture the raw data, and a low NA objective ($\mathrm{NA}=0.1$) for observation. The central wavelengths of LEDs are 632, 532, and 472 nm, and their bandwidths are all $\sim 20\mathrm{nm}$. The defocus abberation of the R/G/B channels is estimated in each setup, and corresponding phase-shift factors are then applied to the reconstruction algorithm. To compare with the original FPM framework, we capture 75 low-resolution RGB images each time to reconstruct the high-resolution results.

3.2.

Validation of the Proposed Approach

As shown in Fig. 4, a USAF-1951 resolution chart is imaged to show the resolution improvement based on the prototype system. Figure 4(a) shows the captured Fourier area (three circles) in one-shot of our WMFPM method compared with the single circle of the original FPM. In other words, the original method needs to scan all the overlapped circles one by one, but our method can scan three circles each time, which means 3 times improvement in the capability of optical throughput in each low-resolution measurement. One of the multicolor images and its separated one channel data are shown in Fig. 4(b). Figure 4(c) is the recovered high-resolution intensity image of RGB-MFPM, compared with the results of the original FPM in Figs. 4(d)–4(f) with monochromatic LEDs. By the way, the results [both the USAF1951 in Figs. 4(d)–4(f) and the kidney cells in Figs. 5(b)–5(d)] of traditional FPM in this article are all captured using the same RGB camera for comparison. As a result, the recovered intensity with green light has the best result because the green channel has the maximum occupation ratio in the Bayer filter of the RGB camera. To the contrary, the recovered results with red and blue illumination have more artifacts.

Fig. 4

The experimental recovery of the USAF-1951 resolution chart. (a) shows the comparison of the one-shot sampling in the Fourier plane between our method (three circles) and the original FPM (1 circle); (b) are the raw data, including the raw color image (left), the separated gray image (middle), and its close-up (right); (c) is the recovered result using RGB-MFPM, including the recovered high-resolution intensity and the close-up; and (d), (e), and (f) are the single-color (red, green, and blue, respectively) results and their close-ups using the original FPM.

Fig. 5

The experimental recovery of kidney cells. (a) are the raw data of 75 RGB images (left), one of the RGB image and its separated gray image (middle), and the close-up of the gray image (right); (b), (c), and (d) represent the recovered high-resolution phase images (middle) and their close-ups (bottom) under single color illuminations, respectively; and (e) shows the recovered phase images using RGB-MFPM (middle) and the close-up of the phase result (bottom).

To be more specific, 255 images with high-dynamic range (HDR) take about $1.332\times 225\mathrm{s}$ (about 300 s for the acquisition in total) in the original FPM setup. By comparison, RGB-MFPM needs only 75 images, which take about $1.404\times 75\mathrm{s}$ (about 100 s for the acquisition in total). The time for data acquisition can be reduced to about $1/3$ of the original method. Much more time can be reduced by combining with multiplexing methods in the Fourier plane in our framework. The resolution improvement is over 3.15 times from 4.92 to $1.56\mu \mathrm{m}$ (group 7 element 5 to group 9 element 3). In the result, not only 3 times reduction in acquisition time but also a better performance of intensity reconstruction can be achieved in our RGB-MFPM. With the comparison, we find that our method has fewer artifacts due to the higher light efficiency of simultaneous multicolor LED illumination.

3.3.

Experiments on Kidney Cells

The recovery of kidney cells and the comparison with the original FPM are shown in Fig. 5. The raw data of 75 color images of kidney cells captured by an RGB camera are shown in the left panel of Fig. 5(a); one of the low-resolution color images and its separated gray image (middle) and the close-up of the gray image (right) are also presented. Figures 5(b)–5(d) are the recovered phase results by traditional monochromatic LEDs (successively, R/G/B light). Figure 5(e) is the recovered phase image of RGB-MFPM, together with the close-up of the recovery phase. As a phase sample, little information can be seen in the captured low-resolution intensity images except for some cell outlines. The slight differences among these phase recoveries in Fig. 5 arise mainly from the discrepancy of the wavelength-dependent resolutions. The intensity-only initial guess of the object’s high-resolution Fourier plane may not provide a good initial guess for the reconstruction algorithm, which causes a few artifacts in all phase results. More robust phase reconstruction might be achieved by using a phase solution based on differential phase contrast deconvolution as a close initial guess.¹⁴ From the experimental results, our method achieves comparable performance to the original FPM methods with only $1/3$ acquisition time (75 images compared with 225 images).

4.1.

Influence of Sparse Sampling by Bayer Filter

Although the pixel size of the utilized RGB camera matches the optical resolution, the data captured by the RGB camera are actually downsampled due to the Bayer filter in front of the sensor. Fortunately, the data of FPM (also WMFPM) are highly redundant and have many overlaps in Fourier domain. This provides enough information for high-resolution recovery even based on the sparse data in spatial domain.¹⁷ The Bayer filter can be viewed as different spatial masks for R/G/B channels. The sparse sampling in spatial domain brought by these masks can be made up by the algorithm due to the large overlap (about 60%) in the frequency domain. The experimental results in Figs. 4 and 5 have verified the ability and accord with the results in the previous paper.¹⁷ In that paper, they have shown that FP algorithm is able to recover the complex image with a maximum of $\sim 70\%$ of empty pixels (random mask), which has $\sim 0.01$ root-mean-square error (RMSE). For RGB-MFPM, the sampling masks repeatedly switch among the red mask (75% empty pixels), green mask (50% empty pixels), and blue mask (75% empty pixels), with an average $\sim 66.7\%$ empty pixels.

Here, we use a numerical simulation to further validate that the FPM method can overcome the resolution loss caused by the mosaic of the Bayer filter, as shown in Fig. 6. The comparison among different sparse sampling masks has been done in terms of the quantitative measurement of the image quality. The simulation parameters are the same as those in the experiment. With the recovered phase images in Fig. 6(a) and the RMSE curve in Fig. 6(b), we find that the recovery with the RGB-MFPM mask caused by the Bayer filter can still achieve reasonable good results. The missing spatial pixels are well compensated for by the large overlap in Fourier domain. Particularly, using our RGB-MFPM mask, we can get the phase image with $\sim 0.015$ RMSE, as shown in Fig. 6(b). Figure 6(c) shows the different masks used in the simulation.

Fig. 6

The influence of sparse sampling by Bayer filter. (a) shows the phase recovery results with different masks, which indicate that the quality decreases correspondingly to the increase of empty pixel percents; (b) is the RMSE curve of different masks, and the upright triangle demonstrates that the phase recovery using our RGB-MFPM mask has a sustainable result with $\sim 0.015$ RMSE; and (c) shows the random mask, color masks, and our RGB-MFPM mask.

Moreover, in a single-camera system, such as the RGB-MFPM system, we can also use a Foveon X3 sensor^39,40 rather than a traditional RGB sensor to avoid the sparse sampling problem. The Foveon X3 sensor layers R/G/B sensors vertically rather than using a mosaic of separate detectors for three (RGB) colors.

4.2.

From Single-Camera System to Camera-Array-Based Wavelength Multiplexed Fourier Ptychographic Microscopy

As the multiplexing ability of WMFPM is equivalent to the number of different wavelengths in use, more multiwavelength LEDs can be used to get a greater reduction in acquisition time. To our knowledge, at least 13 kinds of LEDs with different wavelengths (or more precisely: narrowband nonoverlapping spectral channels) are commercially available. However, to avoid the cross talk between different channels, the RGB camera can only afford three channels within a single shot.

In addition, in the single camera system, such as RGB-MFPM, a trade-off exists between the number of wavelengths and the sampling resolution induced by the color filters (the Bayer filter here). Though the downsampled mask has little influence on the final performance of the reconstruction, it will cause some artifacts and reduce the light efficiency.

To make the WMFPM possible for high-speed gigapixel imaging in microscopy, a camera-array⁴¹-based WMFPM (CA-WMFPM) can be introduced to get around the aforementioned problems. As shown in Fig. 7, the system utilizes a multicolor (more than three colors) LED array to generate multiplexed illuminations with many different wavelengths. A diffraction grating is applied at the image plane in the detection path to separate the multiplexed information by different wavelengths. Then a camera array is employed to capture the separated data in real time. We use an LED array with 10 different wavelengths and a line-arranged illumination pattern as an example in Fig. 7. Both the multiplexing number and patterns are adjustable and related to the accessible narrowband LEDs in the market.

Fig. 7

The CA-WMFPM system. The schematic setup shown in the figure is the structure of the CA-WMFPM. The LED array has more than three different colors (10 in this system), and a special pattern (a line here for simplicity) is illuminated in each exposure, which represents a multiplexed scanning in the Fourier plane. A diffractive grating is used to separate the multiplexed information of different wavelengths in one direction, and a camera array together with a lens array is used to capture the raw data.

In addition to the much higher speed compared with the single-camera WMFPM, the CA-WMFPM system has several other benefits. The camera array uses gray cameras rather than the RGB one, so it will improve the photon efficiency and avoid the downsampling problem induced by the color filter. The contrast balance for different spectral channels can be adjusted in advance by setting different exposure times. There is also no need to do multiexposure measurements of the HDR process. Moreover, the chromatic defocus aberration can be corrected by setting the focal position of each camera independently.

The limitations of the CA-WMFPM are mainly the large size, expensive cost, and complex setup of the system. However, a camera-array-based system is still required for many high-throughput applications.³² To realize high-speed gigapixel imaging in microscopy, the most expensive and challenging problem is to design an optical setup (especially the objective lens) to afford the SBP of gigapixels. Our method gives a way to realize high-speed gigapixel imaging with a normal objective lens. Therefore, the CA-WMFPM system can greatly benefit applications requiring high-throughput observation, such as the research on cell division and drug discovery. Moreover, there is a trend in the image detector becoming smaller and cheaper, which may somewhat ease the above problem.

Furthermore, the design of an optimal pattern for multiwavelength illumination is a meaningful topic for future work with more details of numerical analysis and experimental verification. In addition, many works have applied FPM to thick samples for three-dimensional (3-D) refocusing⁴² and 3-D imaging.⁴³ Our CA-WMFPM system is also compatible with these kinds of methods. In addition, we can further lose the assumption of the similar spectral response of the sample by replacing the multiwavelength LED array with a previous proposed wavelength-scanning scheme in a narrow spectral band,⁴⁴ where a fiber-coupled tunable light source is used to realize 12 different illumination wavelengths between 480 and 513 nm.

4.3.

Comparison Among Existing Multiplexed Fourier Ptychographic Microscopy Methods and the Wavelength Multiplexed Fourier Ptychographic Microscopy strategy

Existing multiplexing techniques mainly work in Fourier domain. In these techniques,^14,21,22 the intensity information corresponding to several different frequency areas is mixed in a single image. Several methods^14,21 multiplex nonaliasing frequency information, which has no overlap in Fourier space, as shown in Fig. 8(a). The others²² multiplex aliasing frequency information, which has overlaps in Fourier space, as shown in Fig. 8(b). The ability of information multiplexing of these methods is limited. An extreme example is that we cannot realize the phase retrieval procedure with all the LEDs on simultaneously. In this case, there is no diversity to provide phase contrast. The multiplexing state commonly used is 8 times in the nonoverlapping case^14,21 and twice in the overlapping case.²²

Fig. 8

Illumination patterns of different multiplexed FP approaches. (a) (top) Random pattern multiplexed coded illumination,²¹ and four randomly chosen LEDs are turned on for each measurement; (bottom) Fourier coverage of the sample’s Fourier space for each LED pattern; (b) (top) state-multiplexed (or adjacent position multiplexed) illumination²² and (bottom) its Fourier coverage; (c) (top) line pattern illumination of the WMFPM strategy, and a line unit of adjacent three color (RGB) LEDs is turned on each time; (bottom) its Fourier coverage. The center unshaded circle (in its Fourier coverage) represents the NA of the objective lens.

Distinguished from existing methods working in this way, our WMFPM strategy labels different frequency information by separated narrowband spectral channels, as shown in Fig. 8(c). The wavelength multiplexing intrinsically bypasses the limitation of information multiplexing in Fourier domain. Due to the narrowband of the commercial LEDs in use, our RGB-MFPM system and the recovery algorithm are both easy to implement. The most attractive advantage of our method is its compatibility with most existing methods. Although the photo-sensitivity of the RGB sensor is lower than a gray camera in the prototype system, we can use the CA-WMFPM system or the Foveon X3 sensor-based WMFPM system to eliminate this problem in the future.

In addition, the WMFPM strategy is different from the spectral multiplexing technique.²² Though both use the LED array with multiple wavelengths to illuminate the sample from different incident angles, our method focuses on improving the acquisition speed by wavelength multiplexing. They try to use a gray camera (rather than an RGB camera or color filters) and their improved phase retrieval algorithm (the state-multiplexed FP algorithm) to achieve multispectral imaging.