2.1.

Design Framework

Traditional catadioptric optical designs (e.g., Cassegrain reflector) utilize the mirror to provide most of the focusing power, and the refractive elements were used for aberration control.^{38, 39, 40} In contrast, catadioptric panoramic designs utilize the mirrors for the redirection of the FOV of the refractive group of elements. Figure 2 illustrates a flowchart of our design framework. The SVP theory³³ was used to identify potential mirror configurations. These potential mirror configurations were imported into separate sequential ray trace simulations to determine the FOV and NA. The most practical mirror configuration served as the initial mirror design for the subsequent design phases. The lens design phase began by the integration of the initial mirror design with a preselected lens template. Lens optimization was then performed iteratively until the radial-view design criteria were satisfied. The refractive elements in the radial-view were then imported into a new separate simulation, which was the forward-view simulation in the subsequent design phases. Additional refractive elements were inserted into the forward-view simulation, and described in Sec. 2.3.3 with more details. Lens optimization was then performed over these additional refractive elements. The most prioritized optimization criterion for the forward-view lens design was to match the back focus of its imaging plane to that of the radial-view design. This was a challenging task that required multiple readjustments of both the radial-view and forward-view lens design. The optimization process concluded when all specified criteria had been satisfied and manual refinement was carried out. After revising minor technical details that were not described in the optimization criteria, the design was deemed to be complete.

Fig. 2

Flow chart of the proposed design framework.

The design criteria for the proposed design are summarized in Table 1. The image quality was evaluated using spot diagrams and modulation transfer functions (MTF). The targeted image detector is a hyperspectral CMOS device that is currently under development. The pixel pitch of this image detector was estimated to be approximately 5 × 5 μm. For efficient light collection onto one pixel, the upper bound of the root mean square (rms) spot size was set to 5 μm. This pixel pitch suggested a spatial sampling frequency of 100 lp/mm. From the Nyquist–Shannon sampling theorem, the analog optical signal should be roughly limited to spatial frequencies below 200 lp/mm to avoid aliasing. To determine the MTF cut-off, the minimum acceptable modulation score was taken to be 50% modulation. The image dimensions were taken from the surveyed endoscope objective patent designs^{17, 18, 19, 20, 21} for ease of comparison. The lower bound on the edge thickness was set to 0.6 mm, as per the recommendations from optical fabrication shops. The minimum radius of curvature and the image dimensions were set to be similar to that of the surveyed patents.^{17, 18, 19, 20, 21} FOV, DOF, and entrance pupil radius (EPR) were parameters in the optimization objective function to be maximized.

Fig. 11

The unprocessed images, (a) 0.5 lp/mm test pattern and (b) business card, are manipulated using precomputed mappings relationships to produce (c) and (d), respectively.

Table 1

Design constraints for the proposed simulation design.

2.2.

Mirror Design

To determine a suitable initial mirror design, the expected FOV from various SVP constrained mirror surfaces were computed. From SVP theory, the solution to the family of single mirror surfaces that satisfy the SVP condition is as follows:³³

Folded mirror configurations were investigated, under the hypothesis that aberrations may be compensated if mirrors with opposite concavity are used. The coma and astigmatism aberrations introduced by the convex mirror were indeed reduced by the concave mirror. The folded mirror configurations shown in Fig. 3 were simulated using commercial sequential ray tracers. The ray angle incident on an optical surface has a significant impact on the resultant aberration; therefore, small tolerance deviations on the mirror curvature may induce significant aberration. Although the hyperbolic-ellipsoid configuration in Fig. 3a was a valid solution, the parabolic-parabolic configuration in Fig. 3b was chosen because the chief rays were practically parallel to each other in the region between the two mirrors. This phenomenon follows from the nature of a parabolic reflector; incident rays from different field angles that were aimed at the focus of a parabolic mirror would all be reflected in a parallel direction. This decoupled two mirrors: focus adjustments from one mirror would not require significant readjustments from the other mirror. This decoupling effect provided a more intuitive design flow, and increased the resilience of the design to tolerance-induced performance degradation.

Fig. 3

Ray trace layout with the aperture stop located at the focus of the convex mirror. The two candidate mirror configurations were: (a) convex hyperbolic and concave ellipsoidal mirrors, and (b) parabolic mirrors. The different colored ray bundles are from different object points within the FOV. The unit of the scale bar is millimeters, i.e., 1.86 mm in (a) and 2.27 mm in (b).

2.3.

Lens Design

2.3.1.

Optimization considerations

The paraxial regions of both mirrors were removed for the forward-view ray paths, and the peripheral regions of both mirrors were used for the radial-view ray paths. The cut-off radius of both mirrors was optimized for maximum radial FOV without significant vignetting of the blue and yellow ray bundles near the edge of the FOV in Fig. 4. With reference to Fig. 4, the blue ray bundle is the lowest bundle from the object, and the yellow ray bundle is the highest bundle from the object. The blue bundle is imaged furthest away from the optical axis, and the yellow bundle is imaged closest to the optical axis. To maximize the amount of area used on the image detector, additional optimization constraints were set on the image heights of selected field points across the FOV. The wavelengths 486.1, 587.6, and 656.3 nm had been used with equal weights for the lens optimization routines. It was assumed that UV LEDs would be used as the excitation source for radial-view AF imaging, and white light LEDs would be used for the radial-view and forward-view white light imaging. In a fully functional florescence imaging system, the UV and the shorter wavelengths of the visible spectrum would be attenuated by other filter components, thus they were not considered in the lens design.

Fig. 4

Side-view optical layout and representative ray traces. The shaded mirror regions would be hollow for light to pass through. For practical lens edge thickness handling, the clear aperture of the last doublet may need to be smaller than specified here, or the FOV would need to be reduced.

2.3.3.

Forward-view lens design process

The lenses responsible for the forward-view light path are shown in Fig. 5. They include the radial-view lenses and additional lenses placed in front of the mirrors. These additional lenses are referred to as the forward-view lens group. One challenge for the forward-view lens design was the matching of the radial-view and forward-view back focuses. Another challenge was the design of a wide FOV for the forward view with a fixed field stop aperture size. The forward-view field stop was the central opening of the parabolic mirrors, and a change in its diameter would compromise the radial FOV. The lens system in Fig. 5 resembled a reverse telephoto lens form, where the first Zeonex™ plastics lens acted as the optical dome of the endoscope. The first two lenses compressed the field rays to implement a moderate FOV. The third lens helped to correct the field curvature. The fourth element had an aspherical front surface that corrected other primary aberrations, and a back surface that controlled the back focus.

Fig. 5

Forward view optical layout and representative ray races. The radial view lenses are the same lenses used in the second forward view lens group (right box). The image plane of the radial view is the same as that of the forward view.

3.1.

Design Evaluation

The simulated performance comparison between the proposed design and the reference forward-viewing design is summarized in Table 2. The reference design was taken to be the patent design²¹ that was used as the starting lens template for the radial-view lenses. The proposed radial-view design had larger NA in the object space and EPR. The largest rms spot sizes were ∼4.0 and ∼1.9 μm for the radial and forward views, respectively. This was beneficial to light collection since most of the rays from the object would be concentrated on an area smaller than the specified pixel pitch of 5 μm in Table 1. The spot diagram in Fig. 6a for the forward-view design did not have similar spot sizes for each of the three color-coded design wavelengths. This indicates a chromatic shift in the focus position. The spot sizes across the different wavelengths were similar for the radial-view optics, as shown in the spot diagram in Fig. 6b.

Fig. 6

Image surface spot diagram for (a) forward-view optics and (b) radial-view optics. The black circle is the simulated Airy Disk radius. The colors correspond to the wavelength legend at the top right of this figure; all units are in micrometers. RMS radius is the root mean square spot size, and GEO radius is the distance from the centroid to the furthest ray intersection with the imaging surface. IMA describes the location of the centroid on the imaging surface. OBJ describes the location of the field points in terms of field angles. The forward optics has smaller spot sizes than the sidewall optics.

Table 2

Simulation results: comparison of the template lens patent with the proposed system.

The simulated MTF was calculated from the modulus of the optical transfer function by the commercial sequential ray tracer. It can be seen from Fig. 7a that the simulated nominal modulation for the forward-view design dropped below 50% at spatial frequencies higher than 160 lp/mm. In Fig. 7b, the simulated nominal modulation for the radial-view design dropped below 50% at spatial frequencies higher than 120 lp/mm. These nominal simulated MTF scores were below the upper bound specified in Table 1, so aliasing should not occur. Due to the presence of fabrication and alignment tolerances,³⁹ the actual performance of a lens design is expected to be lower. As evident from the similar spot diagrams across the design wavelengths, chromatic aberration was not a significant problem for the radial-view design.

Fig. 7

Simulated MTF for (a) forward-view optics and (b) radial-view optics. T is tangential MTF curve at the specified field point. S is sagittal MTF curve at the specified field point. Field points are specified in terms of field angle (measured in degrees). DIFF. LIMIT is the theoretical diffraction limited MTF curve.

3.2.

Irradiance Simulation

Commercial illumination optical design packages are adept at the computation of radiometric quantities at specified detector surfaces. These packages often employ nonsequential ray tracing algorithms that generate a high number of rays that are not required to pass through the specified optical surfaces in sequential order (e.g., no pre-defined ray path). In exchange for computational time, this type of ray tracing allows stray light contributions to the irradiance on the image detector to be accounted for. Light intensity contributions from stray light incident on the image detector could be considered as noise, and could contribute to form ghost images.³⁹ The LightTools^TM commercial illumination design package is based on nonsequential ray tracing, and it was used in this work to provide a relative irradiance collection comparison between the designs. The endoscope sheath material was a poly(methylmethacrylate) (PMMA) tube with a thickness of 1 mm. Another set of simulations were performed for the proposed design without the endoscope sheath. This was to compare the effect when different sheath materials were used.

The light reflected or emitted (such as in the case of AF imaging) from the collapsed GI wall were modeled using point sources placed across the FOV of the radial-view design. Due to the different FOV orientation of the proposed radial-view design and forward-viewing reference design, the source locations for both simulations had to be different. In Fig. 1c, the locations of the simulation field points are marked for the radial-view design and the reference design. The cylindrical endoscope sheath had an outer radius of 5.5 mm, and the linear FOV projected onto this surface was 3.76 to 6.9 mm (Table 3). This corresponded to an imaging length of 3.14 mm along the cylindrical surface. The point sources were set to have 100 W evenly distributed over its entire projection sphere. The large power distribution of 100 W was chosen to avoid the computation of extremely small numbers, therefore, this measure of light collection served only as a relative comparison. The irradiance plot in Fig. 8 was computed using samples at the surface of the image detector. Each bin was set to 95 × 95 μm for practical computation, and to increase the signal-to-noise ratio.

Fig. 8

Irradiance plot of the radial-view design.

Table 3

Radiometry simulation results.

4.1.

Prototype Design Specification

There were three new design considerations for the prototype; 1. to achieve convenient manufacturability, 2. to provide integration with a commercial CMOS camera, and 3. similarity to the proposed simulation design described in Sec. 3. Table 4 summarizes the evaluation and fabrication specifications used for the prototype design. The CMOS camera had an image dimension of 6.66 mm × 5.32 mm, which was different from the image dimension of 2 mm × 2 mm used in the proposed simulation design. Constraints were added such that the design rays would approximately sample the radial-view FOV in a uniform manner. The camera has a pixel pitch of 5.2 μm, which defined the upper bound of the rms spot size. The lower bound on the MTF with tolerance considerations was set to 30 lp/mm, which was approximately one-third of the nominal MTF lower bound for the proposed simulation design in Table 1. The nominal MTF calculation was used during the lens optimization because tolerance simulations are time consuming. The lower bound of the nominal MTF simulation score in Table 4 (60 lp/mm) was set to significantly exceed the tolerance MTF score to compensate for the anticipated performance drop from manufacturing and assembly tolerance. Other fabrication related specifications in Table 4 were based on recommendations from the ITRC-NARL. The additional specifications for the prototype cannot be satisfied with a mere scaled version of the simulation design from Sec. 3. Nevertheless, minor modifications in the optimization constraints yielded a suitable initial prototype prescription. After an initial round of optimization, each lens prescription was compared with stock lenses across different catalogues. Several stock lenses were found to be similar to particular lens prescriptions. Certain elements in the design were functionally separable, such that multiple elements with similar stock prescriptions was be used in its place. After the selected stock lenses were incorporated into the design, the optimization process was iterated to compensate the aberration induced by the stock lenses. This iterative process lasted until the optimization constraints were met.

Table 4

Design constraints for the prototype simulation design.

4.2.

Tolerance Simulation

The tolerance simulation was conducted via the commercial optical design packages, ZEMAX^TM and OSLO^TM. The built-in Monte Carlo tolerance simulation was used to take tolerance deviations into account. Fifty design simulations were generated from random perturbation of each optical surface in the nominal design. These perturbations have a maximum magnitude specified by the tolerance input. Their probability distribution was set to be parabolic, in which the maximum extremes of the permissible perturbation were more likely to occur. The specified MTF criterion (30 lp/mm at 50% modulation for radial-view) was simulated for each of the 50 perturbed designs, then the statistics of these simulations were computed. This tolerance simulation provided a probability of the performance one may expect from a manufactured system with the specified tolerance.

5.1.

Prototype Evaluation

The assembled prototype in Fig. 9 consisted of a lens barrel housing for all the optical components. The assembled prototype was 7.6 cm long and 2.8 cm in diameter. The simulated prototype performance is shown in Table 5. The track length of the prototype prescription was ∼83 mm, and the radial FOV was ∼23°. Figure 10a is the acquired image used for the forward-view MTF test. The USAF1951 target (Thorlabs R3L3S1N) was backlit, and it was placed 2 cm from the front lens surface of the lens barrel. The resolutions of the test patterns in Fig. 10a are listed in Table 6. The top-left horizontal bar pattern is group 0 and element 2, which corresponded to a spatial frequency of 1.12 lp/mm from the USAF1951 specification. The maximum frequency measured for the forward-view was 5.66 lp/mm on the object side, where the modulation was slightly over 50%. After multiplying by a forward view magnification factor of 16.3 (Table 5), this corresponded to a resolution of 92 lp/mm on the image side. The radial-view FOV was not illuminated in Fig. 10a and appeared dark. Figure 10b is the acquired image of a 2 lp/mm pattern. The radial FOV corresponds to the annular region indicated by the double-ended arrows. The forward FOV is the central circular region of the image, and the paper outline of the cylindrical test pattern could be seen near the center of the image. The fixed frequency pattern was rolled into a cylindrical tube [Fig. 10c], and placed around the prototype. The cylindrical test pattern had different line pair orientations for the tangential and sagittal radial contrast measurements. The prototype was then placed in the center of the test pattern tube to acquire the radial-view contrast modulation [Fig. 10d]. Similar fixed frequency test patterns were used to calculate the radial-view contrast modulation in Table 7, whereas the forward-view used the USAF1951 target. Table 7 shows the comparison between the simulated MTF performance after accounting for tolerance and the measured contrast modulation scores.

Fig. 9

Photograph of the 3:1 scale prototype. Four pillars are used to connect the front and back optics groups instead of using transparent housing. The entire device is 7 cm long and 3 cm in diameter.

Fig. 10

Acquired images and the radial-view image acquisition setups. The forward-view acquisition (a) used the USAF1951 target as the test pattern. The four pillars in the radial-view acquisition (b) are part of the housing for the prototype, and the annular region spanned by the double-ended arrows is the radial-view FOV. The scene in (b) was constructed from a custom printed and fixed frequency (2 lp/mm) pattern (c), which was rolled into a tube with a sagittal oscillation direction. This test pattern tube was then fixed around the prototype to create a cylindrical scene, as shown in (d).

Table 5

Simulation results: prototype design.

Table 6

Number of line pairs/millimeter in the USAF 1951 test chart.

Table 7

Comparison of measured MTF and tolerance simulated average MTF.

5.2.

Image Post-Processing

The target application of the proposed imaging system is the diagnostic screening of the GI tract, in which case the gastroenterologist would need to interpret the acquired imagery. Although automated post-processing of the acquired endoscope video may be able to identify potentially malignant regions, a certified diagnosis would require a gastroenterologist to interpret the video frames. The use of curved mirrors had induced distortion on the acquired imagery. This may cause some inconvenience for a human to interpret of the radial-view imagery. There are existing publications^{42, 43} that describe rectification algorithms for panoramic images that were acquired from catadioptric optical devices. Most of these algorithms are based on the pinhole camera model, which is a popular paradigm in the fields of computational vision and computer graphics. In this work, we propose an alternative method for scene reinterpretation.

If successful, a sequential ray trace simulation would establish a mapping between the sampled object points to corresponding blur regions on the image detector. These blur regions are characterized by the point-spread function of the imaging system. This mapping relationship could be simplified by collapsing the blur area on the image to its rms centroid. For an imaging system with a radial FOV, the notion of object depth corresponds to the radial distance between the object and the imaging system. Our method for scene reinterpretation is based the use of precomputed object-image point mappings as a look-up-table. Due to the limited radial DOF range in GI endoscopic applications, it was feasible for us to build a database of the ray traced results of sampled object points. These object points were sampled across the FOV and the DOF of the nominal prototype design simulation. Cubic spline interpolation algorithms were used to interpolate values between the entries of the database. This database was used to process two acquired images for proof-of-concept purposes. The two acquired images in Figs. 11a and 11b were processed by this implementation of the ray database to generate Figs. 11c and 11d. Figure 11a is the radial-view test pattern at 0.5 lp/mm, and Fig. 11b is a business card wrapped around the prototype. The object points were sampled across 10 different object depths within the DOF and 36 samples were taken across the FOV.

6.1.

Performance

Low chromatic aberration in the design was desired, because the emission spectrum of tissue AF due to UV excitation is broadband. The forward-view was designed only to provide white light imaging for navigation purposes, hence the noticeable chromatic focus shift in Fig. 6a was deemed acceptable. The proposed radial-view design achieved similar spot sizes across the three design wavelengths (486.1, 587.6, and 656.3 nm) in Fig. 6b. The forward-view nominal MTF in Fig. 7a was lower than the aliasing upper bound specified in Table 1. Both the tangential and saggital nominal MTF were similar. This was not true for the nominal MTF for the radial-view in Fig. 7b, and extra weights may need to be assigned to equalizing the MTF in both directions for future work. The irradiance chart in Fig. 8 is similar to a spot diagram, but with very coarse sampling. The ability to simulate the amount of irradiance on a pixel could be induced in the optimization framework for future work.

In Table 3, the irradiance collection of the proposed design in the radial-view was much more uniform across its FOV than the reference forward-viewing design. The reference design had better light collection efficiency at objects closer than 3 mm. For objects further than 3 mm, the light collection ability diminished quickly. For object distances shorter than the DOF, Table 2 shows that the rms spot size of the reference design was larger than 10 μm. This suggests more light was collected for objects near the reference design, but at the expense of reduced image quality. The exclusion of a PMMA endoscope sheath gave interesting results; closer objects experienced a decrease in light collection, and distant objects experienced an increase in light collection, relative to the proposed design with a PMMA sheath. The field angle distribution was altered when the endoscope sheath material was changed to air, which led to a modified entrance pupil size. The difference in light collection between sheathed and sheathless designs was small (Table 3). Different endoscope sheath materials may be used with minor impact on the overall light collection characteristics. In depth revision of the imaging system with different sheath materials could be performed by running the lens optimization in most commercial optical design packages.

6.2.

Design Robustness and Issues

Monte Carlo tolerance simulations were conducted using commercial optical design packages. The statistics were set to have larger probability at the endpoints of the tolerance interval. In other words, the perturbations of optical elements during manufacturing were assumed to be more probable at the extremes of the allowed tolerance. This is a stringent assumption that may be relaxed in future simulations. The measured forward-view MTF performance of the fabricated prototype (Table 7) was in fact higher than expected (Table 4). The measured radial-view MTF in the tangential and sagittal directions were dissimilar in Table 7, but the averaged MTF score between the two directions was within the expected range. This was because the averaged MTF was used as the evaluation criterion for the tolerance simulations. For future work, additional constraints should be placed on both the MTF directions during optimization.

The concave mirror in the proposed design could not have a shorter focal length than the distance between the folded mirrors. This was due to practical optomechanical stability issues on the negative singlet lens in Fig. 4. Therefore, the design was re-optimized such that the radial-view lens group contributed a moderate amount of optical power to the system. The mirrors provided the remaining optical power, as well as the redirection of the FOV of the radial-view lens group. The radius of curvature of the doublet in Fig. 4 was the shortest in the entire design (∼1.8 mm). Attempts in splitting the power of the doublet proved to be unsuccessful, since it was instrumental in the correction of coma aberration. The clear aperture of that doublet may need to be smaller, e.g., 10% smaller, to satisfy the minimum edge thickness specified for some lens manufacturing processes. If the clear aperture were to decrease, the ray bundles that are far from the optical axis in Fig. 4 would suffer from vignetting effects. This would reduce the light collection of the field points near the edge of the FOV.

The nonreflecting portions of the mirrors in the manufactured prototype were treated with black micro-arc oxidation to reduce stray light. The four pillars indicated by the single-ended arrows in Fig. 10b are the pillars that hold the two mirrors together in Fig. 9. In the simulation design, there were no pillars and the space between the mirrors would be supported by a transparent cylindrical sheath. As the illumination design was not addressed by this work, backlit illumination was used for both Figs. 10a and 10b.

6.3.

Application Issues

The diameter and track length of typical capsule endoscopes range between 9 to 11 mm and 12 to 26 mm, respectively.^{44, 45} The maximum track length of capsule endoscope optics should be less than 15 mm in order to have room for other components, e.g., antennas and batteries.⁹ From Table 2, the simulated track length of the proposed design was 19.5 mm, which is twice as long as the reference design.²¹ The proposed design was not scaled down further, because a design with shorter radii of curvature is unlikely to be technically or economically feasible with present day lens manufacturing processes. Although the simulated track length of the current design was 19.5 mm (Table 2), it should be suitable for catheter-based devices. One alternative application of the proposed design could be the imaging of colon polyps, which are usually hidden behind folds of the colon.

The radial imagery would require post-processing to generate an interpretable view of the scene. The results of a proof-of-concept algorithm are shown in Figs. 11c and 11d, where the acquired radial panorama imagery was manipulated to appear more user-friendly. Although the FOV between the forward and the radial view is discontinuous, it may be possible to estimate a mosaic map in a post-processing of the acquired video.^{45, 46, 47}