1.

Introduction

Visual inspection for a highly reflective surface is commonly faced with a serious limitation which is that useful information on geometric construction and textural defects is covered by a blur due to specular highlights. With saturating corresponding charge coupled device (CCD) pixels, strong highlights increase the gray value of neighbors in an imaging sensor. The aforementioned phenomenon that reduces contrast and causes image blur is a parasitic image. In the visual system, a parasitic image arises from direct lights on a highly reflective surface (solid line with arrow), scattering glare in the lens element (dash and dot line), stray lights in the camera body (short dash line), and reflection off the lens surface (long dash line), as shown in Fig. 1. In our experiment, these sources causing a parasitic image are mainly scattering glare and direct light. The sum of the two sources saturates image point $P_i$ and increases the light intensity of its surrounding pixels. As shown in Fig. 2, scattering glare, which appears as a parasitic image, obscures the edge of the metallic slice in the presence of highlights. Therefore, it is necessary to remove the parasitic image created by specular highlights and capture realistic scenes.

Fig. 1

Schematic of parasitic image formation. (a) A highlight point $P_o$ could contribute to direct lights (solid line with arrow), scattering glare (dash and dot line), stray lights in camera body (short dash line), and reflection off lens surface (long dash line). (b) The sum of direct light and scattering glare saturates image point $P_i$ and increases light intensity of its surrounding pixels.

Fig. 2

An example of a specular surface.

Many specularity removal techniques have been proposed: color space analysis; neighborhood analysis; polarization; image sequences (IS); multiple-flash images, etc.^1–8 However, when scattering glare is in the neighboring region, these techniques are no longer valid. There are many methods to remove various parasitic images. Jaehyun et al.⁹ propose a multiexposure image fusion algorithm without a ghost effect. Schechner et al.⁷ introduce an approach that can avoid saturation of highlights and improve the quality of such images, in which multiple light sources simultaneously illuminate the object from different directions. Agrawal et al.⁸ present a novel gradient projection scheme that allows removal of reflections and highlights from flash images and uses a flash image and ambient image pair to produce better flash images. Bitlis et al.¹⁰ propose a shift variant analytical parametric model to reduce stray light effects in a digital camera. Liebe et al.¹¹ analyze sun-induced veiling glare. These methods can be broadly classified as illumination techniques, multiexposure imaging, high dynamic range (HDR) camera, and a software algorithm. Multiexposure imaging will take a long time to implement photometric evaluation, spectral calibrations, and image reconstruction.¹² Illumination strategies, which are complex and various, cannot always completely acquire full information of the measured work piece. Although an HDR camera can be used to raise the saturation point by increasing the capacity of the sensor electron well, producing large sensors is excessively expensive and reduces sensor resolution. On average, only a small portion of a scene contains strong highlights and therefore needs high capacity sensors. Software algorithms postprocess an image that already contains parasitic components. As the highlight due to the specular reflection of a metallic slice is very strong, the performance of software algorithms is poor.

This article is inspired by previous works on computational camera as follows. Nayar et al.¹³ describe a programmable imaging system that uses a digital micromirror device (DMD) to alter the geometric and radiometric characteristics. Ri et al.¹⁴ propose phase-measuring profilometry using a DMD camera to extend the intensity range. Ankit et al.¹⁵ present an optical relay system for mechanical or electronic color spectrum control that utilizes a DMD in the optical path to modulate space lights. Adeyemi et al.¹⁶ demonstrate a system that uses precise DMD control of the projector to enhance the dynamic range.

In this article, we have implemented a programmable imaging system that goes by the name of DMD Camera. A method for removal of a parasitic image and elimination of high reflection based on the DMD camera is presented. With this system, we can decrease the intensity of a scene ray based on the needs of the application before it reaches the imaging optics. This article explains the space light modulation (SLM) strategy for inspection of a metallic slice in detail. Finally, we demonstrate the effectiveness of a correction image and a parasitic image.

2.

Prototype System

Here, we describe programmable imaging system with a micromirror array. The system is composed of a CCD, DMD, image processor, and two imaging lenses (Len1 and Len2), as shown in Fig. 3. To avoid unexpected stray light caused by devices ahead of the DMD, the optical system is open. DMD is a two-dimensional (2-D) array optoelectronic element in which every pixel has two stable mirror states ($+12\text{-}\mathrm{deg}$ tilt and $-12\text{-}\mathrm{deg}$ tilt) to control the direction of the scene ray with high precision over space and time. A PC, as the imaging processor, handles the camera image and controls the DMD pattern. The object, which is a Neodymium magnet slice in this article, is imaged at the DMD plane by Len2. Len1 focuses the reflected ray from the DMD to the CCD plane. The DMD, mounted on the intersection of two optical axes, modulates the incident ray from the object and reflects the processed ray to the CCD by 24-deg. Based on the working principle of the DMD, the reflected ray from the DMD is produced by pulse width modulating the mirror elements over the operating refresh time. Thus, the reflected intensity gray level is proportional to the period of time that the mirror is on $+12\text{-}\mathrm{deg}$ tilt state. As the CCD receives different exposure times, the object image will be modulated by the DMD.

Fig. 3

Schematic of system and experimental setup.

2.2.

Mapping from Object to Digital Micromirror Device

Using the thin lens, an arbitrary plane in the object space will be imaged to a corresponding plane in the image space. The object plane is expressed as $a(x-x_0)+by+cz=0$, where $(x_0,0,0)$ is the intersection point of the object plane and optical axis, thus we obtain an image plane:

Eq. (1)

$$a(1+\frac{x_0}{f})(x^{\prime }-\frac{f{x}_0}{f+x_0})+b{y}^{\prime }+x{z}^{\prime }=0.$$

In Fig. 4, the object space coordinate system is defined by the intersection point $(x_0,0,0)$ of the object plane and optical axis which is taken as the origin, the $w$-axis which is taken to be vertical, the $v$-axis which is taken to be horizontal, and the $u$-axis which parallels the normal vector of the object image. The image space coordinate system is defined by the intersection point $[f{x}_0/(f+x_0),0,0]$ of the DMD plane and optical axis which is taken as the origin, the $w^{\prime }$-axis which is taken to be vertical, the $v^{\prime }$-axis which is taken to be horizontal, and the $u^{\prime }$-axis which parallels the normal vector of the DMD image. The angle $\theta$ between $u$ and the optical axis is expressed as $\tan \theta =b/a$. In the same way, the angle ${\theta }^{\prime }$ between $u^{\prime }$ and the optical axis is expressed as $\tan {\theta }^{\prime }=b/[a(1+x_0/f)]$. The relationship between ${\theta }^{\prime }$ and $\theta$ can be written as $\tan \theta =(1+x_0/f)\tan {\theta }^{\prime }$. ${\theta }^{\prime }$ is the incident angle in the DMD plane and is set to 24 deg to insure that the reflected angle is 0 deg. Thus, it can be seen that $\theta$ is only associated with $x_0$ which is the measurement distance when the focal length of Len2 is invariant. The mapping of Len2 from a 2-D point in the image plane to a 2-D point in the object plane is given by

Eq. (2)

$$\{\begin{array}{l}v=-\frac{\cos {\theta }^{\prime }}{\cos \theta }\times \frac{1}{{\beta }_2-\frac{v^{\prime }}{f}\sin {\theta }^{\prime }}{v}^{\prime }\\ w=\frac{1}{{\beta }_2-\frac{v^{\prime }}{f}\sin {\theta }^{\prime }}{w}^{\prime }\end{array},$$

where ${\beta }_2=x_0^{\prime }/x_0=1/(1+x_0/f)$ is the magnification factor of Len2. It is thus clear that the coordinate conversion between the image plane and the object plane conforms to a strictly linear mapping. Hence, the view magnification of the DMD camera can be expressed as $\beta ={\beta }_1{\beta }_2$. As ${\beta }_1$ is invariant in our experiment, the view magnification of the DMD camera is decided by the focal length of Len2 and the measurement distance.

Fig. 4

Model of mapping from object to digital micromirror device (DMD).

3.

Removing Parasitic Image

3.2.

Strategy

We assume that $I(x,y)$ is the intensity of the recorded image which is composed of a correction image $O(x,y)$ and a parasitic image $F(x,y)$.

Eq. (5)

$$I(x,y)=O(x,y)+F(x,y).$$

In our camera system, the result of processing a captured image can be fed again into the DMD as a pattern, and this process could be repeated. So far as the DMD is concerned, the relationship between the incident intensity and reflected intensity is obtained by using an optical power meter. From Eq. (3), one can observe that $O(x,y)$ is linearly modulated by the DMD, but $F(x,y)$ is not. Based on this property, the process of the experiment, which achieves a set of recorded image $I_k(x,y)$ $k=0,1,\dots K$, is described as follows:

• Step 1 Initialize the DMD pattern $P_0(u,v)$ to be a 684 by 608 matrix of 255. Obtain the intensity of the recorded image $I_0(x,y)$ that is totally reflected by the DMD.

• Step 2 Based on clustering methods, select the threshold $t_0$ to determine the modulated region $D_1$ of the recorded image $I_0(x,y)$; threshold $t_0$ is determined by squared intensity differences between pixels and the cluster center; the low frequency component of the scattering glare depends on threshold $t_0$ to be segmented from the recorded image, but direct light and the high frequency component of the scattering glare are almost not affected by it.

• Step 3 $M_1$ represents the corresponding region in the DMD pattern $P_1(u,v)$, which is given by

  Eq. (6)

  $$D_1\stackrel{H}{\to }{M}_1.$$

• Step 4 Define the DMD pattern as

  Eq. (7)

  $$W_1(u,v)=\frac{I_0(x,y)-t_0}{\alpha }+t_0(u,v)\in M_1,$$
  where $\alpha$ is the modulation scale factor.

• Step 5 Repeat steps 2, 3, and 4 until the modulated region has no pixels, and obtain the threshold $t_k$, $k=0$, $1\dots ,\mathrm{K}-1$, and the modulated region of the recorded image $D_k$, $k=0$, $1,\dots ,\mathrm{K}-1$ in addition to the recorded image $I_k(x,y)$.

  ${\widehat{I}}_k(x,y)$ represents the intensity estimate of $I_k(x,y)$, which is inversely computed by

  Eq. (8)

  $${\widehat{I}}_k(x,y)=I_{k+1}(x,y)·\frac{[I_k(x,y)-t_{k-1}]/\alpha +t_{k-1}}{[I_{k+1}(x,y)-t_k]/\alpha +t_k}(x,y)\in D_k.$$

Considering the maximum possible brightness value of the CCD intensity level, there are two cases of solving a parasitic image, to be described in Fig. 5. Solid and dotted curves represent intensities of $I_k(x,y)$ and ${\hat{I}}_k(x,y)$, respectively. Also, the scattering glare is marked by hachures. In Fig. 5(a), the region of glare is enclosed between the boundary lines of $D(255)$ and $D(t_k)$ with the condition that the boundary dash line of $D(255)$ is outside the boundary of $D(I_k={\hat{I}}_k)$. In Fig. 5(b), the region of glare is enclosed between the boundary lines of $D(I_k={\hat{I}}_k)$ and $D(k)$ with the condition that the boundary dash line of $D(255)$ is inside the boundary of $D(I_k={\hat{I}}_k)$. With the condition that the dash line of $D(255)$ is inside the boundary of $D(I_k={\hat{I}}_k)$, we can obtain the estimate of the parasitic image:

Eq. (9)

$${\widehat{G}}_k(x,y)=[I_k(x,y)-{\widehat{I}}_k(x,y)]{\tilde{W}}_k(x,y)\in [D(t_k)-D(I_k={\widehat{I}}_k)],$$

where ${\tilde{W}}_k$ means the weight of $I_k(x,y)$. It is a relational function containing $W_k$ and the reflectivity of the DMD.

Fig. 5

Two cases of solving parasitic image.

That $D[(t_{k+1}-t_k)\alpha +t_k]>D(I_k={\widehat{I}}_k)$ is a necessary condition of global parasitic image:

Eq. (10)

$$\widehat{G}(x,y)=\sum _k^{K-1}{\widehat{G}}_k^{D(t_k)-D[(t_{k+1}-t_k)\alpha +t_k]}.$$

Therefore, all selected thresholds in the experiment should be equal:

Eq. (11)

$$t_0=t_1=\dots t_{K-1}.$$

With another case where the dash line of $D(255)$ is outside the boundary of $D(I_k={\hat{I}}_k)$, we can obtain the estimate of the parasitic image:

Eq. (12)

$${\widehat{G}}_k(x,y)=[I_k(x,y)-{\widehat{I}}_k(x,y)]{\tilde{W}}_k(x,y)\in [D(t_k)-D_k(255)].$$

That $D[(t_{k+1}-t_k)\alpha +t_k]>D_k(255)$ is a necessary condition of the global parasitic image:

Eq. (13)

$$\widehat{G}(x,y)=\sum _k^{K-1}{\widehat{G}}_k^{D[(t_k-t_{k-1})\alpha +t_{k-1}]-D_k(255)}.$$

Therefore, all selected thresholds in the experiment should be a monotone decreasing sequence:

Eq. (14)

$$t_0\ge t_1\ge \dots t_{K-1}.$$

From Eq. (5), we composite the individual subtracted captures together to form a complete image of the scene:

Eq. (15)

$$O(x,y)=\sum _k^K[I_k{\tilde{W}}_k-{\widehat{G}}_k^{D_k(t_k)-D_{k+1}(t_{k+1})}].$$

DMD enables radiometric modulation of the imaged scene rays with very high precision and physically limits the amount of scattering glare created in the camera. Also, with the highlight intensity falling off, signal-to-noise ratios (SNRs) are increasing. A large $\alpha$ allows minimizing the highlight and increasing SNR, whereas IS $I_k(x,y)$ is needed to record a global scene. It is suggested that the regions $D[(t_{k+1}-t_k)\alpha +t_k]$ need to be as large as possible. This creates a tradeoff between the SNR and the integrity of the estimated parasitic image. Based on empirical observation, $\alpha =2$ is suitable for our experiment.

4.

Implementation Results

We provide two examples showing the successful application of the proposed method. The object is mounted on the working scene of a DMD camera in a dark room. Projecting lights illuminate the object from a constant angle from the right side. One example is the removal of parasitic image on metal slice, as shown in Fig. 2. It can be observed that high reflective light due to specular reflection, produces the parasitic image on the left edge of the metallic slice and covers the geometric and textural information of the metallic edge which is the stamping region in the fabrication process. Removing the parasitic image by using the preceding strategy, a correction image is given in Fig. 6. Figure 6(a) shows the recorded image where the edge of the metallic slice is covered by the parasitic image. Figure 6(b) shows the correction image where the parasitic image is removed by our approach. Notice that the edge of the metallic slice is visible. Figure 6(c) shows the parasitic image that is recovered. Figure 6(d) depicts the 2-D luminance distribution of the parasitic image superimposed by the central point sources and spreading components.

Fig. 6

A close-up of experimental result: (a) record image, (b) correction image, (c) parasitic image, and (d) a close-up of parasitic intensity distribution.

The other example is the removal of the parasitic image on a metal hemisphere, as shown in Fig. 7. Figure 7(a) shows the imaging scene of the metal hemisphere. The dashed box is the close-up region where the geometric and textural information of the metal hemisphere are covered by the highly reflective light. Figures 7(b), 7(c), and 7(d) are the close-up image, correction image, and parasitic image, respectively. Luminance estimation of the parasitic image as a by-product of the preceding strategy is shown in Fig. 7(e).

Fig. 7

The other example: (a) metal hemisphere, (b) a close-up of record image, (c) correction image, (d) parasitic image, and (e) intensity distribution of parasitic image.

Experimental results show that the proposed approach successfully remove a parasitic image on metal surfaces of different shapes and sizes. Without multiexposure imaging or multi-illumination, we can recover almost the full resolution information by the SLM strategy. The DMD camera achieves its flexibility by using a programmable array of micromirrors. With our method, the highlight is reduced before image formation. As the SNR is improved, a high quality image is provided. A simple algorithm containing modulated area recognition, precise region mapping, and separation of parasitic image and correction image is effective to decrease time-consumption in host processor.

5.

Limitation

However, our method does suffer from limitations of precision and application. First, our method requires knowledge of the exact corresponding relationship between the CCD and DMD. Second, for achieving high accuracy mapping from the DMD to CCD, the depth of field of our setup should be limited to a small-scale range. Moreover, the high intensity of the central point source is not completely eliminated in our experiment. Third, our method can handle the highlight due to specular reflection, but the dynamic range of the DMD camera limits the removable parasitic image. The relationship between the DMD pixel digital value and reflectivity is given by

Thus, we define the maximum removable parasitic image as

6.

Conclusion

A parasitic image created by strong highlights due to specular reflection covers useful information and reduces image contrast. Removing the parasitic image is a widespread requirement in science, medicine, and photography. In this article, a DMD camera composed of a DMD, CPU, and CCD is developed to achieve programmable imaging. We have developed a new method for removing the parasitic image from an optical system, using iterative modulation with a DMD camera to remove and estimate the parasitic image. Meanwhile, the method obtains the estimate of the parasitic image and provides a novel pathway to analyze and evaluate the parasitic image in optical system. As experimental results, the parasitic images on metal surfaces of different shapes and sizes are successfully removed.