1.1.

Planar-Object-Based Methods

In this category, strategies are summarized, which use two-dimensional (2-D) calibration planes to calibrate affine cameras.

Lanman et al.¹⁷ presented an approach to reconstruct 3-D surface data based on the motion of an object’s depth discontinuities when viewed under orthographic projection. To this end, the authors introduce a model-based calibration approach for a telecentric camera using a planar checkerboard, modified with a pole of known height in order to recover the ambiguity in sign, when estimating the extrinsic parameters for a specific calibration pattern pose. The camera calibration uses a factorization approach inspired by Zhang¹⁸ in order to provide start values for the camera intrinsics and extrinsics. The parameters are further refined in a Levenberg–Marquardt optimization. The authors do not consider lens distortion.

Chen and Liao et al.^6,19 presented a two-step calibration approach for a telecentric stereo camera pair, which comprises a factorization method to determine the initial camera parameters similar to the approach found in Ref. 17. The parameters are refined in a nonlinear optimization routine. The sign ambiguity problem when recovering the rotation matrix is solved with help of a micropositioning stage used to capture two calibration plane poses under known translational displacement. Moreover, the approach considers radial distortion. The authors suggest the acquisition of as many target poses as possible in order to avoid degeneracy and in consequence an “ill calibration” (Ref. 6, p. 88).

Li et al.^11,20 proposed a calibration method for a single camera based on an analytical camera description in order to model the distortion of a telecentric lens correctly (namely radial, decentering, and thin prism distortions) and developed it into an approach to calibrate a structured light sensor with telecentric camera and projector. It is not fully clear how the authors solve the problem of sign ambiguity, when recovering the extrinsics. In their literature review, Li and Zhang⁹ state that “it is difficult for such a method to achieve high accuracy for extrinsic parameters calibration […].”

Yao and Liu²¹ introduced an approach where again an additional stage is used to solve for the extrinsic sign ambiguity. After a camera start value determination based on a distortion-free camera model, two nonlinear optimization steps are executed. In the first step, the calibration plane coordinates are optimized to allow the usage of cheap print patterns. Second, all camera parameters are refined, including radial and tangential lens distortion, and also the distortion center. The approach provides a greater flexibility, as the distortion center is not necessarily fixed to the middle of the sensor. Nevertheless, a comparison between calibration results based on a printed and a precisely manufactured pattern shows great difference in the estimated distortion parameters. The authors argument that the distortion is generally small for telecentric lenses. Therefore, small differences in the optimization procedure result in great parameter differences. Another reason could be the missing re-estimation of the calibration plane coordinates in the second nonlinear optimization step. The distortion-free camera model is considered ground truth when estimating the calibration points.

Hu et al.²² presented an approach for a single camera calibration based on the results by Yao et al., but provided a method to gain an initial estimation for the distortion center to avoid local minima. The distortion center and the parameters are further refined in a subsequent nonlinear full-parameter optimization. The authors consider both radial and tangential distortion coefficients. Their approach is developed into a full calibration and reconstruction routine for a microscopic stereo vision system.⁵

Li and Zhang⁹ introduced a calibration routine for a hardware setup comprising an entocentric projector and a telecentric camera and used the absolute coordinate frame of the projector as a reference for the telecentric camera. In the first step, the projector is calibrated with the standard camera pinhole model. The necessary correspondences are provided by the uncalibrated telecentric camera, capturing multiple calibration plane poses with and without vertical and horizontal phasemap, respectively (cf. concept of image capturing projector in Ref. 23). The feature correspondences used for the projector calibration are then projected back into 3-D (in the projector’s coordinate frame) to calibrate the affine camera. This approach is very stable but requires an entocentric projector, which might not be available in a sensor setup.

1.2.

Affine Autocalibration

This category comprises so-called autocalibration approaches for affine cameras. As most autocalibration approaches require structure-from-motion results as input, exemplary developments in this field are covered as well.

According to Hartley et al., “auto-calibration is the process of determining internal camera parameters directly from multiple uncalibrated images” (cf. Ref. 15, p. 458), without using specially designed calibration devices with known metric distances, or scene properties such as vanishing points. The derivation of the camera intrinsics might be directly connected to the reconstruction of 3-D scene points, upgrading a nonunique projective or affine reconstruction to a Euclidean reconstruction by applying special constraints. Such a constraint could be the assumption of fixed camera intrinsics for all images.

The basic theory for autocalibration of a perspective projection camera is formulated by Faugeras et al.²⁴ Well-known classical structure-from-motion approaches under orthography are suggested for the two-view scenario by Koenderink and van Doorn,²⁵ and for at least three views by Tomasi and Kanade, namely the factorization algorithm.²⁶ The camera is moved around an object and captures images from different positions under orthographic projection. Detected feature correspondences in the sequential images are used to recover the scene’s shape and the camera motion in affine space. Appropriate boundary conditions allow for the reconstruction of Euclidean structure up to scale.

The affine 3-D reconstruction result is used as input in the generalized affine autocalibration approach by Quan.²⁷ The authors introduced metric constraints for the affine camera, comprising orthographic, weak perspective, and paraperspective camera model.

An important precondition for the applicability of the Tomasi–Kanade factorization algorithm is the visibility of the used point correspondences in all views. Using data subsets, Tomasi and Kanade enable the factorization approach to handle missing data points. The subset-based reconstructed 3-D coordinates are projected onto the calculated camera positions in order to obtain a complete measurement matrix. This method nevertheless requires feature points that are visible in all views (the data subsets). It allows patching of missing matrix entries, rather than providing an approach for sparse data sets.

Brandt derived a more flexible structure-from-motion approach, as “no single feature point needs to be visible in all views” (cf. Ref. 28, p. 619). The approach comprises two iterative affine reconstruction schemes, and a noniterative, linear method, using four noncoplanar reference points visible in all views. Brandt and Palander²⁹ furthermore presented a statistical method to recover the camera parameters directly from provided point correspondences without the necessity of an affine reconstruction. As solution, a posterior probability distribution for the parameters is obtained.

Guilbert et al. proposed an approach for sparse data sets using an affine closure constraint, which allows “to formulate the camera coefficients linearly in the entries of the affine fundamental matrices” (cf. Ref. 30, p. 317), using all available information of the epipolar geometry. The authors claim that the algorithm is more robust against outliers compared to factorization algorithms. Moreover, they present an autocalibration method and directly compare it to Quan’s method. The so-called contraction mapping scheme shows a 100% success rate in reaching the global minimum and a lower execution time.

Horaud et al.³¹ described a method to recover the Euclidean 3-D information of a scene when capturing scene data with an uncalibrated affine camera mounted to a robot’s end effector. The authors use controlled robot motions, in order to remove affine mirror ambiguity and guarantee a unique affine reconstruction solution. The camera intrinsics are obtained by performing an QR-decomposition according to Quan.²⁷

An approach of motion recovery from weak-perspective images is presented by Shimshoni et al.³² The authors reformulate the motion recovery problem to a search for triangles on a sphere, offering a geometric interpretation of the problem.

Further information on the concepts of affine autocalibration in general can be found in Ref. 33, p. 163 et seq.

1.3.

Hybrid Method

Liu et al.¹² combined the Tomasi–Kanade factorization algorithm with a 3-D calibration target in order to retrieve the parameters of a fringe projection system with telecentric camera and projector. The authors use a 3-D calibration target with randomly distributed markers. The target consists of two 2-D planes, forming a rooftop structure. As the marker positions on the planes are not required to be known beforehand, the target manufacturing requirements are low.

The suggested approach is basically a two-step routine: the 3-D calibration target is captured by the camera in different orientations, with and without two sets of gray code patterns, generated by the projector. The approach of the so-called image capturing projector by Zhang et al.²³ allows now to solve the correspondence problem between camera, projector, and circular dots on the target. First, the dots’ image coordinates are extracted for camera and projector. Then, using the Tomasi–Kanade algorithm and an appropriate upgrade scheme from affine to Euclidean space, an initial guess for the calibration targets shape (3-D coordinates of the circular dots) and the corresponding projection matrices are obtained. As the point cloud data can only be reconstructed up to scale, the camera’s effective magnification has to be provided in order to reconstruct metric 3-D data of the circular dots. As no metric distances are defined on the 3-D calibration target, the authors suggest the additional usage of a simple 2-D target in a plane-based calibration routine, such as given in Ref. 21. In the second step, the initial guesses are used as start parameters in a nonlinear bundle adjustment scheme to minimize the total projection error. Next to the target poses, also the projector-camera rig parameters and the 3-D coordinates of the calibration target are refined.

1.4.

Contributions in this Paper

The approach by Liu et al. is an alternative to the routines discussed in Sec. 1.1, avoiding among others planarity-based degeneracy problems [e.g., as reported by Chen et al. in Ref. 6 (p. 88) or in general by Collins et al. in Ref. 34]. The approach does not rely on the usage of a plane with linear stage or a pole but on a 3-D rooftop calibration target. The Tomasi–Kanade algorithm provides a good estimation of the camera rotations (even with a relatively low number of captured object poses), which allows for a robust convergence of the subsequent nonlinear refinement.

Nevertheless, in order to obtain a fully calibrated measurement system, the magnification factor has to be determined separately in an individual step, which is cumbersome. Also, the authors do not address the problem of the so-called mirror ambiguity, which is still present when reconstructing affine point data with the Tomasi–Kanade algorithm [cf. Ref. 35 (p. 415), Ref. 36 (p. 7–8), and Ref. 31 (p. 1576)]. As the reconstructed 3-D data might be mirrored, the start values for nonlinear optimization are also estimated based on a mirrored point cloud, resulting in mirror-based camera locations (for further clarification see Sec. 3.2.5). Although the subsequent nonlinear optimization might still converge, triangulated geometry results might be mirrored, as the camera – projector – arrangement is potentially inverted.

The mirror ambiguity is especially in a stereo camera setup problematic. Two individual affine reconstruction schemes for both cameras can result in start values, that are both based on a mirrored and nonmirrored point cloud. A combination of the camera start values in a single stereo optimization directly affects its robustness. The optimizer might converge toward a local minimum or not converge at all.

Therefore, we propose an adapted calibration procedure for a structured light sensor comprising a telecentric stereo camera pair and an entocentric projector as feature generator. The projector is not meant to be used for the calibration of the affine cameras to allow for a direct calibration. Hence, the suggested routine is also valid for a simple stereo camera setup without projector. As the triangulation is conducted between the two cameras, the hardware setup is equivalent to the setup presented by Liu et al. (two telecentric lenses are used for triangulation).

Our routine is also based on the Tomasi–Kanade factorization algorithm to determine the start values. The application of a more recent affine reconstruction and autocalibration scheme might be interesting in the scope of this paper, but the additional effort for the algorithm implementation will prove not to be necessary, as the proposed calibration scheme works just fine. The feature visibility restriction will not prove to be an obstacle in the suggested approach, as the number of detectable features in all views is large enough by introducing an appropriate calibration target.

The contributions of this paper can be summarized to the following points:

3.1.

Calibration Target and Marker Detection

The layout of the 3-D calibration target is shown in Fig. 1(a). The rooftop structure was introduced by Liu et al., but the random dot distribution is substituted by two defined planar dot patterns with individual coordinate frames $\{O_1\}$ and $\{O_2\}$. It is necessary to differentiate between the two patterns. To this end, Aruco markers³⁷ are printed in the left upper corner of each plane. The markers allow for a distinct and robust marker detection [Fig. 1(b, 1)], which permits the masking of everything except for the associated plane data [Fig. 1(b, 2–3)]. After approximate plane detection, the circle markers are identified by a detection algorithm, and the image-plane-correspondences are obtained [Fig. 1(b, 4)].

Fig. 1

(a) Layout of calibration target with two individual coordinate systems $\{O_1\}$ and $\{O_2\}$. (b) Detection procedure. Based on the detected Aruco markers [(id1) and (id2) dots, (b, 1)], the regions of interest (ROI) for each plane are determined (b, 2). The ROIs allow for a planewise masking (b, 3) and dot marker detection [green and red, respectively, (b, 4)].

It is important to notice that at this point, the correspondences of both planes are given in the two individual coordinate frames $\{O_1\}$ and $\{O_2\}$. There is no information on the rigid body transformation which allows for a marker point formulation in a single coordinate frame. The $z$ coordinate for all detected features—independently of the chosen plane—is zero. The necessary transformation will be estimated in the subsequent calibration routine. The advantage is that single planes with individual marker coordinate frames are easier to manufacture than a single 3-D calibration target.

3.2.

Start Value Determination

3.2.3.

Estimation of rigid body transformation between calibration planes

In order to provide a start value for the rigid body transformation ${{}^{O_1}\mathbf{T}}_{O_2}$ (cf. Fig. 2), the transformations ${}^{T_2}{\mathbf{T}}_{O_1}$ and ${}^{T_2}{\mathbf{T}}_{O_2}$ between the plane data and the reconstructed 3-D calibration points have to be estimated. The relationship between the points is given as

Eq. (13)

$${}_{T_2}{\mathbf{X}}_{k,h}={}^{T_2}{\mathbf{T}}_{O_1}{{}_{O_1}\mathbf{X}}_{l_1,h},\text{with}k=l_1=1,\dots ,n_1,$$

Eq. (14)

$${{}_{T_2}\mathbf{X}}_{k,h}={{}^{T_2}\mathbf{T}}_{O_2}{{}_{O_2}\mathbf{X}}_{l_2,h},\text{with}\{\begin{array}{l}k=n_1+1,\dots ,n\\ l_2=1,\dots ,n_2\end{array}.$$

Fig. 2

Rigid body transformations between the reconstructed 3-D data of the calibration target given in $\{T_2\}$ and the coordinate frames of the calibration planes $\{O_1\}$ and $\{O_2\}$.

${{}_{T_2}\mathbf{X}}_{k,h}$ is considered to be scaled according to Eq. (12)—resulting in a metric point cloud—without introducing an additional index indicating scaling. In accordance with the previous section, the total number of calibration points is $n=n_1+n_2$. The number of points on the first plane is $n_1$ and on the second plane $n_2$.

The rigid body transformations ${{}^{T_2}\mathbf{T}}_{O_1}$ and ${{}^{T_2}\mathbf{T}}_{O_2}$ are obtained by an SVD (e.g., as given in Ref. 43), since ${{}_{T_2}\mathbf{X}}_{k,h}$ and the corresponding calibration plane points ${{}_{O_1}\mathbf{X}}_{l_1,h}$ and ${{}_{O_2}\mathbf{X}}_{l_2,h}$ are known.

The desired transformation is then determined according to

Eq. (15)

$${{}^{O_1}\mathbf{T}}_{O_2}={({}^{T_2}{\mathbf{T}}_{O_1})}^{-1}{}^{T_2}{\mathbf{T}}_{O_2}={}^{O_1}{\mathbf{T}}_{T_2}{}^{T_2}{\mathbf{T}}_{O_2}.$$

3.2.5.

Affine mirror ambiguity

Due to the so-called mirror ambiguity of the affine projection, the reconstructed 3-D points obtained by the Tomasi–Kanade factorization algorithm are potentially not accurate but might be mirrored.^35,36 For further clarification Fig. 3(a) is given (inspired by Ozden et al.⁴⁴): a mirror reflection of a 3-D calibration object (here defined by the points $A^{\prime }{B}^{\prime }{C}^{\prime }$) w.r.t. a plane, which is in parallel to the image sensor (mirror plane), will have the same affine projection result in camera 1 as the original object ($ABC$). (In Fig. 3, the sensor plane for camera 1 and the mirror plane are equal.) Therefore, based on multiple views of the calibration object, two different 3-D reconstructions are valid: the mirrored and the original and nonmirrored point cloud.

Fig. 3

Mirror ambiguity of affine projection. (a) Principle outline (based on Ref. 44). The optical axes are indicated by black arrows. (b) Transformations between mirrored and original point clouds for the calibration target.

In consequence, the truncated rigid body transformations for the different camera poses might have been estimated based on a mirrored 3-D point cloud. Both camera poses according to Fig. 3(a) (cam 2′ and 2) result in the exact same image coordinates, when projecting the points $ABC$ or $A^{\prime }{B}^{\prime }{C}^{\prime }$ onto the sensor. This can be shown with help of the inhomogeneous affine projection formulation according to Eq. (2). For the sake of simplicity, the camera matrix $\mathbf{K}$ is set to the identity matrix ($\frac{m}{s_x}=\frac{m}{s_y}=1$, $c_x=c_y=0,\rho =90\mathrm{deg}$), and the translational shift is supposed to be zero ($t_x=t_y=0$), yielding a simple orthographic projection according to

Eq. (19)

$$\left(\begin{array}{c}{}_{c}u\\ {}_{c}v\end{array}\right)=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\end{array}\right]\left(\begin{array}{c}{}_{O}X\\ {}_{O}Y\\ {}_{O}Z\end{array}\right).$$

If Eq. (19) is expanded by a $(3\times 3)$ mirror matrix ${\mathit{Q}}_{\mathrm{mir}}$ (point reflection about $xy$-plane) and its inverse, nothing is changed (as ${\mathbf{Q}}_{\mathrm{mir}}{\mathbf{Q}}_{\mathrm{mir}}^{-1}=\mathbf{I}$), yielding

Eq. (20)

$$\left(\begin{array}{c}{}_{c}u\\ {}_{c}v\end{array}\right)=\left[\begin{array}{ccc}{r}_{11}& r_{12}& r_{13}\\ r_{21}& r_{22}& r_{23}\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& -1\end{array}\right]\left(\begin{array}{c}{}_{O}X\\ {}_{O}Y\\ {}_{O}Z\end{array}\right)=\left[\begin{array}{ccc}{r}_{11}& r_{12}& -r_{13}\\ r_{21}& r_{22}& -r_{23}\end{array}\right]\left(\begin{array}{c}{}_{O}X\\ {}_{O}Y\\ -{}_{O}Z\end{array}\right).$$

In consequence, object point ${}_O\mathbf{X}$ is mirrored, and the $r_{13}$ and $r_{23}$ components of the truncated matrix are changed in sign [cf. Ref. 36 (p. 7–8)]. Still, ${}_O\mathbf{X}$ is imaged onto the same sensor coordinates, as (exemplary given for ${}_{c}u$)

Eq. (21)

$${}_{c}u=r_{11}·{}_{O}X+r_{12}·{}_{O}Y+r_{13}·{}_{O}Z=r_{11}·{}_{O}X+r_{12}·{}_{O}Y+(-r_{13})·(-{}_{O}Z).$$

Therefore, two mathematically equal solutions exist (global minima; in the scope of this paper, the term global minimum stands for a solution with realistic camera intrinsics, but which potentially differs from the physically correct pose estimate due to mirror ambiguity. It is used in distinction to a local minimum, which corresponds to a solution with physically unrealistic intrinsic estimates.), when camera poses (in terms of truncated rigid body matrices ${{}^C\tilde{\mathbf{T}}}_{O_1,i}$) and the shape of the calibration target (in terms of ${}^{O_1}{\mathbf{T}}_{O_2}$) are estimated—one corresponds to the mirrored, the other to the nonmirrored solution.

A yaw–pitch–roll decomposition of ${{}^{O_1}\mathbf{T}}_{O_2}$ with rotation angles $\alpha$, $\beta$, and $\gamma$ can help to identify whether a mirrored scenario is present or not. In case of a mirrored scenario, the transformation is based on the mirrored coordinate system $\{O_{2,\mathrm{mir}}\}$ and not on the nonmirrored system $\{O_2\}$ [cf. Fig. 3(b)], resulting in a different yaw–pitch–roll decomposition: $\alpha$ and $\gamma$ differ in sign.

In summary, in case of an erroneous, mirror-based start value determination, an elementwise sign correction is mandatory for ${{}^{O_1}\mathbf{T}}_{O_2}$ and ${{}^C\tilde{\mathbf{T}}}_{O_1,i}$, with help of corrective matrix ${\mathbf{T}}_{\mathrm{mir}}$

Eq. (22)

$${\mathbf{T}}_{\mathrm{mir}}=\left[\begin{array}{cccc}1& 1& -1& 1\\ 1& 1& -1& 1\\ -1& -1& 1& -1\\ 1& 1& 1& 1\end{array}\right].$$

The elementwise sign correction is realized by the Hadamard product (symbol ∘) according to

Eq. (23)

$${{}^C\tilde{\mathbf{T}}}_{O_1,i}={{}^C\tilde{\mathbf{T}}}_{O_1,\mathrm{mir},i}\circ {\mathbf{T}}_{\mathrm{mir},\cancel{[3,\text{row}]}},$$

Eq. (24)

$${{}^{O_1}\mathbf{T}}_{O_2}={{}^{O_1}\mathbf{T}}_{O_2,\mathrm{mir}}\circ {\mathbf{T}}_{\mathrm{mir}}.$$

Additional information on the necessary matrix correction is given by Shimshoni et al.³²

3.3.

Nonlinear Parameter Refinement

Once the start parameters for both cameras are determined, a nonlinear refinement is executed based on a Levenberg–Marquardt optimization by minimizing

To differentiate between the two stereo cameras, indexes $c$ (and $C$) are extended to $c_1$ and $c_2$ ($C_1$ and $C_2$), respectively, whereas the other parameters are distinguished by indices 1 or 2 (e.g., ${\mathbf{k}}_1$ as the first camera’s distortion coefficients). As the number of correspondences and of captured poses per camera might differ, camera-specific numbers are defined by $n_{c_1}$ or $n_{c_2}$ (correspondences) and $m_{c_1}$ or $m_{c_2}$ (poses), respectively. $e_{\mathrm{stereo}}$ is the sum of the squared geometric errors between the matched feature points ${{}_{c_1}\mathbf{u}}_{ij}$ (or ${{}_{c_2}\mathbf{u}}_{ij}$) and the corresponding projected points ${{}_{c_1}\widehat{\mathbf{u}}}_{ij}$ (or ${{}_{c_2}\widehat{\mathbf{u}}}_{ij}$) (based on the estimated model). The mean absolute projection error $e_{\mathrm{abs},\text{mean}}$ is given in pixel and is defined in the camera sensor coordinate frames $\{c_1\}$ and $\{c_2\}$, respectively, and defined as (here given for the first camera)

The camera matrices ${\mathbf{K}}_1$ and ${\mathbf{K}}_2$ (three parameters per camera), the distortion vectors ${\mathbf{k}}_1$ and ${\mathbf{k}}_2$ (four parameters per camera), the truncated rigid body transformations ${{}^{C_1}\tilde{\mathbf{T}}}_{O_1,i}$ and ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,i}$ (five parameters per view and camera, the Rodrigues’ formula is used to express the rotation), and the rigid body transformation ${{}^{O_1}\mathbf{T}}_{O_2}$ (six parameters, coupling the errors of camera one and two) are optimized simultaneously, resulting in a total number of $2·3+2·4+5·(m_{c_1}+m_{c_2})+6=20+10·m$ parameter, if $m=m_{c_1}=m_{c_2}$.

It should be noted, that a large difference between the camera pose number and/or marker number can result in an unequal weighting of the cameras’ relevance in the optimization. Therefore, it is required that $m_{c_1}\approx m_{c_2}$ and $n_{c_1}\approx n_{c_2}$. Otherwise an appropriate error weighting approach should be introduced.

4.1.

Hardware Setup: Sensor and Calibration Target

The structured light sensor is shown in Fig. 4(a), comprising two monochromatic cameras (Allied Vision Manta G-895B POE) with telecentric lenses (Opto Engineering TCDP23C4MC096 with modified aperture) and a projector with entocentric lens (Wintech Pro4500 based on Texas Instrument’s DLP LightCrafter 4500). The projector is only used as feature generator, not used in the calibration routine and is therefore not addressed in this section.

Fig. 4

(a) Structured light sensor with telecentric stereo camera pair and entocentric projector as feature generator. (b) Experimental calibration target.

The telecentric lenses allow for the application of two cameras per lens, offering different magnification values. In the present scenario, the magnification $m=0.093$ is used, theoretically offering an FOV of $\sim 152.54\mathrm{mm}$ by 80.72 mm, when used with a 1 in CMOS sensor with a resolution of 4112 pixel by 2176 pixel and a pixel size of $3.45\mu \mathrm{m}$. The hardware configuration results in a pixel size on object side of $\sim 37\mu \mathrm{m}$. The sensor is not completely illuminated, as the lens offers a smaller aperture. The lenses’ DOF is $\sim 50\mathrm{mm}$, the telecentric range is smaller (about 20 mm), and the working distance is 278.6 mm according to the data sheet. The triangulation angle is manually adjusted to $\sim 45\mathrm{deg}$.

The calibration target is shown in Fig. 4(b). The target’s basis is formed by a stiff cardboard structure, forming a roof. Two simple planar plastic tiles with circle pattern are fixed on the rooftop sides with double-faced adhesive tape. The target patterns are printed onto an adhesive foil on a standard ink-jet printer and are adhered to the tiles. The dot marker pitch is 3 mm and the diameter is 2.25 mm.

4.2.

Calibration Results

The calibration target is captured in different poses (at least three poses per camera). It is not mandatory that both cameras acquire all images based on the exact same target poses as long as at least one image pair of the same pose exists. This image pair is necessary as it will be used to define the measurement coordinate system based on $\{O_1\}$. In the present scenario, $m_{c_1}=11$ poses are captured for the first and $m_{c_2}=13$ for the second camera. The marker number for camera one is $n_{c_1}=282$ per pose, and for camera two $n_{c_2}=281$ per pose. In consequence, an unequal error balancing due to a large difference in point correspondences can be excluded, but nevertheless should be checked by comparing the individual mean absolute projection error per camera. The first target pose is equal and captured by both cameras, being basis for the measurement coordinate system. The start values for the nonlinear refinement are determined for each camera independently.

4.2.1.

Scenario one: no start value correction

In the first scenario, the necessity of a potential start value correction is not monitored. Hereby, the effect of erroneous start values on the nonlinear refinement is meant to be illustrated. The corresponding calibration result is given in Fig. 5. The start values are listed in the left column, the refinement result in the right column. For the sake of readability and brevity, only exemplary parameters are given.

Fig. 5

Calibration result for exemplary parameters for scenario one. The start values for the first camera are estimated based on a mirrored point cloud and not corrected. ${{}^{O_1}\mathbf{T}}_{O_2,1}$ is used as start value for the stereo optimization.

${{}^{O_1}\mathbf{T}}_{O_2}$ is estimated independently for both cameras in the start value determination and should be ideally equal, as the target geometry is not changed in between the image acquisition for both cameras. A comparison of ${{}^{O_1}\mathbf{T}}_{O_2,1}$ and ${{}^{O_1}\mathbf{T}}_{O_2,2}$ shows a difference in sign [cf. to red (dot underline) and blue (wave underline) boxed values in Fig. 5]. It follows that ${{}^{O_1}\mathbf{T}}_{O_2,1}\approx {{}^{O_1}\mathbf{T}}_{O_2,2}\circ {\mathbf{T}}_{\mathrm{mir}}$, indicating that a mirrored point cloud either for the first or second camera was used to estimate the start values. (The approximately equal sign is used here, as a simple sign correction does only ideally result in the same matrices. Even in case of nonmirrored conditions, the different experimental data sets for both cameras result in slightly different matrix entries.) In the present scenario, the first camera’s point cloud is mirrored, which can be concluded from a yaw–pitch–roll decomposition (cf. Sec. 3.2.5). The nonlinear refinement based on Eq. (25) requires the choice of a single ${{}^{O_1}\mathbf{T}}_{O_2}$—either ${{}^{O_1}\mathbf{T}}_{O_2,1}$ or ${{}^{O_1}\mathbf{T}}_{O_2,2}$. This leads to large deviations when starting the optimization, as either the ${}^{C_1}{\tilde{\mathbf{T}}}_{O_1,i}$ or ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,i}$ matrices do not fit to the chosen calibration target shape in terms of ${{}^{O_1}\mathbf{T}}_{O_2}$. If ${{}^{O_1}\mathbf{T}}_{O_2,1}$ is selected (mirrored point cloud), the refinement results according to Fig. 5 are obtained. In this case, the nonlinear refinement starts with a mean projection error of 36.04 pixel.

A direct comparison of all start and refined values shows no great difference for the chosen parameters but reveals a change in sign for the truncated rigid body transformation ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1}$ for the first target pose [cf. to black (solid underline) boxed values in Fig. 5]. This change in sign happens also for all other target poses and matrices ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,i}$, whereas the erroneously chosen start value ${{}^{O_1}\mathbf{T}}_{O_2,1}$ does only slightly change in the optimization procedure and the critical signs do not change at all [cf. to red (dot underline) and green (dash underline) boxed values in Fig. 5]. The only way to reduce the projection error (if ${{}^{O_1}\mathbf{T}}_{O_2,1}$ is not changing) is therefore the adaption of the second camera’s locations in relation to the target, now in conformance with the mirrored point cloud. This is why the truncated matrices ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,i}$ change in sign [cf. Sec. 3.2.5, Eq. (20)].

Still, the resulting projection errors $e_{\mathrm{abs},\text{mean},1}$ and $e_{\mathrm{abs},\text{mean},2}$ are with about 0.28 pixel low (corresponds to $\sim 10\mu \mathrm{m}$ on object side). Also, the similar error results for both cameras indicate a balanced weighting of the cameras’ relevance in the nonlinear optimization. An analysis of the error histogram (not shown here) indicates a good model fitting and allows the conclusion that the optimizer converged into the desired minimum. This assumption is further supported as also the estimated lens magnification is consistent with the data sheet value of 0.093. The calibrated magnification can be obtained with help of the scaling factor $s$ and the pixel size $s_x$, resulting in $m=s·s_x=27.063\frac{\mathrm{px}}{\mathrm{mm}}·0.00345\frac{\mathrm{mm}}{\mathrm{px}}=0.09336$.

In this scenario, the minimum with erroneous signs is estimated and the mirror ambiguity problem not resolved (according to the results in Fig. 5). (According to Sec. 3.2.5, there are two mathematically equivalent minimum: one for a mirrored and one for the correct, nonmirrored sensor arrangement, just differing in signs.) The $r_{13}$ and $r_{23}$ components of ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1}$ change during parameter refinement (from mirrored to nonmirrored), resulting in a erroneous camera pose estimation [as outlined in Fig. 3(a) (cam 2′, instead 2)]. This is due to the choice of the mirrored target point cloud as start value in terms of ${{}^{O_1}\mathbf{T}}_{O_2,1}$ and the absence of a sign change during refinement (cf. to ${{}^{O_1}\mathbf{T}}_{O_2,1}$ in Fig. 5: $r_{13}$, $r_{23}$, $r_{31}$, $r_{32}$, $t_z$ do not change in sign). In consequence, a mirrored sensor arrangement is estimated, and all following measurements will result in mirrored point clouds.

If ${{}^{O_1}\mathbf{T}}_{O_2,2}$ is chosen as start value (describing a nonmirrored calibration point cloud), the initial mean projection error is higher (111.70 pixel). The optimizer is not converging toward a global minimum, and the routine is aborted with a mean absolute projection error of about 13 pixel. This was not to be expected, as also in this case a sign adaption (in this case, for the matrices ${{}^{C_i}\tilde{\mathbf{T}}}_{O_1,i}$) would result in a global minimum; this time even for the accurate sensor arrangement. The result indicates a basic problem when ignoring affine mirror ambiguity. The necessary sign adaption is not always successful.

4.2.2.

Scenario two: start value correction

In the second scenario, ${\mathbf{T}}_{\mathrm{mir}}$ is used to correct the start values of the first camera. The corresponding calibration result is given in Fig. 6 and extended by the distortion parameters for the first camera. The result is obtained when using the corrected ${{}^{O_1}\mathbf{T}}_{O_2,1}$ as start value, resulting in an initial mean projection error of 0.75 pixel, around 35 pixel lower than in the uncorrected scenario.

Fig. 6

Calibration result for exemplary parameters for scenario two. The start values for the first camera are estimated based on a mirrored point cloud but are corrected by ${\mathbf{T}}_{\mathrm{mir}}$. ${{}^{O_1}\mathbf{T}}_{O_2,1}$ is used as start value for the stereo optimization.

Now not only the absolute values of ${{}^{O_1}\mathbf{T}}_{O_2,1}$ and ${{}^{O_1}\mathbf{T}}_{O_2,2}$ are similar but also possess the same signs [cf. to red (dotted underline), blue (wavy underline), and green (dashed underline) boxed values in Fig. 6] even after refinement. (Just the sign of the $y$ value is changing but is very close to zero. This change is not connected to the mirror effect). The same applies to the truncated matrices ${{}^{C_1}\tilde{\mathbf{T}}}_{O_1,i}$ and ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,i}$; the critical signs do not change in the refinement procedure, meaning that the start values of both cameras have been successfully combined.

The error histogram for the second camera is shown in Fig. 7(a). The error is approximately normally distributed for the $v$ direction, whereas the $u$ direction deviates from a normal distribution. The reason for this cannot be assessed conclusively but could be due to a slightly biased target pose distribution.

Fig. 7

(a) Error histogram for the second camera. (b) Distortion model for the first camera: only the illuminated sensor area is depicted. The corresponding distortion coefficients ${\mathbf{k}}_1$ are given in Fig. 6.

Altogether, error distribution and mean absolute projection errors are equal to the previous scenario without correction, which is comprehensible due to mathematical equivalence of mirrored and nonmirrored solution. This mathematical equivalence should not be confused with a physical equivalence. The determined parameters in the noncorrected scenario are false in sign.

In addition, the lens distortion for the first camera is shown in Fig. 7(b). The telecentric lens does not allow for a complete sensor illumination, resulting in masked areas near the right and left sensor boundaries. This is why the displayed sensor area is reduced by about 500 pixel from the sides. Altogether, the lens distortion is relatively low, as the distortion model introduces a pixel correction distinctly below 0.4 pixel for the greater part of the sensor. Even lower corrective effect is introduced by the distortion model for the first lens, confirming the assumption of low distortion generated by high quality telecentric lenses.

Noteworthy is also the matrix entry (1,1) of ${{}^{C_2}\tilde{\mathbf{R}}}_{O_1,1}$ [as part of the start value of ${{}^{C_2}\tilde{\mathbf{T}}}_{O_1,1}$, indicated by single black (solid underline) box], as it results in a deviation to an orthonormal basis. This might be due to numerical inaccuracies when performing the Euclidean upgrade but apparently had no effect on the parameter refinement in the next step, as the refined matrix represents an orthonormal basis.

The plausibility of the optimized rigid body transformation ${{}^{O_1}\mathbf{T}}_{O_2}$ (here in terms of ${{}^{O_1}\mathbf{T}}_{O_2,1}$) is evaluated by analyzing the angles between the axes of the two coordinate systems $\{O_1\}$ and $\{O_2\}$. As the second orthonormal basis of the refined rotation matrix ${{}^{O_1}\mathbf{R}}_{O_2,1}$ (as part of ${{}^{O_1}\mathbf{T}}_{O_2,1}$) is nearly ${(5.463\times {10}^{-3},0.99998,-4.422\times {10}^{-3})}^T\approx {(\mathrm{0,1},0)}^T$, the angle between the $y$ axis of $\{O_1\}$ and $\{O_2\}$ is approximately zero. This is in good agreement with the obvious orientation of the two planes in Fig. 4(b), in which the $y$ axes are nearly in parallel. The angles between both $x$ and $z$ axes are approximately equal and about 42.80 deg (obtained by scalar product between corresponding orthonormal bases). This is again in agreement with the previous result, as the rotation in order to transfer $\{O_1\}$ to $\{O_2\}$ is performed around the $y$ axis.

The triangulation angle between the cameras has been manually set to 45 deg and is validated by calculating the angle between the sensors’ normal axes based on the calibration result. To this end, the third orthonormal bases of ${{}^{C_1}\tilde{\mathbf{T}}}_{O_2}$ and ${}^{C_2}{\tilde{\mathbf{T}}}_{O_2}$ are calculated. The procedure is exemplary given for the first camera. The cross product of ${}_{C_1}{\mathbf{r}}_1$ and ${{}_{C_1}\mathbf{r}}_2$ yields ${}_{C_1}{\mathbf{r}}_3$. The triangulation angle $\theta$ is calculated based on the scalar product of ${{}_{C_1}\mathbf{r}}_3$ and ${{}_{C_2}\mathbf{r}}_3$, resulting in an angle of 46.425 deg. This is in good agreement with the roughly measured value of 45 deg.

Altogether, postulated model and experiment are in good agreement. If ${{}^{O_1}\mathbf{T}}_{O_2,2}$ of the second camera is used as start value, the initial mean projection error is even a bit lower (0.72 pixel).