2.1

Pinhole camera model and camera calibration

The foundation for the proposed approach is the perspective projection of a 3D Manhattan world scene modelled by an ideal pinhole camera. The pinhole camera model defines the transformation of the three-dimensional Euclidean space onto the two-dimensional image plane according to the principle of central projection. If the 3D coordinate system of the world is set equal to that of the camera with the origin at point C, then by using a notation of Refs. 2, the projection can be described compactly with the equation

Image point x and world point X_C are represented by homogeneous vectors. Both are uniquely determined up to a scale factor. The index C shows that the world point X_C is defined in the camera coordinate system. The 3×3 matrix

is called the camera matrix or the calibration matrix of the pinhole camera model. It consists in its basic form, with zero skew and unit aspect ratio, of the focal length f, the distance of the image plane from the camera centre, and of the principal point (p_x, p_y). Both are expressed here in terms of pixel dimensions.

If the point X is defined in a 3D world, for example in a Manhattan world, and the camera pose (orientation and position) in this world is defined with the rotation matrix R and the translation t, then X must first be rigidly transformed into the camera coordinate frame with before it is projected (1) onto the image plane. In summary, the well-known algebraic relationship of the projective camera is derived as (see Fig. 2, left)

Figure 2.

Pinhole geometry of a camera placed at the coordinate origin C. Camera and Manhattan world coordinate frames are related via rotation and translation (left). Formation of a vanishing point as intersection of two imaged parallel world lines (right).

Traditional calibration approaches, such as in Refs. 3, calculate the unknown camera parameters K, R, t with the help of a geometric calibration target whose known world points {X_i} and corresponding image points {x_i} are used to obtain the solution of (3). Also for the method presented in this article, the cameras are precalibrated intrinsically (K) according to this principle.

2.2

Vanishing points in a Manhattan world

A set of parallel lines with direction d defined in the 3D camera coordinate system intersect in the ideal point, the point at infinity, X_∞ = (d^T, 0)^T of the associated projective space. The perspective projection of this ideal point on the image plane is called vanishing point. It can be calculated by inserting the ideal point into the equation (1) as υ = Kd. Analogously, the inverse projection of the vanishing point is defined as a ray through the camera centre and the point υ with the direction d = K⁻¹υ as shown in Fig. 2, right. Consequently, for all vanishing points found in the image of a Manhattan world, which belong to three mutually perpendicular directions, the vectors d₁ = K⁻¹υ₁, d₃ = K⁻¹υ₃ and d₃ = K⁻¹υ₃ are parallel to the axes of a Manhattan world which are orthogonal to each other (see Fig. 3, left).

Figure 3.

The derivation of the rotation (left) and the translation (right) from vanishing points. The 3D points O′ and P′ lie on the image plane and are defined in the camera coordinate system with its origin in C.

2.3

Derivation of the camera pose’s rotation matrix R

The normalized vectors d₁, d₂, d₃ derived according to the above considerations form an orthonormal basis of a world coordinate system

The sought rotation matrix R = [r₁, r₂, r₃] is composed of these basis vectors. To construct a valid rotation in a right-handed coordinate system, the order of the r₁, r₂ and r₃ has to be chosen accordingly.

2.4

Derivation of the camera pose’s translation vector t

The translation vector t cannot be computed unambiguously from an image without additional knowledge of the scene. A practical way to estimate the translation is to use known distances in the scene. For example, two known point pairs {x_i, X_i} can be used to solve (3) with the previously determined rotation matrix R. Consequently, errors from the calculation of the rotation could propagate in the determination of the translation.

An alternative was presented in Refs. 4. The required parameters consist of a known direction d that is parallel to one of the axes of the Manhattan world, the real length of a line segment on the direction d, and the image coordinates of the start and end points of the depicted line segment. The principle of this approach is shown in Fig. 3, right.

From the similarity of the triangles CO′I and COP, the distance of the point O from the camera centre is and thus the translation vector t can be determined as

3.1

Procedures for line segment extraction

An established and widespread approach uses the robust Canny edge detector and a parametric line search, e.g. in Hough space, to extract lines from an image. Its susceptibility to faults (i.e false-positive lines), especially in regions with fine textures, leads to erroneous results of the subsequent vanishing point detection.

We evaluated two alternative methods, the Fast Line Detector⁵ (FLD) and the Line Segment Detector⁶ (LSD). Both output line segments with sub-pixel accuracy and produce subjectively correct results. The FLD processes the results of the Canny detector by using a co-linearity constraint to track straight line segments along contours. The LSD splits the image into so-called line support regions, i.e. regions of similar local gradient orientation. These regions are merged into the line segments by a region growing approach.

4.1

Line segment extraction

Utilizing an intensity image as an input, both algorithms (LSD and FLD) parameterize line segments as start and end points with subpixel-precision. Some parameters, e.g. minimum line length, may be adjusted depending on the resolution of the input image and scene properties to obtain the best results.

For the example image (Fig. 4, left) FLD outputs 563 and LSD 611 line segments. According to the subjective evaluation, the consistency of the results is similar. Statistical evaluation of the line segments length shows, that the LSD favours shorter line segments with a smaller length variance compared to the FLD. Since no preference on this basis can be made at this point, the final method selection takes place in the context of the subsequent processing steps.

4.2

Edge clustering and vanishing point calculation

Estimated line segments were input to the J-Linkage algorithm and labelled to corresponding vanishing points as output. Since a Manhattan world was assumed, the three clusters with the largest number of assigned line segments correspond to the wanted Manhattan world axes (vanishing points).

The intersection of lines in a cluster was computed using a least squares method. In addition to the calculated vanishing point, the root-mean-square (RMS) over the smallest distances of the lines to the intersection (vanishing point) was computed. This represents the accuracy of the line detectors and gives an indication of the validity of the estimated vanishing points in case of degenerate camera orientation (Sec. 4.3). Some results of the here favoured FLD and the LSD methods are shown in the Tab.1.

Table 1.

Comparison of two favoured line extraction methods with coordinates of the estimated vanishing points and their accuracy for the scene shown in Fig. 4. The value of #v is the number of distinctive vanishing points found by J-Linkage algorithm.

The RMS of the LSD line segments was slightly smaller than that of the FLD in every case. The J-linkage output nearly twice as many clusters for lines extracted with FLD (28) as for LSD (11). Both results depend on the precision of the extracted line segments. The evaluation of additional images resulted in similar differences in favour of the LSD, so it is the recommended method for the line segmentation step.

4.3

Estimation of the camera pose’s rotation

For the calculation of the rotation matrix according to (4), the vanishing points were arranged clockwise with respect to the principle point. For the direction vectors computed from the left image in Fig. 4, the pairwise angles between the basis vectors were: and . The resulting rotation matrix was orthogonalized with the help of Rodrigues’ rotation formula.

Depending on the orientation of the camera in the Manhattan world, a rotation matrix from (4) can not always be estimated. These cases occur when one or two, hence all three, axes of the camera coordinate system are parallel to axes of the world coordinate system. Thus, the projections of parallel lines in the world are also parallel on the image plane and the associated vanishing point is at the point of infinity.

If only one infinite vanishing point is present, the corresponding direction vector can be calculated as the cross product of the other two. The second case should be obviated when possible, for example by changing the camera pose, as further analysis is otherwise impossible.

4.4

Estimation of the camera pose’s translation

The computation of the equation (5) of camera pose’s translation is trivial. Only point I can possibly be misinterpreted. It results as an intersection of the rays and O + λ₂d in 3-space as shown in Fig. 3, right.

We selected points O and P on the upper side of the robot frame (see Fig. 5, left), determined their image coordinates and computed the translation vector t according to the equation (5). The length of the side of the frame is = 880 mm. Thereby the distance from the camera centre to the origin of the Manhattan world is ||t|| = 2411.37 mm and the orientation, for easier interpretation presented as Euler angles (z, y’, x” convention), is: yaw = 138.98°, pitch = −0.68°, roll = 66.72°.

Figure 5.

Input image (left), line clustering based on the lines extracted with FLD (middle) and LSD (right). The edges are shown in colour according to the assigned vanishing points. Black lines could not be assigned to vanishing points of the imaged Manhattan world.

4.5

Estimation of the relative pose between two cameras

The motivation for the proposed scene-based calibration was not only to estimate the exterior orientation of multiple cameras in a common world frame but to determine their relative orientation and position to each other. Given a camera pair with known extrinsics, a selected point in the world is transformed to the respective camera frames with equations and . Substitution of the X from the second equation with the X from the first one gives us the orientation and translation of the camera C₂ with respect to C₁ as

It should be noted, that the R₁, R₂, t₁ and t₂ have to be defined with respect to the same coordinate system. In the procedure described in Sec. 4.3 the rotation matrices were computed by manually assigning the world axes. This step will be automated with image-based matching methods in the future.