Automatic detection of diabetic foot complications with infrared thermography by asymmetric analysis

Chanjuan Liu; Jaap J. van Netten; Jeff G. van Baal; Sicco A. Bus; Ferdi van der Heijden

doi:10.1117/1.JBO.20.2.026003

11 February 2015 Automatic detection of diabetic foot complications with infrared thermography by asymmetric analysis

Chanjuan Liu, Jaap J. van Netten, Jeff G. van Baal, Sicco A. Bus, Ferdi van der Heijden

Author Affiliations +

Journal of Biomedical Optics, Vol. 20, Issue 2, 026003 (February 2015). https://doi.org/10.1117/1.JBO.20.2.026003

Abstract

Early identification of diabetic foot complications and their precursors is essential in preventing their devastating consequences, such as foot infection and amputation. Frequent, automatic risk assessment by an intelligent telemedicine system might be feasible and cost effective. Infrared thermography is a promising modality for such a system. The temperature differences between corresponding areas on contralateral feet are the clinically significant parameters. This asymmetric analysis is hindered by (1) foot segmentation errors, especially when the foot temperature and the ambient temperature are comparable, and by (2) different shapes and sizes between contralateral feet due to deformities or minor amputations. To circumvent the first problem, we used a color image and a thermal image acquired synchronously. Foot regions, detected in the color image, were rigidly registered to the thermal image. This resulted in 97.8%±1.1% sensitivity and 98.4%±0.5% specificity over 76 high-risk diabetic patients with manual annotation as a reference. Nonrigid landmark-based registration with B-splines solved the second problem. Corresponding points in the two feet could be found regardless of the shapes and sizes of the feet. With that, the temperature difference of the left and right feet could be obtained.

1. Introduction

Diabetes mellitus (DM) is one of the major health care problems worldwide and continues to increase in population and significance.¹ Foot diseases are common and costly complications of DM. Approximately 15% to 25% of patients with DM eventually develop a foot ulcer.² This is one of the key complications of DM: if not adequately treated, the risk of amputations and mortality is increased.³ Early identification and subsequent treatment of diabetic foot complications and its presigns, such as ulceration, inflammation, callus formation, and blisters, are fundamental in the prevention of these devastating consequences. Timing is essential for early identification. However, frequent risk assessment by health care professionals is costly and impractical. Manual examination of patients has its limitations because of various health impairments caused by DM such as peripheral neuropathy, visual impairment, limited joint mobility, and cognitive impairments.⁴ Automatic assessment and detection of diabetic foot complications in a noninvasive, noninteractive, and easy to use manner may be a major benefit for foot care. The objective of our project is to develop technology that enables automated foot assessment in order to detect diabetic foot complications. Such a technology could then be applied in an intelligent (tele)medicine system to be deployed at the patients’ homes, at health centers, or perhaps at hospitals. We investigate three possible imaging modalities: hyperspectral imaging, photometric stereo imaging, and thermal imaging. Within the context of the project, the current paper only addresses the image processing and pattern analysis needed for the detection of complications in infrared (IR) thermographic images. The engineering of the telemedicine apparatus, suitable for use in, for instance, a home environment and the exploration of its usability and cost effectiveness are outside the scope of this paper.

1.1.

Related Work

1.1.1.

Telemedicine using digital photography

Frequent foot screening for risk assessment is possible by means of telemedicine systems based on digital photography.⁵^,⁶ However, inflammation and infection, which are the vicarious markers of diabetic foot complications, are difficult to assess using digital photography.⁵ Furthermore, such systems are not yet capable of automatic detection of diabetic foot complications. This limits their application and implementation.

1.1.2.

Thermography

Associations have been found between increased plantar foot temperature and the occurrence of diabetic foot complications.⁷ Clinical studies on the home monitoring of plantar foot temperature have shown that frequent temperature assessment and treatment in the case of temperature differences $> 2.2 ° C$ , between a foot region and the same region on the contralateral foot, can prevent diabetic foot complications.⁸^,⁹ Thus, thermography is a promising modality for an intelligent telemedicine monitoring system.

The technologies for temperature measurement of the plantar foot fall in three categories: handheld dermal IR thermometers,⁸^,⁹ IR camera systems,¹⁰^–¹² and liquid crystal thermography (LCT).¹³^,¹⁴ The use of handheld dermal IR thermometers in the home environment has been validated by randomized controlled trials for the prevention of recurrent diabetic foot ulceration.⁸^,⁹ The shortcoming of this technology is that the temperature is measured manually on specific spots on the foot. This makes it subjective, and it is impossible to obtain the temperature distribution of the whole foot. Furthermore, this technology misses the opportunity for automatic detection of diabetic foot complications. Compared with LCT, IR camera systems have the advantage of being noncontact, which prevents unwanted pressures and the transmission of pathological organisms.¹⁵ The temperatures of noncontact foot regions, such as the medial arch, can be easily measured with IR camera systems. Additionally, it is capable of measuring the dorsal side of the foot as well. As such, IR camera systems show a greater potential for telemedical applications and they will be the focus of this article.

1.1.3.

Asymmetric analysis

The methodology for attaining temperature differences between corresponding areas on the left and right feet, can be referred to as “asymmetric analysis.”¹¹^,¹² Generally, three steps are involved:

1. Foot segmentation: Extract the left and the right feet from the background.
2. Feet registration: Register the two feet to associate areas of one foot with the corresponding areas of the contralateral foot.
3. Detection: Compare the temperature of the associated areas: if the difference between the temperatures of two associated areas is larger than a certain threshold, then one of these two areas is assumed to be at risk.

Kaabouch et al. conducted studies using thermal images to detect neuropathic ulceration, combining a genetic automatic thresholding algorithm with “asymmetric analysis”¹¹ and “line scanning.”¹² The authors noted that the asymmetric analysis tended to find false abnormal areas when the left and right feet had different sizes and shapes. In these cases, noncorresponding areas were compared. The “line scanning” algorithm was proposed to overcome this limitation, but this algorithm was not targeted for strong foot deformations such as the one caused by minor amputations. The method was correspondingly validated: diabetic feet with strong deformations were not included. However, foot deformations and/or minor amputations frequently cause morphological differences between the left and right feet in patients at high risk for developing diabetic foot complications.³ In these cases, the association between left and right parts of the feet is nontrivial. Another problem is in the segmentation of the feet in the thermal image. It cannot be prevented that sometimes the temperature of a part of the foot is similar to the ambient temperature of the setup.¹⁶ The genetic automatic thresholding techniques cannot accurately detect these foot boundaries.¹⁷ For instance, in the case in Fig. 1(a), due to poor blood perfusion the temperature of the top of the toes is almost equal to the room temperature. A stabilization of the temperature of the setup below the ambient temperature is not desirable because of the discomfort for the patient and for technical reasons.

Fig. 1

Examples of acquired thermal images with their RGB images. In example 1, the patient has cold toes, which have temperatures comparable with the background. Ankles have temperatures comparable with the feet (green ellipses). The ankles are also visible in the RGB image (green ellipses). In this modality, segmentation is difficult but not impossible, as shown in Fig. 7. (a) Example 1; (b) RGB image for example 1; (c) example 2; and (d) RGB image for example 2.

This article aims to overcome the shortcomings mentioned above. The improvements are as follows:

• More accurate foot segmentations in the thermal images even when there was no clear thermal contrast between foot and background. This was due to the accompanying color images that guided the segmentation.
• A better association between corresponding areas in the left and right feet regardless of the foot poses, positions, shapes, or sizes, and even with amputated parts. This was obtained by using nonrigid landmark-based registration B-splines.¹⁸^,¹⁹
• The validation of the image analysis was applied to a group of high-risk diabetic patients that showed significant asymmetries between left and right feet, rather than to subjects with healthy feet.

1.2.

Outline of the Article

In Sec. 2, the experimental setup for acquiring thermal images and color images from patients with DM, the measurement procedure, patient recruitment, and data collection are described. The methodology of conducting asymmetric analysis to detect foot complications using the assistance of color images and nonrigid landmark-based registration with B-splines is introduced in Sec. 3. Analysis, results, and discussion follow in Sec. 4. Finally, conclusions and future plans are presented in Sec. 5.

2. Materials and Measurements

Figure 2 presents an experimental setup that housed the acquisition devices for thermal imaging and RGB imaging. As the setup also housed hyperspectral imaging and photometric stereo imaging, thereby offering a versatile platform for experimentation, it was rather large. In a later stage of development, a user friendly, cheaper, and down-scaled version can be engineered for further clinical evaluation and effectiveness studies. The equipment for the thermal imaging modality consisted of an IR camera, FLIR SC305, and a commercial digital RGB camera, Canon EOS 40D. A separate RGB camera was chosen instead of an IR camera with a built-in visual camera to have full freedom in selecting the two cameras with their desired specifications. The IR camera was placed at a distance of 860 mm from the object plane. The camera’s specification is as follows. It has a resolution of $320 \times 240$ pixels and covers a field of view of $400 \times 350 {mm}^{2}$ . The pixel distance on the foot soles is about 1.25 mm. The spectral range is 7.5 to $13 μ m$ . The temperature range is from $- 20^{°} C$ to $+ 120^{°} C$ . The accuracy is $\pm 2^{°} C$ . With external thermal references in the range from $+ 20^{°} C$ to $+ 38^{°} C$ , we were able to reduce this to $\pm {0.25}^{°} C$ . Before deployment, we tested the nonuniformity error over the image plane (it was not specified), and this appeared to be less than 0.25°C. The digital camera, placed at the same distance as the IR camera, covered the same field of view and acquired images of $1936 \times 1288$ pixels (pixel distance: 0.2 mm). The two cameras were mounted in such a way that they had the same field of view, but from slightly different view points.

Fig. 2

The experimental setup for foot scanning: (a) the schematic of the experimental setup with two cameras installed, which are the thermal camera (upper) and the RGB camera (lower). (b) The appearance of the experimental set and (c) positioning of patients’ feet during measurement.

Patients were recruited from the multidisciplinary diabetic foot clinic of the Hospital Group Twente, Almelo, The Netherlands. The patients included in this study were all diagnosed with DM and showed diabetic foot complications, e.g., callus, blisters, redness, ulceration, or had a history of ulceration. The average age of the 76 included patients, 64 male and 12 female, was 66 years ( $SD = 12$ ). Among them, there were 7 patients with type I DM and 69 patients with type II DM.

All patients were asked to remain in a seated position with bare feet for 5 to 10 min before the measurements. This was to achieve the equilibrium of the foot temperature. The feet were then placed on the foot support of the experimental setup. A hospital cloth was placed over the legs of the patients for hygiene reasons and one piece of black cloth was draped over the shield of the setup to block external light entering the setup and to provide a homogeneous background. During thermal imaging, all illumination sources inside the experimental setup were turned off and turned on back during the acquisition of color images with the digital camera. A 30-min interval was assured between measurements of different patients to eliminate heat residue from the illumination or from the body of the previous patient.

For each patient, a live assessment form of the plantar surface of both feet was completed by wound care specialists. This live assessment was used as a reference for the validation of the automatic detection. Examples of the acquired thermal and RGB images are given in Fig. 1.

3. Methodology for Automatic Detection of Diabetic Foot Complications

3.1.

Overview of the Proposed Methodology

The proposed methodology for patient-specific image analysis to automatically detect diabetic foot complications is illustrated in Fig. 3.

Fig. 3

Schematic flowchart of the proposed methodology for automatic detection of diabetic foot complications.

The first step after image acquisition was the segmentation of the feet. As explained in Sec. 1, the temperature difference between some foot parts and the ambient temperature can be so small that accurate segmentation is difficult to perform directly in the thermal image. Thus, foot segmentation in the color images was preceded to assist the segmentation in the thermal images. Details about foot segmentation in the color images can be found in Sec. 3.2.

The next step is the alignment of the RGB images and thermal images. If the object was situated in a single plane, a projective transform would suffice. Since we intended to acquire a thermal image without any additional pressure on the feet, no sidelong support bars were mounted that would force the feet to a standard position and orientation in the imaging system. As a result, the surfaces of the foot soles only approximately coincided with the plane that was used for the projective registration. If the object’s depth varies with a deviation of $Δ Z$ around a mean plane depth of $Z$ , and the distance between the view points is $t$ , then, despite the projective transform, the parallax error could be as high as $(t Δ Z) / Z$ . Thus, if the object distances are large compared with $t$ , the parallax error is negligible. This condition is usually met in commercial handheld IR cameras with a built-in visual camera, e.g., the FLUKE TiR27. In our case, the object distances were in the range from 80 to 90 cm, and the distance between the view points is about 15 cm, yielding a parallax error on the order of 1 cm. In a down-scaled version of the device, the view point distance could be reduced to about 5 cm (like in the FLUKE TiR27), but if the object distance were reduced proportionally, the parallax problem would remain.

The alignment was done in two steps. Before the actual deployment of the device, the parameters of a two-dimensional (2-D) projective transformation that maps the RGB images onto the thermal images were determined from 12 control points. These points were all located in a single plane in front of the cameras at a distance that matched the mean object distance. Details of the fine optimization can be found in Sec. 3.3.

The performance evaluation of the segmentation in the color images and that in the thermal images was achieved through comparison with manually segmented references provided by one of the researchers. The method for manual segmentation with custom-made software has been described elsewhere.²⁰

The last step was to conduct asymmetric analysis to detect the temperature differences of contralateral regions on the left and right feet. To facilitate this, a good registration between left and right feet, irrespective of feet positions, poses, shapes, or sizes, was essential. Nonrigid landmark-based registration was performed for this purpose. Details of this part can be found in Sec. 3.4.

3.2.

Foot Segmentation in Color Images

The aim of the foot segmentation with acquired color images is to determine whether a color pixel is on a foot or on the background.

3.2.1.

Machine learning for color image segmentation

RGB is the most common color representation for storing images. Other color representations can be obtained by a linear or a nonlinear transformation of RGB space. Although many studies have been done on the use of different color representations in segmentation, it is hard to come to a conclusion about which color space is best.²¹ It has been observed that the skin colors differ more in intensity than in chrominance.²² Thus, it is a common practice to ignore the luminance in skin color detection. In this study, we performed segmentation on the six color representations that are commonly used in human skin detection: RGB, normalized RGB (rgb), RGB-ratio, HSV, CIE L*a*b*, and YCbCr.²²

Machine learning techniques play an important role in image segmentation. The goal is to partition the image plane into $K$ disjoint regions. Suppose that the three-color channels of the $ℓ$ ‘th pixel in the image are represented by a three-dimensional (3-D) vector $x_{ℓ} \in R^{3}$ and that the image contains $L$ pixels, so that $ℓ = 1,2, \dots, L$ . The segmentation is accomplished by assigning to each pixel $x_{ℓ}$ a class label $k_{ℓ}$ . The set of possible class labels is $k_{ℓ} \in {1, \dots, K}$ . Each class corresponds to a region.

The techniques for machine learning can be divided into two groups: supervised and unsupervised learning.²³ In supervised learning, the segmentation parameters are obtained during the design stage of the segmentation by means of preselected, fixed training data. In contrast, in unsupervised learning, the segmentation parameters are calculated during the utilization of the segmentation by means of the actual data. As such, unsupervised learning has the ability to adapt itself to the statistics of this actual data.²³

In this study, all patients had a wide variety in skin health status and accordingly in colors of the skin. It is difficult to cover all the possible variations in a training set. As unsupervised learning adapts itself to the data from each individual image, unsupervised learning was selected. Two common techniques in this category were chosen, $K$ -means clustering and expectation-maximization (EM) clustering.

$K$ -means clustering is one of the simplest unsupervised learning algorithms, aiming to define $K$ centroids for $K$ clusters. $K$ is predefined. At the initialization step, the $K$ centroids are randomly picked. A binding step follows to associate each object in the training set to the nearest centroid. When no point is pending, the $K$ centroids are recalculated. With the new $K$ centroids, a new binding step is done. This procedure iterates until the $K$ centroids do not move any more. $K$ -means minimizes the sum of the distances over all objects in $X$ to the nearest center $m_{k}$ , and the algorithm converges to a (local) minimum.²³

The EM algorithm is a popular technique in unsupervised learning.²³ We used the EM algorithm (as defined in Algorithm 1) for the case where the model is a mixture of Gaussian probability densities. Each class is represented by a Gaussian density $N (x | m_{k}, C_{k})$ with mean $m_{k}$ and covariance matrix $C_{k}$ . The prior probabilities of the classes are denoted by $π_{k}$ . We implemented two versions of the EM algorithm; one with class independent covariance matrices, i.e., $\forall k : C_{k} = C_{0}$ , and the other with class-dependent matrices. The first one is referred to as EM-LDC (linear discriminant classifier), and the second one as EM-QDC (quadratic discriminant classifier):

Algorithm 1 EM for image segmentation.
- • Inputs: Data set with $L$ pixels (3-D vectors) $x_{ℓ}$ . Desired number of clusters $K$ .
- • Output:
  - - A model consisting of the parameters of the mixture of Gaussians, i.e., for $k = 1, \dots, K : m_{k}, C_{k}, π_{k}$ .
  - - For each pixel $ℓ$ and each class $k$ , an ownership variable $y_{ℓ, k}$ such that $Σ_{k} (y_{ℓ, k}) = 1$ . The ownership indicates to what degree pixel $x_{ℓ}$ is attributed to cluster $k$ .
- • Initialization:
  - - Determine $K$ centers $m_{k}$ from the $L$ pixels using the $K$ -means algorithm.²³
  - - Initialize the ownerships $y_{ℓ, k}$ as the distance of pixel $ℓ$ to the class center $m_{k}$ relative to the sum of distances to all class centers.
- • Loop while the model improves:
  - - For all $k$ : (re-)estimate the prior probabilities:
    Eq. (1)
    $π_{k} = \frac{1}{L} \sum_{ℓ = 1}^{L} y_{ℓ, k} .$
  - - For all $k$ : (re-)estimate the means:
    Eq. (2)
    $m_{k} = \frac{1}{π_{k} L} \sum_{ℓ = 1}^{L} y_{ℓ, k} x_{ℓ} .$
  - - For all $k$ : (re-)estimate the covariance matrices:
    Eq. (3)
    $C_{k} = \frac{1}{π_{k} L} \sum_{ℓ = 1}^{L} y_{ℓ, k} (x_{ℓ} - μ_{k}) {(x_{ℓ} - μ_{k})}^{T} .$
  - - Equalize the covariance matrices (EM-LDC only):
    Eq. (4)
    $C_{0} = \sum_{k = 1}^{K} π_{k} C_{k} C_{k} := C_{0} .$
  - - For all $k$ and all $ℓ$ : re-estimate the ownerships:
    Eq. (5)
    $y_{ℓ, k} = \frac{π_{k} N (x_{ℓ}, m_{k}, C_{k})}{\sum_{m = 1}^{K} π_{m} N (x_{ℓ}, m_{m}, C_{m})} .$

There are two different regions in this application, foreground (foot regions) and background. Consequently, the number $K$ of clusters should also be set to 2. However, since the background was made up of different parts, it could be advantageous to increase this number. It was set to 2, 3, and 4, respectively. In the case of more than two clusters, only one of them was identified as foreground (foot region). Postprocessing of the segmentation was performed by morphological operations. Small objects either in the foreground (foot area) or in the background were removed.

All the image processing and pattern classification were performed on the MATLAB R2012b platform with an additional toolbox PRtools V5²³ in 64 bits Windows 7.

3.2.2.

Segmentation evaluation

To select the number $K$ of desired clusters, and the most suitable algorithm ( $K$ -means, EM-LDC, or EM-QDC), a quantitative measure of the segmentation performance is necessary. Two measures were included in the evaluation step for the segmentation:

Sensitivity and specificity

Sensitivity and specificity are the measures of the performance of a two-class classification algorithm. Sensitivity, in our case, measures the ratio of the pixels inside the feet regions that are correctly labeled. Sensitivity represents the ability of the segmentation method in finding the feet. Specificity, on the other hand, measures the portion of negatives (the background, in our case) that are correctly identified:

Eq. (6)

Sensitivity = \frac{| F_{R} \cap F_{S} |}{| F_{R} |},

Eq. (7)

Specificity = \frac{| B_{R} \cap B_{S} |}{| B_{R} |},

where

B_{R}

and

F_{R}

represent the background and foreground of the reference image, while

B_{S}

and

F_{S}

denote the background and foreground according to the algorithm under test, respectively.

| \cdot |

is the set cardinality.

Mean distance error of wrongly classified area ${\bar{D}}_{EA}$

This reflects the cost of wrong segmentation. For each pixel on the image, the Euclidean distance was calculated to the nearest foot contours resulting in a distance map $D_{map}$ as shown in Fig. 4. The distance of each pixel that was wrongly assigned was summed up and averaged to get the mean distance error ( ${\bar{D}}_{EA}$ ):

Eq. (8)

{\bar{D}}_{EA} = \frac{\sum_{Nwb} D_{map} W_{B} + \sum_{Nwf} D_{map} W_{F}}{Nwb + Nwf},

where

W_{B}

represents a binary map containing the foot pixels that are wrongly assigned to the background, i.e.,

F_{R} \cap B_{S}

with

Nwb = | F_{R} \cap B_{S} |

, and

W_{F}

represents a binary map containing the background pixels that are wrongly assigned to the foot, i.e.,

B_{R} \cap F_{S}

with

Nwf = | B_{R} \cap F_{S} |

.

Fig. 4

Examples of Euclidean distance map of the reference foot contours (Unit: pixel): (a) $D_{map}$ of example 1; and (b) $D_{map}$ of example 2.

3.3.

Registration Optimization

As explained in Sec. 3.1, deviations from the registration plane caused misalignments between color and thermal images. The main disagreement was located at the toes area. This is explained by the fact that when the patient’s feet were well positioned onto the foot supporters underneath the heels, the heels were near the projective plane. Examples of the misalignments can be found in Fig. 5(a).

Fig. 5

(a) Examples of ill-positioned feet contours, mapping directly from color image segmentation. (b) Illustration of the perpendicular line profiles of the foot contour, which defined the polygonal mask in and out of the foot contour. Note, for illustrative purposes, the left panel displays the longer profile lines ( $length = 51 pixels$ ) and larger polygonal window than actually used.

A patient-specific fine registration was needed to correct this possible misalignment. A rigid transformation consisting of translation, rotation, and vertical scaling was assumed to suffice. Four parameters are needed to define this transformation. Horizontal scaling was not needed as the deviations between foot soles and projective plane were mainly slants in the longitudinal direction of the foot soles.

To optimize the four parameters, a quantitative criterion was needed. We selected a criterion based on the observation that the statistics of the temperatures in a region inside the foot will differ the most when compared with statistics of a region entirely outside the foot. We defined pairs of small regions positioned at both sides of the hypothesized foot contour. The difference between histograms of the temperatures in these opposite regions should be maximized. By the aggregation of a measure of this difference, calculated along different points on the hypothesized foot contour, a global optimization criterion was found. Although parts of the foot, such as the toes, had low contrast against the background, the contour points at these regions had low importance in driving the registration optimization.

The regions were defined as a set of foot contour points ${(x_{c}, y_{c}) | c = 1, \dots}$ resulting from the color segmentation. A polygonal perpendicular was placed at each contour point as shown in Fig. 5. The polygon was split into two parts, $R_{in} (x_{c}, y_{c})$ and $R_{out} (x_{c}, y_{c})$ , representing the parts inside and outside the foot area. The measure of difference between $R_{in}$ and $R_{out}$ at each point $(x_{c}, y_{c})$ was defined as the $χ^{2}$ distance between the histograms $h^{in}$ and $h^{out}$ in the polygons:

Eq. (9)

χ^{2} (x_{c}, y_{c}) = \frac{1}{2} \sum_{i} \frac{{[h^{in} (i) - h^{out} (i)]}^{2}}{h^{in} (i) + h^{out} (i)} .

Aggregation of all these distances along the contour yields:

Eq. (10)

D = \sum_{c} χ^{2} (x_{c}, y_{c}) .

Fine registration was done by changing the four transformation parameters so as to minimize $D$ . The optimization was implemented with the optimization toolbox in MATLAB with the sequential quadratic programming algorithm searching for the local maximum of $D$ within a bounded region of four parameters.

3.4.

Asymmetric Analysis of Diabetic Foot Complications

The temperature difference of the corresponding points on the left and right feet is an indicator for diabetic foot complications.⁸^,⁹ However, the two feet in the thermal images were hardly at symmetric positions in the image, and they often showed asymmetric deformations due to DM or due to amputations. Therefore, the left and right feet needed to be registered with each other.

The goal of the left and right feet registrations was to find the optimal transformation $T : (x, y) \mapsto (x^{'}, y^{'})$ , which maps any point $(x, y)$ in the segmented left foot onto the corresponding point $(x^{'}, y^{'})$ in the segmented right foot. Due to local foot deformations (caused by DM), amputations, and projection distortions, a rigid left to right foot registration did not suffice and a nonrigid registration was needed. We employed a landmark-based deformation model with B-splines.¹⁸^,¹⁹ This model consists of an orthogonal $n_{x} \times n_{y}$ mesh that covers the image plane. The shifts of the mesh points form the parameters $Φ$ of the registration. The shifts of pixels between grid points are obtained by the B-spline interpolation. To find $Φ$ , a number of corresponding landmarks on the left and right feet are needed. Due to deformities and amputations, points at the lower half (near the heel) are more stable than the one near the toes. The procedure to obtain these corresponding points is defined in Algorithm 2.

Algorithm 2

Selection of corresponding points in the left and right feet.

• Inputs: Contours $(x_{n}^{l}, y_{n}^{l})$ and $(x_{n}^{r}, y_{n}^{r})$ with $n = 1, \dots$ . These are the sequences of 2-D coordinates in the thermal image denoting the boundaries of the left and right feet regions in the thermal image.

• Selection in the left foot and in the right foot (see Fig. 6):

- Determine the centroid

(x_{c}^{l}, y_{c}^{l})

of the contour points.

- Determine the midline of the lower half of the foot: (a) Determine segments for each horizontal line that connect the left boundary point to a right boundary point and that are below the centroid, the midpoint. (b) Determine the least squares linear fit of these midpoints.

- Rotate the contour around the centroid such that the midline stands vertical.

- Shift the contour vertical such that the lowest point (the heel) is at the

x

-axis.

- Define the horizontal lines on either foot with normalization with respect to the foot lengths. The vertical spacing of these lines below the centroid is smaller than above the centroid.

- The intersection points of these horizontal lines with the foot contours are the landmarks (together with the heel points). Detect whether a part is missing on either foot. If so, delete these points from the list.

Fig. 6

Selection of corresponding landmarks in the left and right feet contours for left/right foot registration: (a) and (c) The midlines of the lower halves of the foot regions were used to align the feet vertically; and (b) and (d) horizontal line scanning with two different step sizes.

To find the best parameters $Φ$ , the sum of the squared differences between the transformed left landmarks and the corresponding right landmarks was minimized with the steepest descent algorithm. The gradient vector, which is needed for this optimization, was obtained numerically.

After estimation of the parameters of registration, the associated geometric transformation was applied to the left foot to make the pixels correspond with the pixels on the right foot. The likelihood of a foot complication of a spot was calculated simply by subtraction of the temperatures of corresponding pixels. The threshold we used for complications detection is 2.2°C.⁸^,⁹ This criterion is the only one that has been clinically validated for determining diabetic foot complications with temperature measurements.

To evaluate the registration results, four corresponding regions on the left and the right feet of each patient were manually labeled. These regions were located near the border of the foot at top, bottom, medial, and lateral positions. If any region of either foot had been amputated, no region was selected. Examples of the annotation can be found in Fig. 9. Comparison between the labeled regions on the right foot and the regions on the registered left foot was done to obtain a rough measure of the registration quality. This evaluation is limited by the subjectiveness of the manual labeling.

4. Results and Discussion

4.1.

Feet Segmentation Results in Color Images

Initially, six color representations were considered: RGB, normalized RGB (rgb), RGB-ratio, HSV, CIE L*a*b*, and YCbCr. All have three color channels. For two of them, CIE L*a*b* and YCbCr, it might make sense to skip the luminance channel yielding the a*b* and CbCr representations. This left us with eight possible representations. A representation can be processed by the $K$ -means, the EM-LDC, or by EM-QDC. The considered numbers of classes were: $K = 1$ , 2, and 3. These numbers were determined based on the visual inspection of the number of objects (i.e., feet, ankle, black foot side supporting bars, black cloth, aluminium bars, and the white hospital cloth) in the acquired RGB images. Thus, in full, 72 combinations of representations, algorithms, and number of classes could possibly lead to a viable method.

A preselection of combinations/methods was made based on the following criterions:

• Visual inspection of the segmentation results. If a method clearly showed inferior results, then it was deleted from the list.
• Near resemblance of performance. All methods based on the RGB space and on the YCbCr space had nearly identical performances. This might be explained by the way that YCbCr encodes RGB. This resemblance motivated us to delete the YCbCr-based methods.
• Assessed performance measures. If the sensitivity, averaged over the population of patients, was less than 95%, then the method was deleted from the list. Likewise, if the specificity was less than 95%, or the mean distance error ${\bar{D}}_{EA}$ (averaged over the population) was larger than 10 mm, then the method was also excluded from consideration.
• Instability. If the standard deviation of the sensitivity or specificity, calculated over the population, was larger than 15%, then the method was crossed out for it was not capable of segmenting the feet with a stable performance.

Only nine methods out 72 survived these elimination rules. The quantitative assessment of the remaining methods can be found in Table 1. The difference between the performance of these nine methods is not very statistically significant except for a*b* with EM-LDC or EM-QDC and $K = 3$ . This method presented relatively lower mean sensitivities, just above 95%, but the mean specificities were the highest. This indicates an increased risk for assigning the foot area to the background. CbCr with all the methods and $K = 2$ obtained relatively higher ${\bar{D}}_{EA}$ and relatively lower sensitivity. Thus, the segmentations with CbCr for two clusters and those with a*b* for three clusters for all methods were excluded as well.

Table 1

Assessment of foot segmentation in color image over 76 patients.

		Four clusters	Three clusters		Two clusters
		CIE Lab*	CIE Lab*	aba	CbCrb	ab
Sensitivity (%)	$K$ -means				$96.8 \pm 1.2$	$97.3 \pm 1.0$
	EM-LDC	$96.7 \pm 1.8$	$97.9 \pm 0.7$	$95.1 \pm 2.6$	$96.5 \pm 1.3$	$97.2 \pm 1.2$
	EM-QDC	$97.1 \pm 2.6$		$95.8 \pm 1.4$
Specificity (%)	$K$ -means				$98.4 \pm 2.8$	$99.0 \pm 1.0$
	EM-LDC	$98.9 \pm 3.3$	$98.9 \pm 0.7$	$99.5 \pm 0.86$	$98.3 \pm 3.1$	$98.9 \pm 1.0$
	EM-QDC	$99.0 \pm 0.81$		$99.3 \pm 0.69$
${\bar{D}}_{EA}$ (mm)	$K$ -means				$2.86 \pm 5.32$	$1.51 \pm 2.22$
${\bar{D}}_{EA}$ (mm)	EM-LDC	$1.35 \pm 1.61$	$1.61 \pm 1.72$	$1.61 \pm 0.98$	$3.55 \pm 7.00$	$1.90 \pm 5.19$
	EM-QDC	$1.53 \pm 1.90$		$1.65 \pm 1.22$

^aThe two color components in CIE L*a*b* color representation.

^bThe two color components in YcbCr color representation.

To conclude, the foot segmentation from the color images can be achieved in CIE L*a*b*, either using all the three color channels with EM-QDC for four clusters or EM-LDC for three or four clusters or using only the two color components through $K$ -means, EM-LDC, and EM-QDC for two clusters. Two examples of the segmentation results are illustrated in Fig. 7. The sensitivities and specificities of these combinations were all around $98 % \pm 1 %$ and $99 % \pm 1 %$ , respectively, and ${\bar{D}}_{EA} = 2 \pm 2 mm$ . In the following sections, results are presented based on the foot segmentations obtained using all the three color channels in CIE L*a*b* with EM-LDC for three clusters.

Fig. 7

Examples of foot segmentation done with EM-QDC using three color channels in CIE L*a*b* (blue solid line) and the manual segmentation (red dashed line): (a) example 1; and (b) example 2.

4.2.

Registration Optimization in the Thermal Images

The parameters of the fine registrations are the shift $(x_{t}, y_{t})$ , the rotation angle $r$ , and the vertical scale $s$ . Since the pivot of the rotation was near the heel, the foot contour was not rotated around its centroid point, but around the heel point (the furthest point on the foot contour when the foot was rotated to stand vertically).

The length of the perpendicular profile line (shown in Fig. 5), which is equivalent to the window size, was set to 11 pixels with 5 pixels inside and 5 pixels outside the foot region. To bound the search area, the ranges of the parameters were limited; for shift: $x_{t}, y_{t} \in [- 20, 20]$ (pixel), for rotation: $r \in [- 3, 3]$ (deg), and for vertical scaling: $s \in [0.95, 1.05]$ . Examples of the optimization results can be found in Fig. 8.

Fig. 8

Examples of results from optimized registration. The green solid lines represent the foot contours obtained from manual segmentation done by experienced clinicians. The red dashed lines correspond to the foot contours obtained from projective transformation of the contours from the color images. The blue dashed lines come from the optimized registration: (a) example 1 and (b) example 2.

Compared with manual segmentation in the thermal images, the sensitivity and specificity of feet segmentation in the thermal images with transformation parameter optimization, before fine registration, are $96.0 % \pm 2.0 %$ and $98.1 % \pm 0.7 %$ with ${\bar{D}}_{EA} = 2.7 \pm 1.0 mm$ ; and after fine registration: $97.9 % \pm 1.1 %$ and $98.3 % \pm 0.5 %$ with ${\bar{D}}_{EA} = 1.9 \pm 0.6 mm$ .

4.3.

Left and Right Feet Registrations and Detection of Diabetic Foot Complications

For each patient, the right foot was chosen as the static object and the left foot was mapped on the right foot. Examples of the registration between left and right feet, and the regions with more than 2.2°C temperature difference, are given in Fig. 9.

Fig. 9

Examples of registration and risk identification with 2.2°C as criterion. Magenta regions on the right foot and the flipped left foot show the contralateral region selection. The blue lines represent the shifts due to the B-spline registration. The green regions represent the regions on the right foot after transformation while the white region underneath is the one selected manually on the right foot. The detected differences: in red/blue regions the temperature of the right/left foot exceeds the one of the left/right foot by more than 2.2°C: (a) results of a patient with an infected ulcer and an amputated hallux on the right foot; (b) Results of a patient [see Fig. 1(a)] who had callus and an ulcer on the hallux of the right foot (red area); (c) results of a patient [see Fig. 1(c)] who had callus and an ulcer on the arch of the left foot (blue area).

To get a quantitative assessment of the registration results, a comparison between manual association of some regions on the right and left foot and automated association (by our right/left registration algorithm) was performed: the sensitivity and specificity are $85 % \pm 1 %$ and $98.4 % \pm 0.4 %$ , respectively, and ${\bar{D}}_{EA} = 4 \pm 1 mm$ . The error may come partly from the subjectiveness of the manual association and partly from the asymmetric deformations of the feet (which can locally make the association difficult or even impossible). Taking this into account, the registration results are considered to be accurate.

Taking 2.2°C as the cut-off point for risk identification, 35 out of 37 diabetic foot ulcers were successfully detected (95%). The two missed ulcers were small ( $\sim 5 \times 5 {mm}^{2}$ ), one of which was only found after debridement of abundant callus. The database also contained three Charcot foots, which were all detected. This suggests that the clinical assessment of Charcot neuro-osteoarthropathy may benefit from the monitoring system, as it provides a full temperature map of the foot instead of only local temperature changes.

4.4.

Limitations of the Study

A limitation of the proposed asymmetric analysis was that it could only detect diabetic foot complications by comparing the two feet of the patient. This means that when one foot is amputated, any complication in the other foot cannot be detected. Additionally, in case that the both feet have similar complications in corresponding regions, the complications will also be missed. In addition, in finding the asymmetry between left and right feet, registration was always done from left and right feet. This registration itself may introduce artificial asymmetry, which has not been investigated in this article.

Another limitation of the study is that we did not have a “gold standard” to evaluate the segmentation and registration of the feet. We used manually segmented references that were labeled by one clinical investigator. This may be subjected to some bias since another observer may come to different observations. However, we expect that this interobserver variability will not significantly affect the found sensitivity and specificity.

5. Conclusion

In this contribution, an experimental setup with an IR camera system and an RGB camera was developed. With the data collected from diabetic patients with foot complications or patients who are at high risk of foot complications, a methodology was proposed for automatic detection of these foot complications with the acquired thermal images. The method is based on a simple asymmetric analysis combined with foot segmentation from color images and nonrigid landmark-based registration between left and right feet.

Foot segmentation with unsupervised machine learning in different color spaces was investigated. The foot segmentation in CIE L*a*b* color space achieved a sensitivity and specificity of $98 % \pm 1 %$ and $99 % \pm 1 %$ , respectively. With registration optimization to map the foot segments onto the thermal image, the final sensitivity and specificity of the foot segmentation in the thermal images were $97.9 % \pm 1.1 %$ and $98.3 % \pm 0.5 %$ , respectively. The registration between the left and right feet, based on foot contours, presented fairly good results regardless of the shapes, sizes, poses, or positions of the two feet. In this paper, we provide the proof-of-principle that the thermal images acquired through our experimental setup can detect diabetic foot complications.

These promising outcomes of thermal image analysis of the foot may prove to be promising in the early detection of foot complications in patients with DM and patients who are at high risk for these complications. Future research objectives are: (a) Both the left and right feet may be restored to a general and unbiased foot contour template to reduce the asymmetry in the foot registration. (b) Thermal image analysis of the ipsilateral foot to detect diabetic foot complications without the need to image the contralateral foot. (c) Combining the thermography technology with other modalities, such as photometric stereo imaging²⁴ and multispectral imaging,²⁵ in the experimental setup to predict the development or healing of diabetic foot complications. (d) Comparing the effectiveness and efficiency of predicting diabetic foot complications using different modalities. (e) Developing and investigating an intelligent telemedicine system with the most cost-effective modality or modalities to monitor diabetic foot status, thereby also adapting the usability and the operational conditions required in daily clinical practice.

Acknowledgments

This project is supported by public funding from ZonMw, The Netherlands Organisation for Health Research and Development (project ID: 40-00812-98-09031). The authors are grateful to the patients who participated in the data collection and the clinicians in the Diabetic Foot Unit, Department of Surgery, Hospital Group Twente, Almelo, The Netherlands. We also appreciate the contribution of Geert-Jan Laanstra of the Signals and Systems Group, Faculty of EEMCS, University of Twente, who gave us plenty of support in building the experimental setup. We also appreciate the efforts of Tim J.P.M Op’t Root and Marvin E. Klein from DEMCON Advanced Mechatronics BV, Enschede, The Netherlands, who developed the thermal reference elements and calibrated the camera for the purpose of these measurements.

References

1.

J. E. Shaw, R. A. Sicree and P. Z. Zimmet, “Global estimates of the prevalence of diabetes for 2010 and 2030,” Diabetes Res. Clin. Pract., 87 4 –14 (2010). http://dx.doi.org/10.1016/j.diabres.2009.10.007 DRCPE9 0168-8227 Google Scholar

2.

A. J. Boulton et al., “The global burden of diabetic foot disease,” Lancet, 366 12 –18 (2005). http://dx.doi.org/10.1016/S0140-6736(05)67698-2 LANCAO 0140-6736 Google Scholar

3.

J. Apelqvist et al., “Practical guidelines on the management and prevention of the diabetic foot,” Diabetes/Metab. Res. Rev., 24 S181 –S187 (2008). http://dx.doi.org/10.1002/(ISSN)1520-7560 DMRRFM 1520-7552 Google Scholar

4.

E. J. Boyko et al., “Prediction of diabetic foot ulcer occurrence using commonly available clinical information: the Seattle diabetic foot study,” Diabetes Care, 29 (6), 1202 –1207 (2006). http://dx.doi.org/10.2337/dc05-2031 DICAD2 0149-5992 Google Scholar

5.

C. E. Hazenberg et al., “Telemedical home-monitoring of diabetic foot disease using photographic foot imaging—a feasibility study,” J. Telemed. Telecare, 18 32 –36 (2012). http://dx.doi.org/10.1258/jtt.2011.110504 1357-633X Google Scholar

6.

S. Franc et al., “Telemedicine: what more is needed for its integration in everyday life?,” Diabetes Metab., 37 (4), S71 –S77 (2011). http://dx.doi.org/10.1016/S1262-3636(11)70969-7 DIMEFW 0338-1684 Google Scholar

7.

D. G. Armstrong et al., “Skin temperature monitoring reduces the risk for diabetic foot ulceration in high-risk patients,” Am. J. Med., 120 1042 –1046 (2007). http://dx.doi.org/10.1016/j.amjmed.2007.06.028 AJMEAZ 0002-9343 Google Scholar

8.

L. A. Lavery et al., “Home monitoring of foot skin temperatures to prevent ulceration,” Diabetes Care, 27 2642 –2647 (2004). http://dx.doi.org/10.2337/diacare.27.11.2642 DICAD2 0149-5992 Google Scholar

9.

L. A. Lavery et al., “Preventing diabetic foot ulcer recurrence in high-risk patients: use of temperature monitoring as a self-assessment tool,” Diabetes Care, 30 14 –20 (2007). http://dx.doi.org/10.2337/dc06-1600 DICAD2 0149-5992 Google Scholar

10.

P. Sun et al., “Relationship of skin temperature to sympathetic dysfunction in diabetic at-risk feet,” Diabetes Res. Clin. Pract., 73 41 –46 (2006). http://dx.doi.org/10.1016/j.diabres.2005.12.012 DRCPE9 0168-8227 Google Scholar

11.

N. Kaabouch et al., “Enhancement of the asymmetry-based overlapping analysis through features extraction,” J. Electron. Imaging, 20 (1), 013012 (2011). http://dx.doi.org/10.1117/1.3553240 JEIME5 1017-9909 Google Scholar

12.

N. Kaabouch, W.-C. Hu and Y. Chen, “Alternative technique to asymmetry analysis-based overlapping for foot ulceration examination: scalable scanning,” J. Diabetes Metab., S5 003 (2011). http://dx.doi.org/10.4172/2155-6156.S5-003 JDMOAK 2155-6156 Google Scholar

13.

K. Roback, M. Johansson and A. Starkhammar, “Feasibility of a thermographic method for early detection of foot disorders in diabetes,” Diabetes Technol. Ther., 11 663 –337 (2009). http://dx.doi.org/10.1089/dia.2009.0053 DTTHFH 1520-9156 Google Scholar

14.

R. G. Frykberg, A. Tallis and E. Tierney, “Diabetic foot self examination with the tempstat as an integral component of a comprehensive prevention program,” J. Diabetic Foot Complications, 1 (1), 13 –18 (2009). Google Scholar

15.

T. Nagase et al., “Variations of plantar thermographic patterns in normal controls and non-ulcer diabetic patients: novel classification using angiosome concept,” J. Plast. Reconstr. Aethetic Surg., 64 860 –866 (2011). http://dx.doi.org/10.1016/j.bjps.2010.12.003 1748-6815 Google Scholar

16.

T. Delchar, “From thermometry to thermography: physics and its applications,” Physics in Medical Diagnosis, Springer, London (1997). Google Scholar

17.

C. Liu et al., “Infrared dermal thermography on diabetic feet soles to predict ulcerations: a case study,” Proc. SPIE, 8572 85720N (2013). http://dx.doi.org/10.1117/12.2001807 PSISDG 0277-786X Google Scholar

18.

D. J. Kroon, “Segmentation of the mandibular canal in cone-beam CT data,” University of Twente, (2011). Google Scholar

19.

D. J. Kroon, “B-spline grid, image and point based registration,” (2008) http://www.mathworks.com/matlabcentral/fileexchange/20057-b-spline-grid--image-and-point-based-registration January 2015). Google Scholar

20.

J. J. van Netten et al., “Infrared thermal imaging for automated detection of diabetic foot complications,” J. Diabetes Sci. Technol., 7 (5), 1122 –1129 (2013). http://dx.doi.org/10.1177/193229681300700504 JDSTAV 1932-2968 Google Scholar

21.

N. Vandenbroucke, L. Macaire and J.-G. Postaire, “Color image segmentation by pixel classification in an adapted hybrid color space. application to soccer image analysis,” Comput. Vision Image Understanding, 90 (2), 190 –216 (2003). http://dx.doi.org/10.1016/S1077-3142(03)00025-0 CVIUF4 1077-3142 Google Scholar

22.

P. Kakumanu, S. Makrogiannis and N. Bourbakis, “A survey of skin-color modeling and detection methods,” Pattern Recognit., 40 1106 –1122 (2007). http://dx.doi.org/10.1016/j.patcog.2006.06.010 PTNRA8 0031-3203 Google Scholar

23.

F. van der Heijden et al., Classification, Parameter Estimation and State Estimation an Engineering Approach Using MATLAB, John Wiley & Sons, Chichester (2004). Google Scholar

24.

C. Liu, F. van der Heijden and J. J. van Netten, “Towards surface analysis on diabetic feet soles to predict ulcerations using photometric stereo,” Proc. SPIE, 8214 82141D (2012). http://dx.doi.org/10.1117/12.907245 PSISDG 0277-786X Google Scholar

25.

C. Liu et al., “Statistical analysis of spectral data: a methodology for designing an intelligent monitoring system for the diabetic foot,” J. Biomed. Opt., 18 (12), 126004 (2013). http://dx.doi.org/10.1117/1.JBO.18.12.126004 JBOPFO 1083-3668 Google Scholar

Biography

Chanjuan Liu received her BSc degree from the Beijing Jiaotong University in 2006. She received her MSc and PhD degrees from the Faculty EEMCS at the University of Twente in 2010 and 2014, respectively. Her PhD degree research focuses on techniques to build an intelligent telemedicine monitoring system that can be deployed at the patients’ home environment for frequent examination of patients’ feet, to detect precursors of ulceration.

Biographies of the other authors are not available.

Citation Download Citation

Chanjuan Liu, Jaap J. van Netten, Jeff G. van Baal, Sicco A. Bus, and Ferdi van der Heijden "Automatic detection of diabetic foot complications with infrared thermography by asymmetric analysis," Journal of Biomedical Optics 20(2), 026003 (11 February 2015). https://doi.org/10.1117/1.JBO.20.2.026003

Published: 11 February 2015

Access the abstract

JOURNAL ARTICLE
10 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 111 scholarly publications and 19 patents.

Explore citations on Lens.org

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Thermography

Image segmentation

Image registration

RGB color model

Cameras

Infrared cameras

Expectation maximization algorithms

1.

Introduction

1.1.

Related Work

1.1.1.

Telemedicine using digital photography

1.1.2.

Thermography

1.1.3.

Asymmetric analysis

Fig. 1

1.2.

Outline of the Article

2.

Materials and Measurements

Fig. 2

3.

Methodology for Automatic Detection of Diabetic Foot Complications

3.1.

Overview of the Proposed Methodology

Fig. 3

3.2.

Foot Segmentation in Color Images

3.2.1.

Machine learning for color image segmentation

Eq. (1)

Eq. (2)

Eq. (3)

Eq. (4)

Eq. (5)

3.2.2.

Segmentation evaluation

Sensitivity and specificity

Eq. (6)

Eq. (7)

Mean distance error of wrongly classified area D¯EA

Eq. (8)

Fig. 4

3.3.

Registration Optimization

Fig. 5

Eq. (9)

Eq. (10)

3.4.

Asymmetric Analysis of Diabetic Foot Complications

Algorithm 2

Fig. 6

4.

Results and Discussion

4.1.

Feet Segmentation Results in Color Images

Table 1

Fig. 7

4.2.

Registration Optimization in the Thermal Images

Fig. 8

4.3.

Left and Right Feet Registrations and Detection of Diabetic Foot Complications

Fig. 9

4.4.

Limitations of the Study

5.

Conclusion

Acknowledgments

References

Biography

Show All Keywords

Keywords/Phrases

Search In:

Publication Years

Mean distance error of wrongly classified area ${\bar{D}}_{EA}$