1.1.

Background

Over the previous decade, new studies have begun to discuss the convergence of two popular and well-established remote sensing techniques, namely spectral imaging and light detection and ranging (LiDAR). Both techniques are being used in a wide range of different remote sensing applications. However, they both have limitations that prevent their effective use in all measurement settings. Spectral imaging systems use passive sensors that depend on external lighting sources, thus making a measurement setting sensitive to shading and diffuse lighting effects. Moreover, spectral imaging systems cannot provide direct ranging information on the targets. Contrary to this, laser scanning instruments provide an accurate object structure in three dimensions, but are inherently limited to work on a single wavelength. Furthermore, some laser scanning instruments alter their transmission intensity so as to further improve their ranging accuracy, but this limits the usability of their intensity values.^1,2 Since the strengths of each of the two techniques complement the main weaknesses of the other, their combined usage offers clear synergies for obtaining the information, e.g., classification, required in an accurate analysis of complex environments.

First studies with the combined use of spectral imaging and laser scanning data have often been about improving forest and land class recognition from airborne data.^3–9 The results of these studies have shown a clear trend in improving overall classification accuracies over single-sensor results.^3,8,9 Additionally, forest studies have moved from overall forest type mapping to tree species level. Classification can be carried out even for individual tree crowns in the most optimal cases.⁴ Some data fusion studies have also been carried out at ground level.¹⁰ In these studies, the viewing geometry causes extra complications with passive imaging sensors: abrupt changes in lighting and shadows make accurate radiometric calibration of spectral data challenging, if not outright impossible. However, even with these difficulties, it has been possible to classify individual tree species in the test scene with relatively high accuracy while at the same time limiting the overall number of classification features.

The combined use of spectral and spatial data offers clear advantages over single-sensor results. However, this approach still has its limitations: first, the use of passive imaging would still make the measurement dependent on external lighting conditions. Second, multiple sensor datasets are typically collected with different measurement platforms and at different times, although some platforms are capable of simultaneous data collection.^11,12 Datasets collected from the same area but at different times are affected by environmental factors that alter their response and need to be calibrated. Such effects include, for example, changes in radiometric response due to air and ground moisture, spatial changes in the measurement due to wind (short time difference) or seasonal changes (long difference). Temporal effects affect both passive and active systems alike. Third, a combination of two or more separate datasets requires, in most cases, additional data processing, so that the measurement area and targets within it can be described in a unified manner. All additional processing steps during data fusion, such as interpolation, spatial scaling, and sensor cross-calibration, add to the uncertainty in data interpretation and extend the analysis time.

The best way to overcome the limitations related to combined data use is through measurement systems that integrate active spectral sensing with LiDAR ranging. The convergence toward this kind of a system has been approached from two main directions that we call “the LiDAR approach” and the “active imaging approach.”

In the LiDAR approach, data are collected with a single-laser scanning system that simultaneously transmits and records two or more discrete wavelengths, or else such a system is simulated with multiple scanners.^13–22 This type of system was modeled as early as 2009.¹³ Structural three-dimensional (3-D) information with additional spectral information yields more detailed information e.g., for studying forest canopy properties and biomass,^23–25 or for detecting and classifying different object types.^14,18 The setback of using multiple individual, monochromatic, laser sources is that it limits the available spectral range. Additionally, the most common available laser wavelengths may not be optimal for spectral analysis. In the active imaging approach, the target scene is illuminated with a hyperspectral light source and the backscattering light is recorded either on a pixel or image basis.^26–28 Active imaging applications have been widely studied over the previous decade, including detection through limited visibility (e.g., haze, dust, smoke), foliage- and camouflage penetrating vision, and target detection and classification (Ref. 29 and references therein). However, these systems directly cannot provide the 3-D structure of the scene, which may limit target detection when targets are spatially mixed with the background.

At present, new integrated systems with simultaneous ranging and flexible wavelength selection capabilities have been presented.^30,31 The Finnish Geodetic Institute’s (FGI) Hyperspectral lidar (HSL) system is one of these. Its development path has followed the active imaging approach. The earlier versions of the HSL did not provide ranging information, which in early studies was provided with a separate laser scanner.^32,33 The ranging capability was added later with time-gating, making the system an independent integrated hyperspectral laser scanning platform.³⁰ The ranging setup has been shown to work in tree species classification³⁴ and in monitoring the chlorophyll content change in Scots pine (Pinus sylvestris L.).³⁵

In this study, we aim to further examine the feasibility of the FGI HSL system in a potential new measurement concept, where temporal changes in the spectral and spatial responses of natural and man-made targets are detected over a diurnal cycle (24 h) in an outdoor setting. This study opens up whole new possibilities of mapping temporal changes. It will be possible not only to detect changes in the targets’ spectral responses, but also in their shape due to changing lighting conditions. Our hypothesis is that the temporal aspect in data can be exploited to help in the detection of man-made objects within the target scene, even if the spectral response of artificial targets resembles that of live vegetation. We also monitor the change in spectral response over the measurement cycle in order to ascertain whether temporal changes in spectral response cause target mixing.

The study is, to our knowledge, among the first of its kind. Laser scanning studies about change detection do exist, but they usually consider longer time scales of several days, weeks, or months, where individual measurements are repeated once per day or less, e.g., in landslide analysis,³⁶ fluvial studies,^37,38 or forestry.^39,40 There have also been sensor calibration studies, e.g., Ref. 41, with a few measurements both in the day and night time with the aim of validating the proper working of a sensor. Also, a new operational system, VegNET, has recently been introduced for long-term forest monitoring,⁴² but its design is not aimed at individual object detection as it has limited spatial resolution and scans are mainly performed during the night.

At present, hyperspectral laser ranging systems are mainly developed and utilized in the laboratory. Passive sensor systems cannot provide radiometrically stable results in lighting conditions that vary between no-light and full daylight. In this study, we investigate the capability of the HSL to detect spectral and spatial changes in an outdoor measurement setting. Moreover, as an early concept study, not all results have a direct comparison point. Thus, the analysis mainly focuses on spectral trend changes within the measurement area, and the spatial separation of the objects of interest is performed manually. Furthermore, the spectral trend analysis is extended to cover several common vegetation indices. The indices have been originally developed for multispectral and hyperspectral remote sensing platforms in order to detect particular biological activities or organic compound concentrations in biomass.

2.1.

FGI Hyperspectral Laser Scanning System

The FGI HSL is a laser scanning system that transmits hyperspectral (white) laser pulses with a continuous spectrum of 400 to 2500 nm to the target. It can measure up to eight separate wavelength bands from returning pulses. The number of bands is limited by the spectral sensitivity of the silicon detector, but the wavelengths are selectable within the transmittance range. In this study, seven wavelength bands ranging from 500 to 980 nm are used in target analysis. The full-width at half-maximum of the channels was about 20 nm. A 1278-nm short wave(length) infrared band was also measured using a separate detector, but its stability was so low that its data were not utilized in the analysis.

The main components of the HSL system are the SM5-he supercontinuum laser source (Leukos, Limoges, France), a two-dimensional scanning mechanism (Newport Corp., Irvine, California), the wavelength separating spectrograph (Specim, Oulu, Finland), 16-channel high speed detector element (First Sensor AG, Berlin, Germany), and the measurement computer with digitizer cards (National Instruments Corp., Austin, Texas). The HSL works by sending laser pulses to the target in a sweeping pattern and then recording returning waveforms for each wavelength band detected. The waveforms are digitized with a 1-GHz frequency, thus giving the system a nominal 15-cm range resolution. However, individual laser pulses can be localized to the waveforms with a practical resolution of a few centimeters. The HSL can measure up to 5000 waveforms per second and a maximum of three discrete returns are fitted in each waveform. Table 1 lists the properties of the HSL laser source and optics.

Table 1

Finnish Geodetic Institute Hyperspectral Lidar laser source and optics properties.

A more detailed description of the HSL system, its properties, and its measurement configuration are given in articles by Hakala et al.³⁰ and Nevalainen et al.⁴³

2.2.

Other Equipment

A digital remotely readable thermometer with a relative humidity sensor was used in monitoring the temperature and humidity changes over the experiment. The thermometer was a consumer grade device, so its results are used qualitatively so as to detect possible larger trends during the experiment, especially the relative humidity values that were not calibrated beforehand. The thermometer was placed within a few meters of the targets.

3.1.

Test Area

The test area of $7\times 20{\mathrm{m}}^2$ was located in Southern Finland (N. 60° 9.674′, W. 24° 32.807′), near the city of Helsinki. The area included a small Silver birch (Betula pendula), low understory, and a large Silver birch and goat willows (Salix caprea) at the back of the measurement range. The area was surrounded on its eastern side with a sparse, half-open canopy of full-grown birches that gave mixed shading on to the measurement area depending on the location of the sun. The nearby FGI building was located on the western side of the target area so it did not cast a shadow on the target area until close to sunset, at which time the shadow of the building shaded the target area for about half an hour before sunset.

The test area and measurement schematic are illustrated in Fig. 1.

Fig. 1

The measurement scene from the side (a) and above (b). Manually delineated target classes used in analysis are shown by different colors. Scanner was located in the origin.

3.2.

Experiment Setting and Measurement Preparations

The target setting consisted of three camouflage nets and both spectral and geometric reference targets. A small Silver birch (B. pendula), shoots of Scots pine (P. sylvestris L.), and stems and leaves of goat willow (S. caprea) that were also located within the scene acted as natural comparison targets. A plastic chair under one of the nets and a gray block made of Lightweight Expanded Clay Aggregate (LECA®), also called “LECA” in the text, were used as man-made comparison targets. Table 2 lists the different targets in the measurement scene and their relative sizes in terms of laser beam returns.

Table 2

General description of the manually delineated target point clouds in the measurement scene.

Spectral references included two Spectralon reference panels, one white with a 99% reflectance and one multilevel panel (12-25-50-99 reflectances). Their purpose was to monitor the stability of the laser intensity during the experiment.

The geometric reference targets were four styrofoam spheres (25-cm diameter) that were set around the birch. They were used to provide an internal georeference for additional point clouds collected with a separate TLS system (HDS6100, Leica Geosystems Inc., Heerbrugg, Switzerland). The separate datasets were collected once in daylight and once in the dark so as to provide a high resolution spatial cross-reference of the scene.

The camouflage nets were placed in the measurement scene after sunset and left there for the remainder of the measurement. The nets were measured over the night and in the sunlight over the next day. The nets were typical camouflage nets available in outdoor and army surplus stores. Their main use is in hunting and in wildlife photography to meld people and man-made objects into the background. Their general properties are listed in Table 3.

Table 3

Structural properties of the camouflage nets.

Target items Camo 1 and Camo 2 had similar coloring on both of the sides that resembled green foliage. Target Camo 3 had a double-sided print with one side resembling green vegetation and the other resembling dry vegetation or sand. Targets Camo 2 and Camo 3 had similar structural properties. Their surfaces were smooth with regularly scattered holes. The hole diameters were about 3 mm in size. Target Camo 1 differed from the other two structurally. It was made with patches of fabric attached loosely on an underlying net. The patches were partially interlaced when the target Camo 1 was spread out leaving holes up to few centimeters in diameter in a random pattern.

The HSL was located on the edge of the test area. A transportable garden tent was set up to cover the laser scanner and to protect it from condensing water, direct sunlight, and dust. A fan heater was also applied to prevent moisture from condensing on scanner surfaces.

Before scans were taken, a range-dependent intensity calibration was performed by taking the white reference panel to different distances from the scanner and by measuring its reflectance for a few seconds at a time. This guaranteed stable reflectance detection for the calibration. During scans, the intensity range-dependency for each point was corrected using interpolated reference values. Intensity range-calibration data were collected at up to a 12-m distance which covered all targets in the scene. At more than 12 m, only a point location was determined.

The HSL scanner distance to the birch stem was about 7.2 m and about 8.5 m to the camouflage nets. A fixed scanning window of 75-deg wide in a vertical and 21-deg wide in a horizontal direction was set to cover all targets and the references. The scanning distance was limited to 15 m from the scanner after which all returns were filtered during a scan.

Spectralon reference panels were set on the sides of the test area (white panel on the left, multilevel on the right) at roughly the same distance as the birch. During night hours, the white panel was taken into the green-house between scans to prevent surface condensation.

3.3.

Scanning Process

A total of 30 scans were performed over a 26-h time-frame. A single scan took about 17 min from start to finish, in which time the whole scanning scene was covered.

The scans followed a common pattern throughout the measurement. Scans were repeated once per hour except during sunset (about 19:55, GMT+3) and sunrise (about 6:40, GMT+3),⁴⁴ when the scan interval was reduced to 40 min so as to better resolve the possible effects of lighting change.

During each scan, the following attributes were documented to support data processing and interpretation: scan time frame, temperature and relative humidity in the test scene, windiness, sky cloudiness, general visibility (clear, mist, or fog during night time), and any possible issues affecting the scan or changes within or around the test scene (e.g., possible surface moisture, ambient night lighting from street or FGI building lights). All attributes were documented on a qualitative level, as their purpose was to explain possible data discontinuities or drastic changes in noise level (e.g., significant spatial noise due to gusts of wind during a scan).

The laser source was kept powered up over the whole experiment to minimize intensity fluctuations resulting from switching the laser power source on and off.

4.1.

Raw Data Conversion

The conversion of raw waveforms to usable spectral point cloud data included the following procedures. First, negative overshoot and ringing effects were removed from a signal. Then, the waveforms were normalized with the transmit pulse intensity. Laser pulse intensity and time of flight were determined using the widths and the positions of the trigger and return pulse echoes. Next, the intensities were then converted to backscattered reflectance by applying the distance and spectral calibration. After this, up to three Gaussian peaks were fitted to the waveforms to determine return echo positions. The last step in the spectral point cloud production was to combine the backscattered reflectance spectra with the corresponding times-of-flight and concurrent scanner orientation data. See Hakala et al.³⁰ for detailed preprocessing descriptions.

After the raw data were preprocessed, they were converted into a compressed binary format (.laz) for faster data access and smaller storage size.⁴⁵

4.2.

Target Delineation and Spectral Normalization

After the preprocessing steps, the targets listed in Table 1 were manually delineated from the point cloud. The manual delineation was carried out for each target with a few centimeters tolerance. Targets were not spatially mixed except in the case of the camouflage net Camo 1 and the plastic chair.

Before analysis, the point intensities were normalized using the white spectralon panel as a reference. A circular section of about 15 cm in diameter was selected from the center of the panel. The median value of all point intensities within this area was then used to normalize the intensities of all the other points in that scan. Omitting the edge points guaranteed better spectral stability for scene normalization.

Normalized backscattering intensities of all channels were saved for each measured point. Backscatter values smaller than zero were set to zero as their values were a result of instrumentation noise (and/or results from misfitted waveforms). Similarly, occasional high backscatter intensity values were limited to two as they were probably misinterpreted intensities. Each scan was normalized separately to limit the effects of inherent intensity fluctuation in the laser source.

4.3.

Vegetation Indices

After backscattering intensity normalization, a set of 29 vegetation indices were calculated for each laser return point based on a previous study by Nevalainen et al. (Ref. 35 and references therein). After a qualitative visual inspection of their target separation, eight indices were selected for further analysis. The indices were selected based on their relative rank as classification features. The index ranking was calculated with the methods included in the Arizona State University (ASU) Feature Selection Repository (FSR) (Ref. 46 and references therein). An ensemble of all ranking results was collected and the most highly and frequently ranked indices were selected. They are listed in Table 4. All the indices selected have been used to estimate vegetation chlorophyll content in earlier literature.⁴⁷ Normalized Difference Vegetation Index (NDVI) has also been used to estimate vegetation Leaf Area Index and green biomass.⁴⁸

Table 4

Vegetation indices selected for target differentiation. The index formulas have been adapted to correspond to the nearest available HSL wavelength channels. Relation column lists typical uses for each selected vegetation index. These include chlorophyll (Chl), Leaf Area Index (LAI), and biomass (bio) detection. The index REP Li is also included as it was used to differentiate targets Camo 1 and Chair from each other. The indices are ordered based on their relative significance as classification features.

Vegetation indices have been developed for specific remote sensing systems. Thus, wavelength channels and their widths measurable with the FGI HSL system do not match precisely with those of the indices. In such cases, we opted to use the nearest available channels to replace the missing ones. This selection meant that the results of indices with one or more wavelength channels replaced cannot be interpreted or compared directly with the original ones. However, this was not critical to the experiment, as its purpose was to test the potential of HSL data for general object classification.

Due to the wavelength channel compensation and notable noise in the data, each vegetation index value was filtered once again before analysis in order to remove overflow and NaN points. The new filtration step was individually carried out for each vegetation index as the discontinuities affected different points in the cloud.

4.4.

Spectro-Spatial Object Presentation and Its Interpretation

After spatial delineation and vegetation index calculation, each object in the measurement scene was represented with a spectro-spatial representation (SSR). Each SSR contained information about the point cloud structure, point-wise values of normalized backscattering intensities and vegetation indices, and echo and total number of point returns. The objects that were present over the whole experiment had 30 SSRs each, whereas the camouflage nets that were later placed in the scene had 22 SSRs each. The SSRs allowed the monitoring of possible changes in the shape and spectral responses of objects over the duration of the whole experiment.

5.1.

Variance in Target Spectral and Vegetation Index Responses

Figure 2 illustrates the median spectra of all targets in the measurement scene after an individual scan. The figure shows some general trends in the target spectra. The Birch stem and the chair spectra that were separated with a priori thresholds in NDVI and red edge position linear interpolation (REP Li) clearly have a different median spectrum compared to other targets. LECA block has a flat median spectrum over the measured spectral range making it separated from other classes for wavelengths around and over 750 nm. All other object median spectra have relatively similar shapes, but some variation is present at the longest measured wavelength, 978.0 nm.

Fig. 2

The median reflectance spectrum of all targets. The situation during scan 11 (at 22:00 h local time, dark hours). All channel values are spectrally separate and the dashed lines between the measured channels are not an approximation of the actual spectrum, but a visual cue to ease interpretation.

However, the classification task is more complicated than the median spectra imply. There exists significant spectral overlap between different target spectra due to the spectral variance within object point clouds. All target point clouds, excluding the Chair, consist of thousands of laser hits collected during a single scan. As target shapes and surfaces in the measurement scene were irregular, this means that the individual laser hits occurred at varying angles of incidence and thus there is a wide variation in the point reflectances. These variations are illustrated for each object point cloud in Fig. 3.

Fig. 3

Variance of spectral response over target point clouds in wavelength channels of (a) 641.2, (b) 675.0, (c) 711.0, and (d) 778.4 nm. The center line of the boxplots marks the 50th percentile (median) of the point cloud, and the upper and lower edges of the box mark the 25th and 75th return percentiles, respectively. About 99% of all points are within the black vertical lines. Outlying points are marked with red crosses. Target names are abbreviated as listed in Table 2. The situation during scan 11 (at 22:00 h local time, dark hours).

Figure 3 illustrates significant variance in the spectral returns for all target point clouds. The spectral returns overlap between the classes at several different wavelengths. Moreover, confidence intervals between the 25th and 75th percentiles are at the same level for most targets. In practice, this means that attempting to classify an individual point based on its spectral response is likely to be erroneous. A clear example of this is the high variance of the birch stem (Birch, S.) returns whose confidence interval covers roughly half of the reflectance range for the red and near-infrared (NIR) regions. The reason for the wide spread in the spectral response of the stem is a result of its round shape and black and white bark, which together emphasize the effect of incidence angle.

The wavelength has a clear effect on the spectral responses of the targets. All targets with foliage or material trying to simulate it have an increasing spectral response and interquartile variance at longer wavelengths. The remainder of the targets, namely the LECA block, the chair (Chair), and the birch stem (Birch, S.), show a much weaker increase. The increase in overall spectral response at longer wavelengths is expected for foliage due to the red edge close to 700 nm. However, the results also show that the variances within the camouflage net point clouds also increase.

Moreover, the following remarks were also made on the targets’ spectral responses. First, the LECA block had the most consistent returns and the smallest variance over the measured wavelength range. Second, the birch stem (Birch, S.) showed both the highest variance and the highest absolute returns over the measured wavelength range. Returns from the plastic chair (Chair) also presented clear a deviation from both the foliage and camouflage nets below and around 700 nm. After this, its lower boundary and median returns started to get closer to that of the other targets.

For the vegetation, the returns from the willow stems (Willow, S.) were clearly brighter than the foliage-only hits of the birch canopy (Birch, C.) and pine shoots (Pine). The probable reason is that the willow hits contained a mix of leaves, branches and stems that could not be differentiated with a single-feature threshold as in the case of the birch. This characteristic also separates the willow object from camouflage nets in the visible wavelengths, but the differences become less distinct with increasing wavelengths: first with Camo 2, then with Camo 3, and finally with Camo 1 at 978.0 nm.

Pure hits from foliage, i.e., Birch canopy (Birch, C.) and Pine shoots (Pine), are on the same level as the camouflage nets in the visible red area (641.2 nm). However, their spectral response becomes increasingly different as the wavelength is increased over 700 nm. Near a 1000-nm wavelength, all the nets have clearly higher median spectral returns than the natural targets (Fig. 2), but spectral overlap still exists (Fig. 3).

Overall, the spectral responses of the targets overlap each other on several wavelengths. Moreover, the target characteristics were calculated here using manually separated point clouds. In field conditions, it is likely that different target types are mixed together at least to some extent and thus render already small spectral differences undetectable.

To account for such cases, we tested the eight selected vegetation indices so as to determine whether some of them could give even higher separations between the targets than pure spectral differences. Our hypothesis was based on the assumption that since vegetation indices are derived to correspond with particular spectral responses in vegetation, then any deviation from these would imply that the target could be (in part) of inorganic or man-made origin.

The results of this test are shown in Figs. 4 and 5. In Fig. 4, the indices were NDVI, Optimized Soil-Adjusted Vegetation Index (OSAVI), Normalized Difference Red Edge (NDRE), and Renormalized Difference Vegetation Index (RDVI). In Fig. 5, the indices were Gitelson, Green Normalized Difference Vegetation Index (GNDVI), Triangulated Vegetation Index (TVI), and Double Difference (DD). The results showed both similarities and clear differences with the reflectance responses. The LECA block, birch stem (Birch, S.), and chair (Chair) showed similar responses in half of the indices. Their responses differentiated them clearly from rest of the targets that were of green vegetation or simulated such. The separation of the birch stem and the chair is probably related to their initial segmentation where a hard NDVI threshold of under 0.3 was used for the birch, and a hard REP Li threshold of over 705 and an NDVI threshold of under 0.3 were used for the chair.

Fig. 4

Division of the 25th, 50th, and the 75th percentiles of vegetation indices (a) NDVI, (b) OSAVI, (c) NDVI, and (d) RDVI for the target objects in the scanning scene during scan 11 (at 22:00 h local time, dark hours). Target name abbreviations are the same as in Table 2. Vegetation index listing is given in Table 4.

Fig. 5

Division of the 25th, 50th, and the 75th percentiles of vegetation indices (a) Gitelson, (b) GNDVI, (c) TVI, and (d) DD for the target objects in the scanning scene during scan 11 (at 22:00 h local time, dark hours). Target name abbreviations are the same as in Table 2. Vegetation index listing is given in Table 4.

Although the interquartile ranges of the camouflage net returns mainly overlap with those of the green vegetation, there are also clear differences between them. In Fig. 4, Camo 1 and Camo 3 to some extent stand out from all other targets when viewed with NDRE that is developed to correspond to the red edge in the target spectrum. Also, the interquartile range of Camo 2 is always on the lower end of the range of the vegetation targets.

In Fig. 5, all camouflage nets show a higher than average response in GNDVI compared to the other targets. Also, Camo 1 and Camo 3 have higher overall return values in DD. Camo 1 also clearly shows a higher response to the Gitelson index than any other target.

5.2.

Target Differentiation Based on Their Spectral and Vegetation Index Response

To see the effect of variance in automatic classification, we classified the spatially delineated object point clouds using a $k$-nearest neighbor algorithm ($k\mathrm{NN}$, $k=5$, Euclidean distance). We used an implementation available in MATLAB Statistics Toolbox, but the algorithm is also readily available on other scientific toolboxes and libraries.

The object point clouds were classified with ten-fold cross-validation. Since the birch canopy (Birch, C.) had significantly more points than the rest of the objects, a random sample of 15,000 points was taken from it before the training and the cross-validation. This was done to prevent a training bias. However, all remaining birch canopy points were classified during the object prediction.

After cross-validation, we also tested the effect of spatial aggregation on classification results. Since the hypothesis was that an individual point that may be misclassified due to the high variance in point responses in the spectral domain, the goal of the aggregation was to determine if the classification information in the local point neighborhood in the spatial domain would smooth out the variance and result in improved classification accuracy. The aggregation assumption was made because individual point return footprints were considerably smaller than the object dimensions. The point aggregation was performed by calculating a 3-dimensional binary tree (so called k-d tree) of the combined object point cloud and then taking the most frequent class of the points within a preset distance.

The overall classification results of the cross-validation and subsequent spatial aggregation with different point neighborhoods are shown in Table 5. Table 6 shows the class-wise user’s and producer’s accuracies for different classes. Figure 7 illustrates the classification results in the point cloud and the effect of spatial aggregation. The four most significant spectral channels were selected using an ensemble of all ranking results calculated with the ASU FSR⁴⁶ as in the case of vegetation indices. The four channels were located at 641.2, 675.0, 711.0, and 778.4 nm (channel numbers 2, 3, 4, and 6).

Table 5

The overall classification results and their kappa coefficients for all nine targets. Targets were classified with a kNN classifier (k=5, Euclidean distance). The most significant reflectance channels, vegetation indices, and their combinations were tested. Classifications were validated with 10-fold cross-validation. Spatial aggregation was performed on cross-validated results. The bolded cells mark the class-wise results shown in Table 6.

Table 6

The producer’s and user’s accuracies of all targets. Results are calculated using the cross-validation results and 5-cm spatial aggregation. Corresponding overall results have emphasized background in Table 5.

The results show the following: first, the best overall classification results are obtained by using all reflectance channels as classification features. However, the absolute difference in classification accuracy when compared to the results obtained with four most significant channels or with the four most significant vegetation indices is within 5% points. Second, the use of all possible vegetation indices or all reflectance values and indices does not improve the overall classification results. Third, application of spatial aggregation with a small preset value improves the overall classification accuracy with over 10% points due to little spatial overlap between the targets. This is clearly seen in Fig. 6, where individual misclassified points are relabeled after comparing them to their neighborhood. For systematically misclassified areas, as seen in Pine or in Camo 3, this approach does not improve the overall result.

Fig. 6

The classification results of the (a) manual reference classification, (b) $k\mathrm{NN}$ classification (k = 5$\text{k =}5$, Euclidean distance) with cross-validation and (c) the subsequent spatial aggregation with a 5-cm neighborhood. The four most significant reflectance channels were used in classification. The target scene is viewed from the direction of the scanner.

On a class-level, the Birch canopy, LECA block, and Camo 2 are classified with the best producer’s accuracy. Also Camo 1 is detected with a high accuracy. The remainder of the targets with green foliage, namely Pine and Willow shoots, show lower accuracies due to mixing with the Birch canopy. Camo 3 has a low producer’s accuracy due to parts of it mixing with Camo 1 and the Birch canopy. The Chair is not detected at all. The likely reasons for this are its low total point number, spatial mixing with Camo 1, and the use of spectral thresholding to separate it from Camo 1.

5.3.

Temporal Variance in Object Reflectance and Vegetation Indices

Different target classes are distinguishable with a few correctly selected spectral or spectrally derived features at a single instance. However, the effects of possible temporal changes in spectral response over a day should also be accounted for. Such changes can be induced by lighting changes, wind, or moisture condensation on surfaces. Thus, it is of interest to see how target responses develop and vary during the day. Figure 7 visualizes the effects of temporal changes in all target point clouds with pairs of three vegetation indices, namely NDVI, OSAVI, and NDRE. The indices were selected based on their rank in feature selection (Table 4).

Fig. 7

Target separation in feature space. Scatter plots show two vegetation indices at a time. Panel (a) illustrates the scatter plot of NDVI and OSAVI, panel (b) NDVI and NDRE, and panel (c) OSAVI and NDRE. Each point in the graph corresponds to a single-percentile value (10th to 90th) from a single scan. Camouflage nets and the chair are shown by 198 points each (9 percentiles from 22 scans) and all other targets are shown by 270 points each (9 percentiles from 30 scans).

The scatter plots in Fig. 7 show the following: (1) time-related effects are shown as a spread in each target point cluster. The spread is mainly linear meaning that the changes in vegetation index values are correlated over time. The only exclusion to this is the Willow stem response which shows leveling in the left pane. (2) The overlap between the individual classes is present at all times. However, the relative order between classes mainly stays the same and different target types (i.e., organic and inorganic) are still clearly differentiated. For example, all targets with foliage are in their own group as seen in the middle pane. Also, the camouflage net Camo 1 stands out clearly due to its high response in NDRE. These results imply that if a classifier is trained with one dataset, then its prediction power is likely to remain for scans taken at other times given that the surface characteristics do not change drastically (e.g.,. due to surface condensation). Nevertheless, verification of this is out of the scope of this article.

Figure 7 shows that the target class mixing remains over the whole experiment. To investigate whether the time development would provide any additional information on target types, we made another time series visualization of the spectral index changes in each target. Figure 8 shows the change in the NDVI interquartile range of target point clouds over time. The NDVI was selected as a vegetation index example as was it was found to be the most significant index in feature selection (Table 4).

Fig. 8

Change of the (a) 75th, (b)  the 50th, and (c) the 25th NDVI percentiles during the experiment for the target point clouds. Reported times start from the first scan. Sunset (at 19:55) and sunrise (at 6:40) are shown with vertical dashed lines, respectively, blue and red. Note: the birch stem point cloud was defined with a hard NDVI threshold of 0.2 and its results are only indicative.

The NDVI results are separately reported for the 25th, 50th, and the 75th percentiles to better describe possible trends in the responses. The results are the following: (1) inorganic material responses show relatively small trends during the experiment. Overall, there is a slight change in the response after sunset (dashed blue vertical line). Camo 1 and Camo 3 show a slight increase in their NDVI, whereas for the Camo 2 the response slightly decreases. For the LECA block, its variance increases (due to growing maxima and minima). After this, any changes and trends are weak until sunrise (red dashed vertical line) where they revert back to the daytime response. The sharp change in the response of the Camo 3 close to sunrise derives from turning it around to measure its other, differently colored, side. The response from the chair under Camo 1 is relatively flat for both the 50th and the 25th percentiles. The 75th percentile shows a slight downward turn over the experiment time frame. One contributing factor to the lack of notable changes in the NDVI response from the chair is probably the use of NDVI and REP Li filtration during preprocessing.

(2) The temporal variation in the NDVI was stronger for organic targets. For birch canopy returns, the 25th percentile does not show any trend and the median shows a slight increase until 2 h after sunrise. After that the median is reversed close to its starting value. However, the 75th percentile shows a steady increase after sunset which peaks around sunrise. Then, the change is reversed for around 45 minutes about 2 h after sunrise. On the other hand, the birch stem response was flat over the experiment. The main reason for this was that the stem was differentiated from the birch canopy point cloud with an NDVI threshold that capped its highest value to 0.2. However, neither the median nor the 25th percentile values, which were both uncapped, show any clear trends. Pine shoots present a different temporal pattern. The 25th and the 50th percentiles stay relatively constant until (solar) midnight after which the responses begin to drop until 1 h before sunrise. Then, the responses increase back to their daytime values about 2 h after the sunrise. The 75th percentile has a similar, but much weaker, pattern. Willow stems show yet another temporal variance pattern for the NDVI response. The response starts to decline toward its minimum about two and a half hours after sunset. The minimum is then reached about an hour before sunrise. Return to the daytime values happens about 2 h after sunrise. The changes in willows are most pronounced in the 25th percentile, but are also clear in the median response. In the 75th percentile, the changes are still visible, but not as obvious.

Overall, all the organic targets in the experiment show clear, albeit different, responses to diurnal variation. The species-wise sequences differ from each other, but also have common features. First, the return to their daytime values seems to happen about 2 h after sunrise. Also, the decrease in the response begins about the same time for both the pine shoots and the willows.

6.1.

Spectral Separability of the Targets

All targets, except perhaps excluding the LECA block, showed large variations in their spectral responses over a whole point cloud when tested with different wavelengths and vegetation indices. The variations were typically so wide that there was overlap between several targets at the given index values. One of the main reasons for the spectral overlap was the small laser footprint hitting complex target surfaces (about 20 mm in diameter at a 10-m distance). This is demonstrated by comparing the spectral variance of the LECA block and the birch canopy. The LECA block, which was the smoothest and had a continuous surface, showed the most constant variance over the measured wavelength range (Fig. 4). On the other hand, the birch canopy that consists of thousands of small branches, twigs, and leaves in different positions and angles shows limited variance at visible wavelengths that widen significantly when moving into the NIR region. However, the target with the widest variance was the birch stem. The reason is in its white bark combined with black spots and a curved surface. Therefore, returning signals may have hit either a bright or a dark spot on the bark. In addition, the angle of incidence with the laser signal and the stem may vary from parallel to perpendicular. Together, this means that a single return from a stem may have almost any reflectance value resulting in significant variances.

These examples show the complexity of determining, without additional information, where an individual point return is coming from. However, the classification results show that different classes could be still differentiated with accuracies of over 85%. The accuracies are even higher if similar classes are considered together. Such classes include green vegetation (Birch canopy, Willow stems), camouflage nets, nongreen vegetation (Birch stem), and other man made targets (LECA). This can be seen in Fig. 6 which illustrates how misclassifications mostly happen within similar class types.

In this study, the targets were separately located in the experiment setting and their point clouds could be manually delineated. This allowed the use of supervised classification. In natural conditions, the situation will definitely be more complex. There, it is probably futile to classify an individual point in an unsupervised manner without using any knowledge of its immediate surroundings. However, the results also show that with spatial aggregation it is possible to locally bring out clear spectral differences between most target types. This should improve their differentiation. A manual separation, as performed here, is a possible option for smaller datasets and spatially constrained cases. But, from a practical point of view, automatized segmentation routines and their combined use will be a close-to necessary requirement for larger data sets. The literature has a wide selection of different techniques available for different object types both in the built and natural environments.^57–60 In practice, an iterative approach that makes use of both spectral and spatial cues of the known and expected object types will probably be the most robust solution.

After the scene is segmented into suitable parts, a straight-forward application of descriptive statistics can be applied for target differentiation. Here, promising results were obtained by calculating percentiles of both spectral intensities and vegetation indices for objects in order to determine their spectral properties. In this study, the number of classifying features could be kept down to two or three to differentiate natural and man-made target types from each other. With a percentile approach, the point-wise feature variation within individual target classes are dampened and class characteristics become more evident. Here, percentile selection covered the 25th, 50th, and 75th percentiles of the analyzed features, thus avoiding extreme values. Moreover, just a few percentiles per each selected wavelength or vegetation index seemed to be sufficient for target separation if segmentation had been successful.

6.2.

Temporal Variation of Spectral Response in Target Point Clouds

In addition to spectral separation of targets, this study included a novel component of monitoring and detection of possible effects induced by changing illumination or nighttime conditions. To our knowledge, no similar tests have been reported in the literature so far. In this experiment, the measurement area was scanned with intervals of 1 h or less, regardless of external lighting status and the increased scan frequency revealed some interesting developments. In imaging, longer term time-series studies have recently been carried out both in the field⁶¹ and in the laboratory,⁶² but they are inherently limited to light hours or when using external light sources. Additionally, measurement radiometry and its calibration are complicated.

The results showed that the green vegetation presented clear variance over time, whereas the variance within man-made targets was much less pronounced. The temporal variance is probably a real effect because: (1) the temporal trends were detected both in the light and dark hours. This means that the external illumination, which was nonexistent during the night time, could not be a dominant factor driving the changes. Similarly, the air was calm during the night time, which rules out wind as a major contributing factor for the trends, (2) the man-made objects within the scene like the LECA block, camouflage nets, and reference targets did not demonstrate nearly as large temporal changes as vegetation, excluding a modest increase in variance at late night. Moreover, the changes differed between the targets, which should also rule out pure equipment-related contributions which would probably have shown as a similar drift in data.

Although none of the above factors were dominant, they all added to the changes. Other factors contributing to the detection are related to the targets themselves. When the delineated point clouds of the whole birch (Birch, C. and Birch, S.) and the willow stems (Willow, S.) were visualized in rapid consecutive order, a detectable fluctuation could be noticed in their canopy shapes (Fig. 9). When the point clouds were pseudocolored with a vegetation index, it could be seen that the spatial change in the canopy induced a change in the detected spectral response. A likely explanation is that when the canopy shape changes a little, mainly in the dark hours, then leaf inclinations change, and laser beam backscatter happens at a different angle leading to a change in the measured response. Although the absolute shape change is subtle, it is canopy-wide and thus large enough to cause a detectable effect in the point cloud statistics. On the other hand, the coniferous pine shoots (Pine) did not present a similar change in shape, while their spectral response also showed a temporal trend. Possible reasons for this may be that the footprint size should be even smaller to detect possible changes on the needle level or moisture condensation on the needles. However, determination of the spectral changes in the pine sample will require further studies in the future.

Fig. 9

Spatial change in the birch canopy point cloud over night hours. Panel (a) shows the point cloud shape variation for the whole birch. Panels (b) and (c) show the point cloud variation with more detail for branches marked with black boxes (upper and lower, respectively) in the panel (a). The largest changes were detected close to the sunrise. This has a clear visual correlation with the changes detected in the NDVI response (Fig. 8). The scanner was located to the left, in the origin.

6.3.

Summary

The results showed that it is possible to determine inorganic camouflage nets from vegetation with the FGI HSL system. The detection was based on differences in their spectral response compared to those of vegetation. The nets were best separated from other targets in the NIR region, located over 700 nm. The spectral differentiation was obtained after the point clouds were manually delineated from the measurement scene, and the individual point variance was reduced by taking percentiles of backscattering intensities or vegetation indices. The object detection was performed during the dark hours.

The temporal effects in all target spectra were also monitored. According to the results, overlapping of spectral responses between similar class types is likely when scans have been taken at different times during a day. This means that data acquisition times and conditions should be given extra consideration even when radiometrically calibrated data are available in order to avoid target mixing due to temporal variance. However, even with temporal variance included, the first qualitative results imply that it is possible to discern man-made objects from vegetation in a suitably selected feature space consisting of only a few different wavelength channels around the so-called “red edge” near 750 nm or vegetation indices derived near it.

In natural conditions, where different target classes are likely to have at least partial spatial mixing, the classification task will be significantly harder, especially for automatized processes. Such conditions will present a challenge to segmentation routines due to returns and small continuous surfaces. Nevertheless, the task should be feasible at least to a level, where man-made objects could be outlined into potentially interesting segments for further inspection. Here, spatial aggregation of points that were individually classified with supervised classification was shown to already increase overall classification accuracy using a close spatial range. However, this approach typically requires that a priori information about the target response is available and that scans can be taken with an active multispectral or hyperspectral laser scanning system. The plastic chair (Chair) that was covered with the camouflage net Camo 1 is a good example of this. Although the chair was not initially separable from the point cloud, it could be differentiated using thresholds for two vegetation indices that divided the mixed point cloud in two. However, due to low number of returns and the small continuous visible surface area, the chair could not be detected as a separate class in a supervised classification.

Although the temporal effects complicate the object classification in correct target classes due to increased variance in the spectral response and in shape for vegetation, they may also have a potential for exploitation. Since vegetation showed a larger temporal response this could be taken into account during analysis if data are available from several time points in different illumination conditions. However, more studies are required to test whether quantitative analysis is possible. Also, an open question remains as to whether there are detectable differences between healthy and recently cut vegetation over a short time difference. If the latter are reliably detectable, they could provide a useful proxy for locating man-made objects concealed within vegetation.