Hyperspectral imaging and neural networks to classify herbicide-resistant weeds

Bryan Scherrer; John Sheppard; Prashant Jha; Joseph A. Shaw

doi:10.1117/1.JRS.13.044516

14 November 2019 Hyperspectral imaging and neural networks to classify herbicide-resistant weeds

Bryan Scherrer, John Sheppard, Prashant Jha, Joseph A. Shaw

Author Affiliations +

Journal of Applied Remote Sensing, Vol. 13, Issue 4, 044516 (November 2019). https://doi.org/10.1117/1.JRS.13.044516

Abstract

A segment of the field of precision agriculture is being developed to accurately and quickly map the location of herbicide-resistant and herbicide-susceptible weeds using advanced optics and computer algorithms. In our previous paper, we classified herbicide-susceptible and herbicide-resistant kochia [Bassia scoparia (L.) Schrad.] using ground-based hyperspectral imaging and a support vector machine learning algorithm, achieving classification accuracies of up to 80%. In our current work, we imaged kochia along with marestail (also called horseweed) [Conyza canadensis (L.) Cronquist] and common lambsquarters (Chenopodium album L.) and the crops barley, corn, dry pea, garbanzo, lentils, pinto bean, safflower, and sugar beet, all of which were grown at the Southern Agricultural Research Center in Huntley, Montana. These plants were imaged using both ground-based and drone-based hyperspectral imagers and were classified using a neural network machine learning algorithm. Depending on what plants were imaged, the age of the plants, and lighting conditions, the classification accuracies ranged from 77% to 99% for spectra acquired on our ground-based imaging platform and from 25% to 79% on our drone-based platform. These accuracies were generally highest when imaging younger plants.

1. Introduction

Weed control is a major problem in worldwide agriculture, costing an estimated $25.2 billion a year to control.¹ The emergence of herbicide-resistant weeds has compounded this problem, threatening to reduce yields of certain crops by up to 93%.²^–⁷ In the United States and Canada alone, the combined potential losses from uncontrolled weeds in just corn and soybean would total $43 billion annually,⁸ and production of both crops would be cut in half. Current methods of identifying herbicide resistance require weeks or months,⁹ longer than a complete growing cycle in many cases. To achieve more real-time classification and detection capabilities, we have been developing remote sensing methods based on hyperspectral imaging and machine learning to classify different biotypes of herbicide-resistant weeds.

Our previous work¹⁰ classified between mature herbicide-susceptible and herbicide-resistant biotypes of the weed kochia, which has been demonstrated to be resistant to both glyphosate¹¹^,¹² (commonly marketed as Roundup^®) and dicamba (the active ingredient in the herbicides Banvel^® and Clarity^®). There we used ground-based hyperspectral imaging and a support vector machine (SVM) learning algorithm to correctly identify herbicide-susceptible kochia, glyphosate-resistant kochia, and dicamba-resistant kochia with 80%, 76%, and 67% classification accuracies, respectively.

The current paper builds on that work, which used our hyperspectral imager to collect ground-based imagery in a greenhouse and in the field, and used the same imager to collect field data while mounted on a drone. Furthermore, the newer data set included a larger set of crops and weed species and included multiple levels and types of resistance. This includes imaging herbicide-susceptible and -resistant kochia biotypes at different growth stages to characterize classification accuracies as plants grow. Most importantly, the results in this paper were obtained using neural networks instead of SVM algorithms, which improved the classification accuracies from on the order of 80% to up to 99%.

Hyperspectral imaging produces a two-dimensional spatial map with an additional dimension of spectral information, allowing researchers to collect reflectance profiles and predict herbicide resistance in a variety of weeds and situations. This is a site-specific weed control technique¹³ designed not only to locate weeds among crop fields but also to determine the appropiate herbicide(s) to use on each individual plant prior to any use of herbicides. The process involves a detailed spectral analysis of different weeds (e.g., kochia, marestail, and common lambsquarters) in different crops (e.g., wheat, barley, and sugar beet), aimed at differentiating between herbicide-resistant and -susceptible biotypes of the weed. Other work in this field includes distinguishing between susceptible- and glyphosate-resistant Palmer amaranth biotypes,¹⁴ differentiating between glyphosate-susceptible and glyphosate-resistant Italian ryegrass,¹⁵ and detecting the injury on crops as a result of dicamba and glyphosate.¹⁶

The paper follows this format: Sec. 2 presents the methodology used to grow the plants, the species and biotypes imaged, and the imaging and calibration process; Sec. 3 details the machine learning algorithm used to discriminate between plant species and biotypes; Sec. 4 discusses the classification accuracies on different combinations of crops and weeds; Sec. 5 offers concluding remarks and proposals for further work.

2. Physical Experiments and Data Collection

Data were collected at the Montana State University Southern Agricultural Research Center (SARC) in Huntley, Montana, from June 2017 to August 2017 and in August 2018. A total of nine different crops (barley, corn, dry pea, garbanzo, lentils, pinto, safflower, sugar beet, and wheat) and three different weeds (kochia, marestail, and common lambsquarters) were imaged, producing a total of 255 hyperspectral images. Vegetation spectra were extracted from these using a normalized vegetation index (NDVI) filter that identified plants and had the added benefit of excluding pixels from unhealthy parts of a given leaf as well as some pixels in shadow, as shown in Fig. 1. Further shadowed pixels were removed by keeping only those spectra with a summed reflectance equal to or higher than the sum of the median spectral reflectance in the image.

Fig. 1

An RGB image of herbicide-susceptible kochia leaves made from extracting the 460, 550, and 642 nm layers from (a) a hyperspectral image and (b) the pixels that only contain healthy vegetation found using our NDVI filter.

This paper draws from a subset of these images and spectra, focusing on the crop sugar beet and herbicide-susceptible and -resistant biotypes of kochia, marestail and common lambsquarters. In total, 19,188,642 spectra were extracted from 67 hyperspectral images and used as part of the process to generate and test the neural networks described in Sec. 3. The number and type of spectra for each weed species and biotype are detailed in Tables 1 and 2.

Table 1

Number of spectra collected and date they were collected from each weed species and biotype grown and imaged in greenhouse conditions, including susceptible (SUS), glyphosate-resistant (GR), and dicamba-resistant (DR) weed biotypes. Each image was taken under diffuse solar illumination using the Pika L imager in its tripod-mounted configuration.

Weed	Experiment date
Weed	June 23, 2017	June 28, 2017	June 30, 2017	July 28, 2017	August 3, 2017
Kochia (SUS)	2,840,882	182,578	n/a	277,856	43,269
Kochia (GR)	583,484	1,035,149	n/a	1,663,241	294,086
Kochia (DR)	1,583,024	555,565	228,602	1,371,050	332,874
Marestail (SUS)	185,226	n/a	n/a	n/a	n/a
Marestail (GR)	n/a	n/a	596,261	n/a	n/a
Common lambsquarters (SUS)	n/a	n/a	n/a	n/a	401,195
Common lambsquarters (GR)	n/a	n/a	n/a	3,398,223	2,013,450

Table 2

Number of spectra collected from kochia and sugar beet in outdoor conditions. The images taken in 2017 were taken under direct solar illumination using the Pika L imager in its tripod-mounted configuration. The images taken in 2018 were taken under direct solar illumination using the Pika L imager in its drone-mounted configuration.

Plant	Experiment date
Plant	August 4, 2017	August 9, 2018	August 10, 2018
Sugar beet	129,881	545,813	752,774
Kochia (SUS)	13,796	6952	31,684
Kochia (GR)	45,998	9368	11,996
Kochia (DR)	31,127	9039	13,835

All kochia, marestail, and common lambsquarters plants used in the experiments were grown using the method detailed in our earlier paper.¹⁰ During the 2017 and 2018 data collection in the field, we imaged potted kochia plants (at identical growth stages) among sugar beet at the full canopy growth stage, this time in direct sunlight. Hyperspectral images were collected individually for crops, and herbicide-resistant versus herbicide-susceptible kochia, as well as for a combination of various crops and different kochia biotypes. We also imaged greenhouse-grown barley, corn, dry pea, garbanzo, lentils, pinto bean, safflower, and wheat, but their spectra are not analyzed in this study.

For our experiments, we used a Pika L imaging system from Bozeman-based Resonon, Inc. with a Schneider Xenoplan $f / 1.4$ , 17-mm-focal-length lens. This hyperspectral imager had 300 spectral channels spread across the visible and near-infrared spectrum, from 387.12 to 1023.5 nm in increments averaging 2.13 nm. Each raster scan had a vertical height of 900 pixels and a horizontal width set by the scan parameters, with a 2500-pixel maximum under most conditions. The lens had an angular field of view of 0.88 mrad per pixel, giving us 0.79 rad (45.4 deg) by 2.20 rad (126 deg) angular field of view.

The imager provided us with a series of images with two spatial dimensions and one spectral dimension. Each spectrum was then extracted from its corresponding pixel in the spatial plane. For our outdoor experiments, the data collected were taken during moments when the wind was calm to minimize both spatial and spectral blurring (both occur as each frame has one spatial and one spectral dimension). It should be noted that while this study could be done with a single point spectrometer to obtain data, a hyperspectral imager allows us to collect what a spectrometer would see en masse, quickly and relatively easily.

For any ground-based image of an entire kochia plant, the camera was placed 2.25 m away from the base of the weeds (a pixel size of $1.98 \times 1.98 mm$ ), a distance similar to having the camera look at weeds if mounted on the boom of a tractor-mounted herbicide sprayer. The crops in those images were placed at most 0.25 m from the weeds so both the crops and weeds could be contained within the same image. This resulted in the camera being between 2.0 and 2.5 m from the imaged crops. For images of individual leaves taken in the greenhouse, we placed the imager 10 cm away (a pixel size of $0.088 \times 0.088 mm$ ) to capture more spatial features and more spectra of each type of plant. Examples of these setups are shown in Fig. 2.

Fig. 2

A Resonon Pika L hyperspectral imager mounted on a tripod imaging weeds at (a) the SARC greenhouse and (b) a similar setup in the field.

The drone-based images were all taken while the Pika L was mounted on a DJI Matrice 600 Pro drone, flying 9.7 m above the ground ( $\sim 9.0 m$ from the canopy of the crops and weeds) at a velocity of $1 m / s$ . The camera was pointed with a DJI Ronin-MX gimbal, and the imager was activated using a Resonon flight computer connected to a GPS receiver using a target map generated by Google Earth. The drone was flown along a path that included crops, weeds, and the spectralon panel, with the path designated by the DJI Ground Station Pro application. Each image generated using this method was used independently and not stitched together to make a single hyperspectral image. The setup in flight and a resultant image are shown in Fig. 3.

Fig. 3

A Resonon Pika L hyperspectral imager mounted on the DJI Matrice 600 Pro while the drone is (a) in flight taking images of (b) a sugar beet field. The red ellipse notes the position of herbicide-susceptible, glyphosate-resistant, and dicamba-resistant potted kochia next to a calibration panel.

3. Machine Learning Experiments

3.1.

Image Processing

A Spectralon panel was used to calibrate the digital numbers in every image to spectral reflectance (the percentage of light that is reflected from the object at a given wavelength). During each of the observations using the Pika L, we began the imaging sequence by recording dark images of a tightly sealed lens cap and placed a portion of the spectralon panel in the field of view when we recorded a vegetation image. This allowed us to subtract the electronic sensor noise (dark current) from the raw image and then divide out the spectral variation of the light source using an algorithm we developed. Further work is ongoing to make an algorithm independent of the calibration panel.

Figure 4 is a sampling of the calibrated data for every crop and weed species (susceptible biotypes only) imaged. Each spectrum is an algebraic mean and standard deviation derived from 4000 randomly selected individual spectra of the noted species. The plot shows the differences and similarities in the spectral reflectance for crop and weed species, which suggests that there is information that can be useful for classifying plants in images using their spectra. At the same time, the overlapping standard deviations suggested that machine learning algorithms and not simple spectral comparison are required to accurately discriminate them from one another. Unless otherwise stated, every spectrum in this and all subsequent plots will show the means derived from 4000 randomly chosen spectra of the noted plant biotypes imaged on the day specified. In addition, plots of spectra will no longer have standard deviations plotted, in order to more clearly show the mean spectra.

Fig. 4

Means and standard deviations of reflectance spectra for sugar beet, kochia, marestail, and common lambsquarters (herbicide-susceptible biotypes only). The faded vertical dashed line at 700 nm denotes the divide between visible (400 to 700 nm) and near-infrared (700 to 1000 nm) wavelengths.

3.2.

Neural Networks

As noted in Sec. 1, our previous work applied an SVM to this problem, and the results were promising.¹⁰ Even so, those results suggested that there may be room for improvement, thus motivating the current work applying a feedforward neural network. As will be demonstrated later, the results from the neural network yielded substantial improvement over the SVM. In this section, we provide a brief introduction to feedforward networks with a focus on our specific implementation.

The main component in a feedforward neural network is the neuron, also sometimes referred to as a node. A neuron can be described as a weighted combination of inputs passed through an “activation function,”

y = f (w_{0} + \sum_{i = 1}^{m} w_{i} x_{i}),

where

x_{i}

is the

i

’th input to the node,

w_{i}

is the weight on that input, and

f (\cdot)

is the activation function. Here,

w_{0}

corresponds to a special input known as a “bias” in which the input value is clamped to

+ 1

. The weight that is learned on this bias input can be interpreted as a “threshold” that drives the activation of the node. It can also be interpreted as an intercept term for the corresponding linear function.

A neural network connects these nodes together in the form of a directed acyclic graph. In the specific context of a feedforward neural network, the orientation of the directed edges in this graph proceeds from input forwards toward the output. No lateral connections within a layer are permitted. Furthermore, it is customary (though not required) that adjacent layers be fully connected, meaning every node in one layer connects to every node in the next layer. It is also customary that edges not skip layers, but layer skipping is permitted and is sometimes beneficial. For our purposes, we used fully connected layers with no layer skipping. An example of a feedforward network with two hidden layers is shown in Fig. 5.

Fig. 5

A two-hidden-layer neural network.

One of the primary advantages to using feedforward neural networks is that they are capable of learning arbitrary nonlinear functions. Indeed, it has been proven that a feedforward neural network with at least two hidden layers corresponds to what is called a “universal approximator,” meaning that the network can approximate any function (whether continuous or not) to an arbitrary level of accuracy.¹⁷ Key to enabling the network to model nonlinear functions is to apply nonlinear activation functions on the outputs of the nodes. When performing regression, however, it is customary for the output node to apply a linear (identity) activation function where $f (x) = x$ .

The most common activation functions in feedforward networks are the logistic function

f (x) = \frac{1}{1 + e^{x}},

the hyperbolic tangent

f (x) = \tanh x,

and the rectified linear unit (ReLU),

f (x) = \max {0, x} .

As discussed below, the network is trained using a gradient-based process, so it is necessary for the activation functions to be differentiable. The ReLU unit has a discontinuity at $x = 0$ , so it is customary to replace the ReLU activation with an approximation—the softplus function:

f (x) = \log (1 + e^{x}) .

The ReLU and softplus functions are widely used in so-called deep neural networks because they mitigate the problems of so-called vanishing and exploding gradients.¹⁸

As mentioned, the traditional way to train a feedforward neural network is to apply a gradient-based method for updating the weights in the network. The most commonly applied method is called backpropagation, where weights are updated from the output layer back toward the input layer by following the gradient of the error function of the network.¹⁹ More specifically, the mean squared error is generally used to evaluate the performance of the network:

MSE (D) = \frac{1}{| D |} \sum_{x \in D} {[y_{x} - f (x)]}^{2},

where

D

corresponds to the data set used for training,

y_{x}

is the true value expected when passing example

x

through the network, and

f (x)

is the output of the network. At the output layer, weights are updated as

Δ w_{j i} = - η \frac{\partial MSE (x)}{\partial w_{j i}} .

For the hidden nodes, the updates are propagated backward in proportion to the current weights. For example, given a logistic function, the update rule for the hidden nodes corresponds to

Δ w_{j i} = - η δ_{j} z_{j i},

where

δ_{j} = o_{j} (1 - o_{j}) \sum_{k \in downstream (j)} δ_{k} w_{k j},

and

o_{j}

is the output of node

j

after passing

z_{j i}

through the activation function. The training process can apply the update rules after each example is presented (called incremental updating), by averaging all of the updates after the entire data set is processed (called batch updating), or by averaging over a random subset of the training examples (called stochastic gradient descent).

The neural network architecture and parameters were selected by performing a hyperparameter search on the activation function (rectified linear and hyperbolic tangent) number of hidden layers (from 1 to 3), the number of nodes in each layer (125, 250, 500, and 1000), the learning rate (0.05, 0.01, 0.005, 0.001, 0.0005, and 0.0001), the momentum (0.5 and 0.9), and the regularizer coefficient (0.01 and 0.005).

The algorithm was trained on the full spectrum observed by the hyperspectral imager, providing a 300-dimensional vector for the neural network to train and classify. The data used to test the neural network are shown in Fig. 6. Each spectrum is the average of 4000 randomly selected spectra from susceptible, glyphosate-resistant, and dicamba-resistant kochia biotypes that were imaged on June 23, 2017. These 4000 spectra for each class of plant were randomly and evenly split into training and validation sets. The training sets were combined, and validation sets were combined, forming a training set of 6000 spectra and a valiation set of 6000 spectra, 2000 spectra from each kochia biotype. The neural network was trained on the training set and used to predict the class of the spectra in the validation set. The roles of the training and validation sets were then reversed: the validation set was used to train the neural network, and the trained neural network predicted the class of the spectra in the training set.

Fig. 6

Reflectance spectra for susceptible (SUS), glyphosate-resistant (GR), and dicamba-resistant (DR) biotypes of kochia and the confusion matrix showing the classification accuracies of the neural network.

This process was repeated five times in an experimental design method known as $K \times 2$ stratified cross validation,²⁰ which is used to ensure that a particular instance of a machine learning algorithm is not randomly showing accurate classifications. The average and standard deviation of the classification accuracies are presented in the multiclass confusion matrix also shown in Fig. 6. Any values of 1.0 or 0.00 in this and subsequent confusion matrices are rounding errors, indicating an accuracy equal to or greater than 0.995 or a standard deviation less than 0.005, respectively.

A confusion matrix is a table that visualizes the classification accuracy of a machine learning algorithm, indicating the fraction of measurements that the algorithm predicted correctly or incorrectly. In this case, the neural network correctly identified dicamba-resistant, glyphosate-resistant, and susceptible (susceptible to both glyphosate and dicamba) kochia with 99%, 99%, and 98% accuracy, respectively. More generally, the correct prediction rates occur on the diagonal and various incorrect prediction options exist in the off-diagonal positions. For example, the algorithm incorrectly predicted susceptible kochia as glyphosate-resistant 1% of the time and as dicamba-resistant less than 0.005% of the time.

In total, it took an average 2.5 h to extract weed spectra from a hyperspectral image, train the neural network, and produce classification accuracies. While this varies widely depending on a given computer system and any other processes the machine is running simultaneously, it shows that this methodology has the potential to decrease the time needed to classify herbicide-resistant weeds by at least an order of magnitude.

4. Results and Analysis

This section details the classification of sugar beet and weed biotypes using the neural network algorithm described in Sec. 3 applied to spectra obtained from hyperspectral images recorded at the SARC in Huntley, Montana. We assumed the grower would know which crop is in which part of each field and which weeds are dominant, so we trained the neural network to discriminate between a single crop (sugar beet) and three different weed species (kochia, marestail, and common lambsquarters). The plants were then classified assuming different levels of knowledge about an imaginary field, starting with a discrimination between sugar beet and all potential weeds. This process was then iterated, assuming more knowledge, either known previously, or gained through the previous classification.

4.1.

Ground-Based Imaging Results

We did one-, two-, and three-step processing of the spectra. In the one-step processing, we fed the reflectance spectra into a neural network to classify crops, herbicide-susceptible and -resistant kochia, marestail, and common lambsquarters. In the two-step process, we first classified crops and weed type (kochia, marestail, and common lambsquarters), then we further classified each weed type as herbicide-susceptible or herbicide-resistant. In the three-step process, we first classified between crops and all weeds, then classified among the different weed species, finally classifying between herbicide-resistant and -susceptible biotypes. The multiple-step classifications were explored to see if small increases in processing complexity could yield significant improvements in the classification accuracies. We found that generally the multistep processing increases classification accuracies by up to 11% over single-step processing. Not all details of these processing variants are discussed here and can be found in related work.²¹

Figure 7 is the first step in our crop/weed discrimination process, accurately classifying 99% of the sugar beet spectra and greater than or equal to 99.5% of the weed spectra. We next discriminated among the weeds themselves and used the neural network to classify just those spectra classified as “weeds.”

Fig. 7

Confusion matrices for sugar beet and the average spectrum of (a) all weed species and biotypes used in this study and (b) kochia, marestail and common lambsquarters weed species. The three weed species were classified against each other, assuming that the crop has already been isolated.

Once the weeds have been discriminated against each other, the algorithm was used to classify herbicide-susceptible and -resistant biotypes. Figure 8 shows that the algorithm classified glyphosate-resistant and -susceptible biotypes of marestail at least 99% of the time. Similarly, Fig. 8 shows that the algorithm accurately discriminates between glyphosate-resistant and -susceptible biotypes of marestail at least 99% of the time.

Fig. 8

Confusion matrices for (a) glyphosate-susceptible (MS) and -resistant (MR) marestail biotypes, (b) glyphosate-susceptible (LS) and -resistant (LR) common lambsquarters biotypes, and (c) glyphosate-susceptible (KS) and -resistant (KR) kochia biotypes.

The final classification accuracies and propagated errors for sugar beet and the susceptible and resistant biotypes were collected in Table 3. We also note that while we did attempt to classify the plants in a single step, that method produced lower accuracies: sugar beet spectra, for example, were only accurately classified 88% of the time.

Table 3

Final classification types and propagated errors of sugar beet, herbicide-susceptible (SUS), and herbicide-resistant (RES) kochia, marestail, and common lambsquarters biotypes.

	Classification accuracies (%)
Sugar beet	$0.99 \pm 0.01$
Kochia (SUS)	$0.94 \pm 0.03$
Kochia (RES)	$0.83 \pm 0.04$
Marestail (SUS)	$0.98 \pm 0.01$
Marestail (RES)	$0.97 \pm 0.01$
Common lambsquarters (SUS)	$0.96 \pm 0.01$
Common lambsquarters (RES)	$0.94 \pm 0.01$

Discriminating between kochia biotypes proved more difficult. Figure 8 shows the classification accuracies for susceptible kochia and all resistant biotypes, which are 96% and 85%, respectively, as well as the misclassification accuracies of 15% and 4%, respectively. Given 2000 spectra (1000 spectra of each of two biotypes), 1110 out of 1000 plants would be identified as susceptible (true susceptible and false suceptible), and 890 out of 1000 plants would be identified as resistant (true resistant and false resistant).

This does not, however, immediately throw the results of a map over an entire crop field in doubt. A single spectrum might be classified incorrectly, but the probability of misclassifying an entire plant is lowered if multiple pixels on a single plant are used to classify the plant.

If, for example, an herbicide-resistant kochia plant observed in a hyperspectral image consists of $p$ pixels (and thus has $p$ spectra that have been measured), it will have on average of $0.85 p$ spectra that are classified as resistant and $0.15 p$ as susceptible. To falsely classify the plant as a whole as susceptible, $0.96 p$ of the resistant spectra would have to be misclassified as susceptible, a probability of ${0.15}^{0.96 p}$ . If $p = 10$ , i.e., the imager is relatively far from the plants it is imaging, the probability of a resistant kochia plant being misclassified as susceptible is $1.23 \times 10^{- 8}$ . (The numbers used in this example are drawn from Fig. 8.)

In other words, if we assume that every genuine plant has the average number of correctly and incorrectly classified spectra (an assumption that needs to be tested), this algorithm wrongly classifies an herbicide-resistant kochia plant with 10 spectra as herbicide-susceptible approximately once every 10 million classification attempts. If the kochia plant was herbicide-susceptible, the algorithm would wrongly classify it as herbicide-resistant once every trillion classification attempts. Thus, the more pixels a plant in a given hyperspectral image consists of, the less likely an entire plant will be wrongly classified. Using this methodology, even classification accuracies that might seem low can be used to correctly identify plants as a whole.

A classified scene of sugar beet and three biotypes of kochia can be seen in Fig. 9. It is immediately apparent that while the neural network classifies between kochia and sugar beet well, it does not do as well as suggested by the confusion matrices when attempting to discriminate between herbicide-resistant biotypes. This is possibly due to mixed pixels on kochia leaves in front of sugar beet leaves, especially when one notes that the bases of the five kochia plants are correctly identified when compared to their tops.

Fig. 9

(a) An RGB image of sugar beet and potted kochia plants, (b) a classification map of sugarbeet and kochia, and (c) a classification map of sugar beet, susceptible kochia, glyphosate-resistant kochia, and dicamba-resistant-kochia. Sugar beet is green, kochia in (b) and susceptible kochia in (c) are brown, glyphosate-resistant kochia is blue, and dicamba-resistant kochia is purple.

There is one further variable that has yet to be considered in the classification algorithms: time. As part of the 2017 imaging campaign, we imaged susceptible, glyphosate-resistant, and dicamba-resistant kochia biotypes as they aged. Figure 10 plots the reflectance curves of susceptible, glyphosate-resistant, and dicamba-resistant kochia at 2, 3, 7, and 8 weeks old, corresponding to heights of 7.6 to 10 cm tall, 13 to 15 cm tall, 25 cm tall, 30 to 38 cm tall, respectively. While we can use absolute ages in this case because the kochia were grown in a greenhouse, in a field its age must be estimated based on its height.

Fig. 10

Spectra of susceptible (SUS), glyphosate-resistant (GR), and dicamba-resistant (DR) biotypes of kochia over time. Note that none of the spectra evolved linearly.

An analysis of the kochia at each stage of its development revealed that the classification accuracies of at least kochia heavily depend on age. We found that kochia biotypes imaged when they were about 2 weeks old were the easiest for our algorithm to discriminate, correctly discriminating between glyphosate- and dicamba-resistant kochia with 99% accuracy.

This changes as the plant grows older, though not strictly decreasing. Week 8, for example, has a higher classification accuracy than week 7. This strongly suggests that attempts to discriminate between herbicide-susceptible and -resistant kochia should be done in the plant’s earlier growth stages. These spectra and classification accuracies are shown in Fig. 11.

Fig. 11

Confusion matrices for susceptible (SUS), glyphosate-resistant (GR), and dicamba-resistant (DR) biotypes of kochia at (a) 2 weeks, (b) 3 weeks, (c) 7 weeks, and (d) 8 weeks old.

4.2.

Drone-Based Imaging Results

Figure 12 shows the classification accuracies and confusion matrices for kochia imaged in a sugar beet field by a drone flying at $1 m / s$ at a height 9 m above the canopy of sugar beet and kochia biotypes and flying level with the ground. The accuracies were considerably worse than the ground-based images due to factors including mixed pixels, added background light in the uncontrolled field, and possible drone motion-induced smearing, though the trend of younger kochia being easier to classify continues. These difficulties are expressed in Fig. 13, which shows that the standard deviation of data taken using the drone-based imager is twice that of data taken with the ground-based imager.

Fig. 12

Confusion matrices for susceptible (SUS), glyphosate-resistant (GR), and dicamba-resistant (DR) biotypes of kochia at (a) 10 to 15 cm tall and (b) 20 to 30 cm tall.

Fig. 13

Mean spectra and standard deviations for young, herbicide-susceptible kochia imaged using the drone- and ground-based hyperspectral imager.

Similar to the discrimination of herbicide-resistant kochia biotypes presented earlier, the accurate classification of whole plants will take an additional layer of statistics. The lower accuracies produced by data gathered from the drone-based platform also suggested that the parameters involving the drone itself should be further explored. For example, the drone could have been flown at a lower altitude to increase the resolution of the plants. The drone could also in theory have flown at a slower velocity to minimize any noise in the image caused by traveling too fast. This will, however, require different software to fly the drone as the DJI Ground System Pro application enforces a minimum flight speed of $1 m / s$ .

5. Conclusion

Herbicide-resistant weed biotypes are an emerging and costly problem for crop production internationally. While the underlying biochemistry of herbicide resistance is well understood, the time it takes for traditional methods to detect these biotypes makes them unwieldy to apply on a large scale. In addition, these methods generally require one to spray whole fields of crops, a costly endeavor both economically and environmentally. We developed a neural network machine learning algorithm that uses spectral reflectance observed by a hyperspectral imager that can discriminate for herbicide-resistant weeds with up to 99% accuracy. Spectra can either be classified against each other all at once or in stages, depending on the accuracy desired, and are classified on time scales an order of magnitude smaller than traditional methods. This process is currently implemented for an individual plant and has not yet been scaled up to entire crop fields.

The classification accuracies are shown to have a strong dependence on the growth stage of an individual plant and are higher when a given plant is younger. Further work on this needs to be done investigating the age dependence of the classification algorithm on glyphosate-resistant marestail and common lambsquarters. In addition, given the scope of the problem, more weeds and crops should be imaged and classified against one another and over time.

The neural network also has areas where it can be refined, particularly when discriminating between herbicide-resistant kochia biotypes. This includes investigating parameters such as dropout, bias nodes, reducing the dimensionality of the data, and changing the learning rate and momentum (annealing) while the network is training.

Concurrently, the imaging parameters of future drone-based campaigns should be changed in order to provide spectra closer to those observed in greenhouse conditions to help gain more reliable classification accuracies.

Lastly, the trained neural network should be applied to more hyperspectral maps of herbicide-susceptible and -resistant weeds among crops to test the model under multiple and varying field conditions. These maps should consist of data from the field as well as those modeled using the data collected in order to vary the crops, weeds, lighting, mixed pixels, distance of imager from plants, environmental conditions, and other factors to explore the efficacy of the methods developed in this study.

Acknowledgments

This research began with support from the Montana Research and Economic Development Initiative (MREDI) through Contract Nos. 51040-MUSRI2015-01 and 51040-MUSRI2015-02. Additional opportunities to develop machine learning capabilities were provided by Intel Corporation through IL/ER/Montana-Intel SRA 10-18-17. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the state of Montana or the Intel Corporation.

References

1.

D. Atwood and C. Paisley-Jones, “Pesticides industry sales and usage 2008-2012 market estimates,” 1 –24 (2017). Google Scholar

2.

E. E. Schweizer, “Predicting sugarbeet root losses based on kochia densities,” Weed Sci., 21 565 –567 (1973). https://doi.org/10.1017/S0043174500032483 WEESA6 0043-1745 Google Scholar

3.

D. M. Weatherspoon and E. E. Schweizer, “Competition between kochia and sugar beets,” Weed Sci., 17 464 –467 (1969). https://doi.org/10.1017/S0043174500054564 WEESA6 0043-1745 Google Scholar

4.

G. K. Dahl, A. G. Dexter and J. D. Nalewaja, “Kochia competition and control in wheat,” in North Central Weed Control Conf., (1982). Google Scholar

5.

G. A. Wicks, A. R. Martin and G. W. Mahnken, “Control of triazine-resistant kochia (Kochia scoparia) in conservation tillage corn (Zea mays),” Weed Sci., 41 225 –231 (1993). https://doi.org/10.1017/S0043174500076104 WEESA6 0043-1745 Google Scholar

6.

G. A. Wicks et al., “Control of triazine-resistant kochia (Kochia soparia) in sorghum (Sorghum bicolor),” Weed Technol., 8 748 –753 (1994). https://doi.org/10.1017/S0890037X00028633 WETEE9 Google Scholar

7.

V. Kumar, P. Jha and A. J. Jhala, “Confirmation of glyphosate-resistant horseweed (Conyza canadensis) in Montana cereal production and response to post herbicides,” Weed Technol., 31 799 –810 (2017). https://doi.org/10.1017/wet.2017.49 WETEE9 Google Scholar

8.

L. V. Wychen, “WSSA calculates billions in potential economic losses from uncontrolled weeds,” (2016). Google Scholar

9.

H. J. Beckie et al., “Screening for herbicide resistance in weeds,” Weed Technol., 14 428 –445 (2000). https://doi.org/10.1614/0890-037X(2000)014[0428:SFHRIW]2.0.CO;2 WETEE9 Google Scholar

10.

P. Nugent et al., “Discrimination of herbicide-resistant kochia with hyperspectral imaging,” J. Appl. Remote Sens., 12 016037 (2018). https://doi.org/10.1117/1.JRS.12.016037 Google Scholar

11.

L. F. Friesen et al., “The biology of Canadian weeds. Kochia scoparia (L.) Schrad,” Can. J. Plant. Sci., 89 141 –167 (2009). https://doi.org/10.4141/CJPS08057 CPLSAY 0008-4220 Google Scholar

12.

I. Heap, “The international survey of herbicide resistant weeds,” (2009) http://www.weedscience.org Google Scholar

13.

M. Weis et al., “Precision farming for weed management: techniques,” Gesunde Pflanz., 60 171 –181 (2008). https://doi.org/10.1007/s10343-008-0195-1 GEPFAG 0367-4223 Google Scholar

14.

M. Matzrafi et al., “Hyperspectral technologies for assessing seed germination and trifloxysulfuron-methyl response in Amaranthus palmeri (palmer Amaranth),” Front. Plant Sci., 8 474 (2017). https://doi.org/10.3389/fpls.2017.00474 FOPSAC 0016-2167 Google Scholar

15.

M. A. Lee et al., “Differentiating glyphosate-resistant and glyphosate-sensitive Italian ryegrass using hyperspectral imagery,” Proc. SPIE, 9108 91080B (2014). https://doi.org/10.1117/12.2053072 PSISDG 0277-786X Google Scholar

16.

Y. Huang et al., “Ground-based hyperspectral remote sensing for weed management in crop production,” Int. J. Agric. Biol. Eng., 9 98 –109 (2016). https://doi.org/10.3965/j.ijabe.20160902.2137 Google Scholar

17.

K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural Networks, 4 (2), 251 –257 (1991). https://doi.org/10.1016/0893-6080(91)90009-T NNETEB 0893-6080 Google Scholar

18.

Y. Bengio, P. Simard and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. Neural Networks, 5 (2), 157 –166 (1994). https://doi.org/10.1109/TNN.72 ITNNEP 1045-9227 Google Scholar

19.

D. E. Rumelhart, G. E. Hinton and R. J. Williams, “Learning representations by back-propagating errors,” Nature, 323 533 –536 (1986). https://doi.org/10.1038/323533a0 Google Scholar

20.

R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” in Int. Joint Conf. Artif. Intell., (1995). Google Scholar

21.

B. Scherrer, “Weed and crop discrimination with hyperspectral imaging and machine learning,” (2019). https://scholarworks.montana.edu/xmlui/handle/1/15148 Google Scholar

Biography

Bryan Scherrer is a research engineer at the Optical Remote Sensor Laboratory at Montana State University in Bozeman, Montana. His current work focuses on the development of hyperspectral imaging and lidar systems for use in precision agriculture, ecology, and food processing. He received his MS degrees in physics and electrical engineering from Montana State University.

John Sheppard is a Norm Asbjornson College of Engineering Distinguished Professor of computer science in the Gianforte School of Computing at Montana State University. He received his PhD in computer science from Johns Hopkins University and is a fellow of the Institute of Electrical and Electronics Engineers. His research interests include extending and applying algorithms in deep learning, probabilistic graphical models, and evolutionary optimization to a variety of application areas, including electronic prognostics and health management, precision agriculture, and medical diagnostics.

Prashant Jha is an associate professor and extension weed specialist in the Department of Agronomy at Iowa State University. His current work includes basic and applied aspects of weed science and identifying innovative, multitactic weed management strategies for herbicide resistance mitigation in diversified cropping systems, and precision weed management. He received his PhD in weed science from Clemson University and is an editorial board member of Weed Science Society of America journals.

Joseph A. Shaw is the director of the Optical Technology Center and professor of optics and electrical engineering at Montana State University in Bozeman, Montana, USA. His current work includes the development of optical remote sensing systems for use in environmental studies, ecosystems research, and precision agriculture. He received his PhD in optical sciences from the University of Arizona and is a fellow of SPIE and the Optical Society of America.

CC BY: © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.

Citation Download Citation

Bryan Scherrer, John Sheppard, Prashant Jha, and Joseph A. Shaw "Hyperspectral imaging and neural networks to classify herbicide-resistant weeds," Journal of Applied Remote Sensing 13(4), 044516 (14 November 2019). https://doi.org/10.1117/1.JRS.13.044516

Received: 3 June 2019; Accepted: 25 October 2019; Published: 14 November 2019

Access the abstract

JOURNAL ARTICLE
15 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

CITATIONS

Cited by 31 scholarly publications.

Explore citations on Lens.org

KEYWORDS

Neural networks

Hyperspectral imaging

Imaging systems

Machine learning

Reflectivity

Matrices

Algorithm development

1.

Introduction

2.

Physical Experiments and Data Collection

Fig. 1

Table 1

Table 2

Fig. 2

Fig. 3

3.

Machine Learning Experiments

3.1.

Image Processing

Fig. 4

3.2.

Neural Networks

Fig. 5

Fig. 6

4.

Results and Analysis

4.1.

Ground-Based Imaging Results

Fig. 7

Fig. 8

Table 3

Fig. 9

Fig. 10

Fig. 11

4.2.

Drone-Based Imaging Results

Fig. 12

Fig. 13

5.

Conclusion

Acknowledgments

References

Biography

Show All Keywords

Keywords/Phrases

Search In:

Publication Years