2.1

Introduction to RTS

RTS results from dark current fluctuations of pixels between two or more levels. Such effects are expected to be randomly distributed over the detectors, unless there is a specific circumstance allowing its particular location on some detector surface region e.g. physical defect. Pixels suffering from RTS are commonly assigned as blinker pixels in dark, since such defects could be observed as temporal flashing elements in top of noise images. It is paramount to have RTS pixels characterized in a way that the rates of both false positives and negatives are strongly mitigated, because of the potential impact such pixels might represent for an optical instrument. Pixels with RTS defects will degrade system performance in a global manner. The impact leads to a degraded radiometric accuracy and subsequent lower quality mission data products, especially if the quantity and severity of such defects are also degrading during mission’s lifetime. This effect is normally illustrated with the pixel signal time-series (TS) plot for a large number of frames, see example below in Figure 1. The goal of a RTS detection method [1] is to identify the location of the defective pixels. Regarding the pixels RTS defects characterization, the objective is to measure:

Figure 1.

(Top) normal pixel behavior i.e. noise. (Bottom) pixel presenting multi-level RTS behavior. The x-axis represents the frame number and y-axis the pixel signal normalized for the given time-series.

There are several algorithms able to both detect and characterize RTS defects. The so-called classical RTS detection methods limit the need to perform human visual inspection on dubious cases. Such methods could be nevertheless specified and optimized for distinct detector technologies in order to minimize false detection rates but also to increase the characterization accuracy. At Airbus, we have used in recent years two of the most frequently used methods for detections and characterization of RTS effects for optical instruments development, for two distinct optical instruments:

However, the classical methods require significant configuration and adjustments based also on empirical observations for a given yet to be tested detector type or architecture. Despite that, since large test data sets are often required in order to perform a representative RTS effect analysis, the processing framework, allowing run long lasting algorithms, becomes one major driver.

2.2 RTS

processing using deep learning

As stated above, the analysis of RTS pixels on a detector requires human intervention which could impact the outcome of the results with respect to false detections (both negative and positive). Such intervention becomes particularly relevant in cases with a large number of detections i.e. detectors with large number of detected pixels represent a higher effort to inspect. Given this, harnessing the capability of training a model able to distinguish these false detections, will improve the accuracy of the detection and as consequence of its defects characterization – quality. The approach tested is to use automatic detection (and characterization) based on machine learning using the inspector‘s experience to train the system. The demonstration setup is a sequential model with a simple CNN [4].

Below on figure 2, it is illustrated one pixel time-series in perspective to the complete test data set i.e. large number of detector images. On figure 3 is depicted the simplified overview on the RTS detention and characterization process done pixel-by-pixel.

Figure 2.

Test data set including the pixel time-series to be checked for RTS.

Figure 3.

Simplified overview on the RTS detection and characterization process.

The current results represent a preliminary outcome on using machine learning along with optical space borne instrument on-ground characterization. Nonetheless, these initial results are very promising due to the early phase of development on the machine learning application demonstration, which is not mature for systematic instrument data analysis yet. The test accuracy performed with the trained module reached ~96% on the training data set. There is still room to improve the data used for training. Comparing detection results between the machine learning model and the classical time-series histogram algorithm using a data set of time-series for detector images of approximately 600x260 pixels, reach less than 1% of absolute uncoherent detections between the two approaches i.e. classical vs machine learning. The accurate and detailed characterization of detected RTS defects, using this new machine learning application, is a key aspect still to be archived.

Additionally, since in this subject where processing power and long lasting data acquisitions are crucial to estimate the pixel defects impact on system performance, the time required to obtain a representative outcome is one quantitative benchmark. The machine learning application achieved results that are substantially faster compared to a classical algorithm based on histograms time-series standards. The machine learning method has generated the same equivalent output approximately 20 times faster. Such outcome is a clear demonstration justifying the feasibility of this approach.

Table 1.

Result using the same test data set with two independent RTS analysis algorithms for equivalent outcomes.

Still as major enabler to use machine learning for RTS analysis: it is mandatory to have an accurate classical method in place to classify and characterize the pixels presenting RTS behavior on high resolution detectors. It is the only possibility to allow model’s training data generation since data acquisitions obtained with physical hardware could then be used along. As alternative, instrument synthetic (simulated) data should be used within the same model training objective. Despite the early stage of development of this breakthrough methodology to address RTS, the use of artificial intelligence and specifically machine learning along with optical instruments characterization and calibration represents a disruptive approach with the current methodologies applied with the space systems development during the last decades. Thus, using machine learning on detection chain validation and more generally at system instrument performance plus calibration brings a substantial improvement: more quality in substantial less time.

3.1

Introduction to ghost stray light within space-spectrometers

Airborne space instruments and spectrometers particularly need an extremely competitive design and long calibration campaigns before launch in order to reach high performance. To improve the radiometric accuracy, it is very important to characterize and correct the whole system response. It concerns detector features like detector non-uniformity, defective pixels but also optical components as none optics are perfects. Airbus produces many cutting-edge space spectrometers, however to continuously improve products, there is a need to develop co-designed spectrometers or space optical components, mixing the best existing optical know-how and state-of-the-art processing and post-processing computing methods. This need is particularly important for ghost stray light characterization and processing. Stray light, which is non-nominal light reaching sensors, is mainly caused by contamination or optical roughness leading to scattering or Mie diffraction. Another cause is Fresnel reflection also known as ghost stray light which is caused by in-field or out-field reflections on opto-mechanical elements, as shown in figure 4.

Figure 4.

Stray light examples (red rays) and nominal light (blue rays).

Figure 5, adapted from [5], shows for a given scene the ghost stray light impact on radiometric accuracy. Therefore, there is a need to use post-processing ghost stray light methods in order to improve radiometric accuracy. Such a common method uses kernel deconvolution of the acquired image. Therefore, kernels (or impulse response) have to be measured, simulated or either extrapolated for each sampling point in the spectrometer parameters space during calibration campaigns (in the case of a slit-imaging spectrometer the number of measurement might be the number of pixels of the sensors, representing wavelength and position in the imaging slit). Then, those kernels are stored and used to post-process images coming from the calibrated spectrometer. However, this kind of calibration campaign is very expensive and could be imprecise and non-robust to every ghost shape with state-of-the-art ghost stray light identification algorithm. Indeed, ghost power could be very low compared to nominal light response, overlaps could occur in between ghosts or between ghosts and nominal light. Ghost imaging could also be noisy and this render the task of ghost detection very difficult and time-consuming. Moreover, as ghosts could be unpredictable during design they are often seen lately in the integration phase and are either characterized by costly and long calibration campaign or either ignored. As consequences, the balance cost versus performance is not worthy, and most of the time, a lot of ghost are ignored or either estimated by extrapolation on a few measurements. Therefore, a very precise and less expensive pixel-wise segmentation of ghost stray light is necessary to increase performance of the whole system.

Figure 5.

Ghost map and ghost impact on radiometric accuracy, adapted from [5].

3.2

Ghost stray light segmentation with deep learning

A breaking new solution imagined by Airbus is to use the calculation power of machine learning methods in order to automatize ghost stray light characterization on impulsion response within a spectrometer. For this task, Airbus used a deep learning method based on CNN. The purpose of the algorithm is to automatically learn to segment ghost stray light from the whole point spread function (PSF) over the set of calibration images. This method, then allow to precisely identify ghost stray light with fast implementation over correction kernels. Moreover, the more precise the segmentation, the fewer images are needed during calibration and then, ghost evolution over space-parameters could be precisely extrapolated. This method paves the way of AI applied for space spectrometer characterization and is the subject of a 2019 patent filing [6].

Convolution neural networks used for this segmentation task are successive layers constituted by local tunable convolution filters. The algorithms learn using gradient descent in order to compare given outputs to ground truth and then tune their own filters in order to match output and ground truth [4]. Basically, tunable parameters are kernel coefficient for each neuron. In our use case, the purpose is to generate a filter capable to individually and pixel-wisely track ghost stray light from other kind of light reaching the sensor and for every calibration images as for example shown in figure 6.

Figure 6.

CNN purpose: identify pixel-wisely each ghost stray light and segment from nominal path light for each images acquired or simulated.

This part focuses on multi-reflection induced ghost stray light 2D segmentation within spectrometers images, so called pixel-wise segmentation. This is a first step for tackling the problem of ghost identification. Therefore, the first need is to build an adapted training database for the model, where each pixel is associated to a label referencing to nominal light, noise or kinds of ghost stray light. With such a database, it is then possible to train a model of CNN by showing images and labels in order to automatically fine-tune the model. Several databases have been generated from optical simulation model. The principle used to generate revealing database and associated labels is the described by the following 3 steps; as filed as patent in 2020 [7]:

Figure 7.

Independent generation with an optical model of nominal light (a), ghost 1 (b) and ghost 2 (c), for example each for 2 positions in the field.

Figure 8.

Reconstruction of images with nominal light and ghosts, where labels are associated to each images from figure 7 and fully characterized.

Figure 9.

Different artificial images acquired with previous methods, nominal light and ghost shapes are in different relative positions within the sensors.

Two algorithms have been developed in order to compare results and find an adapted architecture to the problem. The first one is a version of the well-known U-net CNN segmentation algorithm [8], particularly used for medical images segmentation. The second one (figure 10) is a 4 layers-CNN where hyper-parameters like the number of layers, filters or activation functions have been fine tuned to optimize results.

Figure 10.

A CNN architecture used with example of input/output compared to ground truth.

The characteristics of the CNN (figure 9) for a total of 4 million tunable filter parameters are shown in table 2:

Table 2.

CNN features for figure 10 model.

In comparison, the U-net segmentation architecture adapted to ghost stray light clustering has 60 000 independent tunable parameters. Two indicators of results have been chosen in order to quantify the quality of ghost segmentation, including noised background detection. Accuracy shows the ratio of pixel-wise label detected over all the test images and for every label (nominal light, ghosts, noisy background). The Intersection over Union (IoU) index is a more useful criterion representing the overall matching percentage between each label and each detected ghost area. Experiments have been done on a regular computer. The given database presented here and used to train CNN is made of 5 labels with 3 ghosts to segment. The basis of this database is made of 60 images, and then data augmentation has been used in order to efficiently train the models on many examples with different SNR, power variation between ghosts and nominal light, overlapping shapes. The characteristics of this given database, where we got best results for a given training time is shown in table 3.

Table 3.

Training database constitution before and after data augmentation.

It has been possible during test with new images (which have never been seen by the model during training) to reach a 97 % accuracy, and a 65 % IoU index level with the 4-layers CNN (Figure 9). The results for test and revealing images for the 2 architectures and this given training database are shown in table 4.

Table 4.

Training database constitution before and after data augmentation.

The two architectures could reach similar results however the U-net architecture is quite longer to train and that is why Airbus trained a new CNN structure more efficient giving the Figure 10 outputs.