To efficiently use the limited bandwidth available on the downlink from satellite to ground station, imager data
is usually compressed before transmission. Transmission introduces unavoidable errors, which are only partially
removed by forward error correction and packetization. In the case of the commonly used CCSD Rice-based
compression, it results in a contiguous sequence of dummy values along scan lines in a band of the imager data.
We have developed a method capable of using the image statistics to provide a principled estimate of the missing
data. Our method outperforms interpolation yet can be performed fast enough to provide uninterrupted data
flow. The estimation of the lost data provides significant value to end users who may use only part of the data,
may not have statistical tools, or lack the expertise to mitigate the impact of the lost data. Since the locations of
the lost data will be clearly marked as meta-data in the HDF or NetCDF header, experts who prefer to handle
error mitigation themselves will be free to use or ignore our estimates as they see fit.
This paper reports a comparative study of current lossless compression algorithms for data from a representative
selection of satellite based earth science multispectral imagers. The study includes the performance of compression
algorithms on Advanced Very High Resolution Radiometer(AVHRR), SEVIRI, the Moderate Resolution
Imaging Spectroradiometer(MODIS) imager, as well as a subset of MODIS bands as a proxy for the upcoming
GOES-R series. SEVIRI aboard the ESA/EUMETSAT operated Meteosat Second Generation (MSG) satellites
is a geostationary imager. The AVHRR aboard the NOAA Polar Orbiting Environmental Satellites and MODIS
aboard the NASA Terra and Aqua satellites have polar orbits. Thus this study will present representatives
from both polar and geostationary orbiting imagers. The imagers we include have sensors for both reflected
and emissive radiance. We also note that the older satellites have coarser quantizations and present our conclusions
on the impact on compression ratios. Faced with a enormous growing large volume of data on a new
emerging current generation images from faster scanning, finer spatial resolution, and greater spectral resolution,
this study provides a comparison of current compression algorithms as a baseline for future work. With
growing satellite Earth science multispectral imager volume data, it becomes increasingly important to evaluate
which compression algorithms are most appropriate for data management in transmission and archiving. This
comparative compression study uses a wide range standard implementations of the leading lossless compression
algorithms. Examples include image compression algorithms such as PNG and JPEG2000, and widely-used file
compression formats such as BZIP2 and 7z. This study includes a comparison with the Consultative Committee
for Space Data Systems (CCSDS) recommended Szip software which uses the extended-Rice lossless compression
algorithm as well as the most recent recommended compression standard which relies on a wavelet transform
followed by an entropy coder. To establish statistical significance of our analysis, we have developed a system to
acquire and manage a large number of imager granules: currently over 1000 MODIS granules, over 2400 AVHRR
granules, and over 220 SEVIRI granules.
Multispectral, hyperspectral and ultraspectral imagers and sounders are increasingly important for atmospheric
science and weather forecasting. The recent advent of multipsectral and hyperspectral sensors measuring radiances
in the emissive IR are providing valuable new information. This is due to the presence of spectral channels
(in some cases micro-channels) which are carefully positioned in and out of absorption lines of CO2, ozone, and
water vapor. These spectral bands are used for measuring surface/cloud temperature, atmospheric temperature,
Cirrus clouds water vapor, cloud properties/ozone, and cloud top altidude etc.
The complexity of the spectral structure wherein the emissive bands have been selected presents challenges
for lossless data compression; these are qualitatively different than the challenges offered by the reflective bands.
For a hyperspectral sounder such as AIRS, the large number of channels is the principal contributor to data size.
We have shown that methods combining clustering and linear models in the spectral channels can be effective
for lossless data compression. However, when the number of emissive channels is relatively small compared to
the spatial resolution, such as with the 17 emissive channels of MODIS, such techniques are not effective. In
previous work the CCNY-NOAA compression group has reported an algorithm which addresses this case by
sequential prediction of the spatial image. While that algorithm demonstrated an improved compression ratio
over pure JPEG2000 compression, it underperformed optimal compression ratios estimated from entropy. In
order to effectively exploit the redundant information in a progressive prediction scheme we must, determine a
sequence of bands in which each band has sufficient mutual information with the next band, so that it predicts
it well.
We will provide a covariance and mutual information based analysis of the pairwise dependence between
the bands and compare this with the qualitative expected dependence suggested by a physical analysis. This
compression research is managed by Roger Heymann, PE of OSD NOAA NESDIS Engineering, in collaboration
with the NOAA NESDIS STAR Research Office through Mitch Goldberg, Tim Schmit, Walter Wolf.
Multispectral imaging is becoming an increasingly important tool for monitoring the earth and its environment
from space borne and airborne platforms. Multispectral imaging data consists of visible and IR measurements
from a scene across space and spectrum. Growing data rates resulting from faster scanning and finer spatial and
spectral resolution makes compression an increasingly critical tool to reduce data volume for transmission and
archiving. Research for NOAA NESDIS has been directed to finding for the characteristics of satellite atmospheric
Earth science Imager sensor data what level of Lossless compression ratio can be obtained as well as appropriate
types of mathematics and approaches that can lead to approaching this data's entropy level. Conventional
lossless do not achieve the theoretical limits for lossless compression on imager data as estimated from the
Shannon entropy. In a previous paper, the authors introduce a lossless compression algorithm developed for
MODIS as a proxy for future NOAA-NESDIS satellite based Earth science multispectral imagers such as GOES-R.
The algorithm is based on capturing spectral correlations using spectral prediction, and spatial correlations
with a linear transform encoder. In decompression, the algorithm uses a statistically computed look up table to
iteratively predict each channel from a channel decompressed in the previous iteration. In this paper we present
a new approach which fundamentally differs from our prior work. In this new approach, instead of having a
single predictor for each pair of bands we introduce a piecewise spatially varying predictor which significantly
improves the compression results. Our new algorithm also now optimizes the sequence of channels we use for
prediction. Our results are evaluated by comparison with a state of the art wavelet based image compression
scheme, Jpeg2000. We present results on the 14 channel subset of the MODIS imager, which serves as a proxy
for the GOES-R imager. We will also show results of the algorithm for on NOAA AVHRR data and data from
SEVIRI. The algorithm is designed to be adapted to the wide range of multispectral imagers and should facilitate
distribution of data throughout globally. This compression research is managed by Roger Heymann, PE of OSD
NOAA NESDIS Engineering, in collaboration with the NOAA NESDIS STAR Research Office through Mitch
Goldberg, Tim Schmit, Walter Wolf.
Multispectral imaging is becoming an increasingly important tool for monitoring the earth and its environment
from space borne and airborne platforms. Multispectral imaging data consists of visible and IR measurements
from a scene across space and spectrum. Growing data rates resulting from faster scanning and finer spatial
and spectral resolution makes compression an increasingly critical tool to reduce data volume for transmission
and archiving. Examples of multispectral sensors we consider include the NASA 36 band MODIS imager,
Meteosat 2nd generation 12 band SEVIRI imager, GOES R series 16 band ABI imager, current generation
GOES 5 band imager, and Japan's 5 band MTSAT imager. Conventional lossless compression algorithms are
not able to reach satisfactory compression ratios nor are they near the upper limits for lossless compression
on imager data as estimated from the Shannon entropy. We introduce a new lossless compression algorithm
developed for the NOAA-NESDIS satellite based Earth science multispectral imagers. The algorithm is based
on capturing spectral correlations using spectral prediction, and spatial correlations with a linear transform
encoder. Our results are evaluated by comparison with current sattelite compression algorithms such the new
CCSDS standard compression algorithm, and JPEG2000. The algorithm as presented has been designed to
work with NOAA's scientific data and so is purely lossless but lossy modes can be supported. The compression
algorithm also structures the data in a way that makes it easy to incorporate robust error correction using FEC
coding methods as TPC and LDPC for satellite use. This research was funded by NOAA-NESDIS for its Earth
observing satellite program and NOAA goals.
Despite tremendous efforts to avoid them, stripes are a re-occurring problem for many remote imaging sensors.
Much work has focused on suppressing or eliminating them in order to recover accurate observed radiances.
Beyond the obvious need to eliminate stripes to obtain accurate scientific measurements, stripes can also significantly
impact the performance of compression algorithms. Many compression algorithms are based on linear
representations of image space or assume the data to be relatively smooth. In contrast stripes produce nonlinearities
in the data as well as sharp discontinuities which make it seem necessary to describe the images with
many parameters. Yet the sources and nature of the stripes are often not well known, they could come from
specific irregularities with the sensors. If the a priori construction of the sensor is accounted for, and the stripe
statistically modeled, it is possible to transmit the stripe parameters separately along with de-striped images.
The de-striped images have image statistics whose assumptions are much closer to those for which standard
compression algorithms are optimized. As an example, we show this yields a significant boost in the performance
of these algorithms when applied to the de-striped MODIS images.
This paper reports a comparative study of lossless compression algorithms for MODIS data. MODIS, The
Moderate Resolution Imaging Spectroradiometer, is a 36 band Visible and IR multispectral imager aboard the
Terra and Aqua satellites, having spatial resolution ranging from 0.250 to 1 kilometer and spectral resolution
ranging from 0.405 -0.420 to 4.482-4.549 microns. MODIS data rates are 10.6 Mbps (peak daytime); and 6.1
Mbps (orbital average). Faced with such an enormous volume of data on a current generation imager, this study
provides a comparison of current compression algorithms as a baseline for future work. The Hierarchical Data
Format (HDF) is standard format selected for data archiving and distribution within the Earth Observing System
Data and Information System (EOSDIS). Currently this system handles over one terabyte of data daily, and this
volume continues to increase over time. With growing satellite Earth science multispectral imager volume data
compression, it becomes increasingly important to evaluate which compression algorithms are most appropriate
for data management in transmission and archiving. This comparative compression study uses a wide range
standard implementations of the leading lossless compression algorithms. Examples include image compression
algorithms such as PNG and JPEG2000, and widely-used file compression formats such as BZIP2 and 7z. This
study includes a comparison with the Consultative Committee for Space Data Systems (CCSDS) most recent
recommended compression standard. by a significant margin.
As new instruments are developed, it is becoming clear that our ability to generate data is rapidly outstripping our ability to transmit this data. The Advanced Baseline Imager (ABI), that is currently being developed as the future imager on the Geostationary Environmental Satellite (GOES-R) series, will offer more spectral bands, higher spatial resolution, and faster imaging than the current GOES imager. As a result of the instrument development, enormous amounts of data must be transmitted from the platform to the ground, redistributed globally through band-limited channels, as well as archived. This makes efficient compression critical. According to Shannon's Noiseless Coding Theorem, an a upper bound on the compression ratio can be computed by estimating the entropy of the data. Since the data is essentially a stream, we must determine a partition of the data into samples that capture the important correlations. We use a spatial window partition so that as the window size is increased the estimated entropy stabilizes. As part of our analysis we show that we can estimate the entropy despite the high-dimensionality of the data. We achieve this by using nearest neighbor based estimates. We complement these a posteriori estimates with a priori estimates based on an analysis of sensor noise. Using this noise analysis we propose an upper bound on the compression achievable. We apply our analysis to an ABI proxy in order estimate bounds for compression on the upcoming GOES-R imager.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.