chapter 4, Optimal Filtering

from: Random Processes for Image and Signal Processing

Author(s): Edward R. Dougherty

Published: 27 October 1998

Chapter DOI: 10.1117/3.268105.ch4

Chapter Page Count: 176 pages

Previous Chapter | Next Chapter

Front Matter

1. Probability Theory

2. Random Processes

3. Canonical Representation

4. Optimal Filtering

5. Random Models

Back Matter

Preview

Chapter Contents

4.1. Optimal Mean-Square-Error Filters
4.1.1. Conditional Expectation
4.1.2. Optimal Nonlinear Filter
4.1.3. Optimal Filter for Jointly Normal Random Variables
4.1.4. Multiple Observation Variables
4.1.5. Bayesian Parametric Estimation
4.2. Optimal Finite-Observation Linear Filters
4.2.1. Linear Filters and the Orthogonality Principle
4.2.2. Design of the Optimal Linear Filter
4.2.3. Optimal Linear Filter in the Jointly Gaussian Case
4.2.4. Role of Wide-Sense Stationarity
4.2.5. Signal-plus-Noise Model
4.2.6. Edge Detection
4.3. Steepest Descent
4.3.1. Steepest Descent Iterative Algorithm
4.3.2. Convergence of the Steepest-Descent Algorithm
4.3.3. Least-Mean-Square Adaptive Algorithm
4.3.4. Convergence of the LMS Algorithm
4.3.5. Nonstationary Processes
4.4. Least-Squares Estimation
4.4.1. Pseudoinverse Estimator
4.4.2. Least-Squares Estimation for Nonwhite Noise
4.4.3. Multiple Linear Regression
4.4.4. Least-Squares Image Restoration
4.5. Optimal Linear Estimation of Random Vectors
4.5.1. Optimal Linear Filter for Linearly Dependent Observations
4.5.2. Optimal Estimation of Random Vectors
4.5.3. Optimal Linear Filters for Random Vectors
4.6. Recursive Linear Filters
4.6.1. Recursive Generation of Direct Sums
4.6.2. Static Recursive Optimal Linear Filtering
4.6.3. Dynamic Recursive Optimal Linear Filtering
4.7. Optimal Infinite-Observation Linear Filters
4.7.1. Wiener-Hopf Equation
4.7.2. Wiener Filter
4.8. Optimal Linear Filter in the Context of a Linear Model
4.8.1. The Linear Signal Model
4.8.2. Procedure for Finding the Optimal Linear Filter
4.8.3. Additive White Noise
4.8.4. Discrete Domains
4.9. Optimal Linear Filters via Canonical Expansions
4.9.1. Integral Decomposition into White Noise
4.9.2. Integral Equations Involving the Autocorrelation Function
4.9.3. Solution via Discrete Canonical Expansions
4.10. Optimal Binary Filters
4.10.1. Binary Conditional Expectation
4.10.2. Boolean Functions and Optimal Translation-Invariant Filters
4.10.3. Optimal Increasing Filters
4.11. Pattern Classification
4.11.1. Optimal Classifiers
4.11.2. Gaussian Maximum-Likelihood Classification
4.11.3. Linear Discriminants
4.12. Neural Networks
4.12.1. Two-Layer Neural Networks
4.12.2. Steepest Descent for Nonquadratic Error Surfaces
4.12.3. Sum-of-Squares Error
4.12.4. Error Back-Propagation
4.12.5. Error Back-Propagation for Multiple Outputs
4.12.6. Adaptive Network Design
Exercises for Chapter 4

Excerpt

4.1. Optimal Mean-Square-Error Filters

A fundamental problem in engineering is estimation (prediction) of an outcome of an unobserved random variable based on outcomes of a set of observed random variables. In the context of random processes, we wish to estimate values of a random function Y(s) based on observation of a random function X(t). From a filtering perspective, we desire a system which, given an input X(t), produces an output Ŷ(s) that best estimates Y(s), where the goodness of the estimator is measured by a probabilistic error measure between the estimator and the random variable it estimates.

For two random variables, X to be observed and Y to be estimated, we desire a function of X, say ψ(X), such that ψ(X) minimizes the mean-square error (MSE)

If such an estimation rule ψ can be found, then ψ(X) is called an optimal mean-square-error estimator of Y in terms of X. To make the estimation problem mathematically or computationally tractable, or to obtain an estimation rule with desirable properties, we often restrict the class of estimation rules over which the MSE minimum is to be achieved. The constraint trade-off is a higher MSE in return for a tractable design or desirable filter properties. In this section we examine the theoretically best solution over all functions of X. Subsequently, we focus on linear estimation in order to discover systems that provide optimal linear filtering. The theory, both linear and nonlinear, applies to complex random variables; however, our exposition assumes that all random variables are real valued.

4.1.1. Conditional Expectation

If two random variables X and Y possess joint density f(x, y) and an observation of X, say X = x, is given, then the density f(y|x) of the conditional random variable Y given X = x is defined by Eq. 1.135. The conditional expectation (or mean) E[Y|x] (or μ_Y|x) is defined by Eq. 1.137. Letting x vary, the plot of E[Y|x] is the regression curve of Y on X.