Probability Models (Vaseghi - Advanced Digital Signal Processing and Noise Reduction)

Advanced Digital Signal Processing and Noise Reduction, Second Edition.Saeed V. VaseghiCopyright © 2000 John Wiley & Sons LtdISBNs: 0-471-62692-9 (Hardback): 0-470-84162-1 (Electronic)3The small probability of collision of the Earth and a comet can become verygreat in adding over a long sequence of centuries.

It is easy to picture theeffects of this impact on the Earth. The axis and the motion of rotation havechanged, the seas abandoning their old position...Pierre-Simon LaplacePROBABILITY MODELS3.13.23.33.43.53.63.7Random Signals and Stochastic ProcessesProbabilistic ModelsStationary and Non-stationary ProcessesExpected Values of a ProcessSome Useful Classes of Random ProcessesTransformation of a Random ProcessSummaryProbability models form the foundation of information theory.Information itself is quantified in terms of the logarithm ofprobability. Probability models are used to characterise and predictthe occurrence of random events in such diverse areas of applications aspredicting the number of telephone calls on a trunk line in a specified periodof the day, road traffic modelling, weather forecasting, financial datamodelling, predicting the effect of drugs given data from medical trials, etc.In signal processing, probability models are used to describe the variationsof random signals in applications such as pattern recognition, signal codingand signal estimation.

This chapter begins with a study of the basic conceptsof random signals and stochastic processes and the models that are used forthe characterisation of random processes. Stochastic processes are classes ofsignals whose fluctuations in time are partially or completely random, suchas speech, music, image, time-varying channels, noise and video. Stochasticsignals are completely described in terms of a probability model, but canalso be characterised with relatively simple statistics, such as the mean, thecorrelation and the power spectrum. We study the concept of ergodicstationary processes in which time averages obtained from a singlerealisation of a process can be used instead of ensemble averages. Weconsider some useful and widely used classes of random signals, and studythe effect of filtering or transformation of a signal on its probabilitydistribution.Random Signals and Stochastic Processes453.1 Random Signals and Stochastic ProcessesSignals, in terms of one of their most fundamental characteristics, can beclassified into two broad categories: deterministic signals and randomsignals.

Random functions of time are often referred to as stochastic signals.In each class, a signal may be continuous or discrete in time, and may havecontinuous-valued or discrete-valued amplitudes.A deterministic signal can be defined as one that traverses apredetermined trajectory in time and space. The exact fluctuations of adeterministic signal can be completely described in terms of a function oftime, and the exact value of the signal at any time is predictable from thefunctional description and the past history of the signal.

For example, a sinewave x(t) can be modelled, and accurately predicted either by a second-orderlinear predictive model or by the more familiar equation x(t)=A sin(2πft+φ).Random signals have unpredictable fluctuations; hence it is not possibleto formulate an equation that can predict the exact future value of a randomsignal from its past history. Most signals such as speech and noise are atleast in part random. The concept of randomness is closely associated withthe concepts of information and noise.

Indeed, much of the work on theprocessing of random signals is concerned with the extraction ofinformation from noisy observations. If a signal is to have a capacity toconvey information, it must have a degree of randomness: a predictablesignal conveys no information. Therefore the random part of a signal iseither the information content of the signal, or noise, or a mixture of bothinformation and noise. Although a random signal is not completelypredictable, it often exhibits a set of well-defined statistical characteristicvalues such as the maximum, the minimum, the mean, the median, thevariance and the power spectrum.

A random process is described in terms ofits statistics, and most completely in terms of a probability model fromwhich all its statistics can be calculated.Example 3.1 Figure 3.1(a) shows a block diagram model of adeterministic discrete-time signal. The model generates an output signalx(m) from the P past samples asx(m) = h1 ( x(m − 1), x(m − 2), ..., x(m − P) )(3.1)where the function h1 may be a linear or a non-linear model. A functionaldescription of the model h1 and the P initial sample values are all that isrequired to predict the future values of the signal x(m). For example for asinusoidal signal generator (or oscillator) Equation (3.1) becomesProbability Models46h 1 (·)z...–1x(m)=h1 (x(m–1), ..., x(m–P))z –1z –1(a)Randominput e(m)h2 (·)z–1...z –1x(m)=h2(x(m–1), ..., x(m–P))+e(m)z –1(b)Figure 3.1 Illustration of deterministic and stochastic signal models: (a) adeterministic signal model, (b) a stochastic signal model.x (m)= a x (m − 1) − x (m − 2)(3.2)where the choice of the parameter a=2cos(2πF0 /Fs) determines theoscillation frequency F0 of the sinusoid, at a sampling frequency of Fs.Figure 3.1(b) is a model for a stochastic random process given byx(m) = h2 (x(m − 1), x(m − 2), ..., x(m − P) )+ e(m)(3.3)where the random input e(m) models the unpredictable part of the signalx(m) , and the function h2 models the part of the signal that is correlatedwith the past samples.

For example, a narrowband, second-orderautoregressive process can be modelled asx(m) = a1 x(m − 1) + a2 x(m − 2) + e(m)(3.4)where the choice of the parameters a1 and a2 will determine the centrefrequency and the bandwidth of the process.Random Signals and Stochastic Processes473.1.1 Stochastic ProcessesThe term “stochastic process” is broadly used to describe a random processthat generates sequential signals such as speech or noise. In signalprocessing terminology, a stochastic process is a probability model of a classof random signals, e.g.

Gaussian process, Markov process, Poisson process,etc. The classic example of a stochastic process is the so-called Brownianmotion of particles in a fluid. Particles in the space of a fluid moverandomly due to bombardment by fluid molecules. The random motion ofeach particle is a single realisation of a stochastic process. The motion of allparticles in the fluid forms the collection or the space of differentrealisations of the process.In this chapter, we are mainly concerned with discrete-time randomprocesses that may occur naturally or may be obtained by sampling acontinuous-time band-limited random process. The term “discrete-timestochastic process” refers to a class of discrete-time random signals, X(m),characterised by a probabilistic model. Each realisation of a discretestochastic process X(m) may be indexed in time and space as x(m,s),where m is the discrete time index, and s is an integer variable thatdesignates a space index to each realisation of the process.3.1.2 The Space or Ensemble of a Random ProcessThe collection of all realisations of a random process is known as theensemble, or the space, of the process.

For an illustration, consider a randomnoise process over a telecommunication network as shown in Figure 3.2.The noise on each telephone line fluctuates randomly with time, and may bedenoted as n(m,s), where m is the discrete time index and s denotes the lineindex. The collection of noise on different lines form the ensemble (or thespace) of the noise process denoted by N(m)={n(m,s)}, where n(m,s)denotes a realisation of the noise process N(m) on the line s. The “true”statistics of a random process are obtained from the averages taken over theensemble of many different realisations of the process.

However, in manypractical cases, only one realisation of a process is available. In Section 3.4,we consider the so-called ergodic processes in which time-averagedstatistics, from a single realisation of a process, may be used instead of theensemble-averaged statistics.Notation The following notation is used in this chapter: X ( m) denotes arandom process, the signal x(m,s) is a particular realisation of the processX(m), the random signal x(m) is any realisation of X(m), and the collection48Probability Modelsmn(m, s-1)mn(m, s)mn(m, s+1)SpaceTimeFigure 3.2 Illustration of three realisations in the space of a random noise N(m).of all realisations of X(m), denoted by {x(m,s)}, form the ensemble or thespace of the random process X(m).3.2 Probabilistic ModelsProbability models provide the most complete mathematical description of arandom process.

For a fixed time instant m, the collection of samplerealisations of a random process {x(m,s)} is a random variable that takes onvarious values across the space s of the process. The main differencebetween a random variable and a random process is that the latter generatesa time series. Therefore, the probability models used for random variablesmay also be applied to random processes. We start this section with thedefinitions of the probability functions for a random variable.The space of a random variable is the collection of all the values, oroutcomes, that the variable can assume.

The space of a random variable canbe partitioned, according to some criteria, into a number of subspaces. Asubspace is a collection of signal values with a common attribute, such as acluster of closely spaced samples, or the collection of samples with theiramplitude within a given band of values. Each subspace is called an event,and the probability of an event A, P(A), is the ratio of the number of49Probabilistic Models65Outcome from event A : die1+die2 > 8Die 24Outcome from event B : die1+die2 ≤ 8310P =A36226P=B361123456Die 1Figure 3.3 A two-dimensional representation of the outcomes of two dice, and thesubspaces associated with the events corresponding to the sum of the dice beinggreater than 8 or, less than or equal to 8.observed outcomes from the space of A, NA, divided by the total number ofobservations:NAP ( A) =(3.5)∑ NiAll events iFrom Equation (3.5), it is evident that the sum of the probabilities of alllikely events in an experiment is unity.Example 3.2 The space of two discrete numbers obtained as outcomes ofthrowing a pair of dice is shown in Figure 3.3.