Bayesian Estimation (779797), страница 2

Просмтор этого файла доступен только зарегистрированным пользователям. Но у нас супер быстрая регистрация: достаточно только электронной почты!

Текст из файла (страница 2)

The difficulty in signal interpolation is that the underlyingexcitation e of the signal x is purely random and, unlike θ, it cannot beestimated through an averaging operation. In this chapter we are concernedwith the parameter estimation problem, although the same ideas also applyto signal interpolation, which is considered in Chapter 11.4.1.4 Performance Measures and Desirable Properties ofEstimatorsIn estimation of a parameter vector θ from N observation samples y, a set ofperformance measures is used to quantify and compare the characteristics ofdifferent estimators. In general an estimate of a parameter vector is afunction of the observation vector y, the length of the observation N and theprocess model M. This dependence may be expressed asθˆ = f ( y,N ,M )(4.4)Different parameter estimators produce different results depending on theestimation method and utilisation of the observation and the influence of theprior information.

Due to randomness of the observations, even the sameestimator would produce different results with different observations fromthe same process. Therefore an estimate is itself a random variable, it has amean and a variance, and it may be described by a probability densityfunction. However, for most cases, it is sufficient to characterise anestimator in terms of the mean and the variance of the estimation error. Themost commonly used performance measures for an estimator are thefollowing:(a) Expected value of estimate:(b) Bias of estimate:(c) Covariance of estimate:E[θˆ ]E[θˆ − θ ]= E[θˆ ] − θCov[θˆ ] = E [(θˆ − E [θˆ])(θˆ − E [θˆ ]) ]Optimal estimators aim for zero bias and minimum estimation errorcovariance. The desirable properties of an estimator can be listed as follows:(a) Unbiased estimator: an estimator of θ is unbiased if the expectationof the estimate is equal to the true parameter value:E [θˆ ] = θ(4.5)95Basic DefinitionsAn estimator is asymptotically unbiased if for increasing length ofobservations N we havelim E [θˆ ] = θN →∞(4.6)(b) Efficient estimator: an unbiased estimator of θ is an efficientestimator if it has the smallest covariance matrix compared with allother unbiased estimates of θ :Cov[θˆEfficient ] ≤ Cov[θˆ](4.7)where θˆ is any other estimate of θ .(c) Consistent estimator: an estimator is consistent if the estimateimproves with the increasing length of the observation N, such thatthe estimate θˆ converges probabilistically to the true value θ as Nbecomes infinitely large:lim P[| θˆ − θ |> ε]=0N →∞(4.8)where ε is arbitrary small.Example 4.1 Consider the bias in the time-averaged estimates of the meanµ y and the variance σ 2y of N observation samples [y(0), ..., y(N–1)], of anergodic random process, given asµˆ y =σˆ 2y =1N1NN −1∑ y ( m)(4.9)m =0N −1∑ [y (m) − µˆ y ]2(4.10)m =0It is easy to show that µˆ y is an unbiased estimate, sinceE [µˆ y ]= 1NN −1∑ E [ y ( m)] = µ ym =0(4.11)96Bayesian EstimationfΘ |Y (θˆ | y )N1 < N2 < N3θˆ1θθˆ2 θˆ3θˆFigure 4.4 Illustration of the decrease in the bias and variance of an asymptoticallyunbiased estimate of the parameter θ with increasing length of observation.The expectation of the estimate of the variance can be expressed asE[σˆ y2]1=E NN −1 1∑  y ( m) − Nm =0 2 21σ +N yN1= σ 2y − σ 2yN= σ 2y−N −1∑ y(k ) k =02σ 2y(4.12)From Equation (4.12), the bias in the estimate of the variance is inverselyproportional to the signal length N, and vanishes as N tends to infinity;hence the estimate is asymptotically unbiased.

In general, the bias and thevariance of an estimate decrease with increasing number of observationsamples N and with improved modelling. Figure 4.4 illustrates the generaldependence of the distribution and the bias and the variance of anasymptotically unbiased estimator on the number of observation samples N.4.1.5 Prior and Posterior Spaces and DistributionsThe prior space of a signal or a parameter vector is the collection of allpossible values that the signal or the parameter vector can assume.

Theposterior signal or parameter space is the subspace of all the likely valuesof a signal or a parameter consistent with both the prior information and theevidence in the observation. Consider a random process with a parameter97Basic Definitionsf(y,θ)f(θ| y2 )θf(θ|y1 )yy21yFigure 4.5 Illustration of joint distribution of signal y and parameter θ and theposterior distribution of θ given y.space Θ observation space Y and a joint pdf fY,Θ(y,θ).

From the Bayes’ rulethe posterior pdf of the parameter vector θ, given an observation vector y,f Θ |Y (θ | y) , can be expressed asfΘ |Y (θ y ) =f Y |Θ ( y θ ) f (θ )Θf ( y)Y=f Y |Θ ( y θ ) fΘ (θ )(4.13)∫ fY |Θ ( y θ ) fΘ (θ ) dθΘwhere, for a given observation vector y, the pdf fY(y) is a constant and hasonly a normalising effect. From Equation (4.13), the posterior pdf isproportional to the product of the likelihood fY|Θ(y|θ) that the observation ywas generated by the parameter vector θ, and the prior pdf f Θ (θ ). The priorpdf gives the unconditional parameter distribution averaged over the entireobservation space asfΘ (θ ) = ∫ fY ,Θ ( y ,θ ) dyY(4.14)98Bayesian EstimationFor most applications, it is relatively convenient to obtain the likelihoodfunction fY|Θ(y|θ).

The prior pdf influences the inference drawn from thelikelihood function by weighting it with f Θ (θ ). The influence of the prioris particularly important for short-length and/or noisy observations, wherethe confidence in the estimate is limited by the lack of a sufficiently longobservation and by the noise.

The influence of the prior on the bias and thevariance of an estimate are considered in Section 4.4.1.A prior knowledge of the signal distribution can be used to confine theestimate to the prior signal space. The observation then guides the estimatorto focus on the posterior space: that is the subspace consistent with both theprior and the observation.

Figure 4.5 illustrates the joint pdf of a signal y(m)and a parameter θ. The prior pdf of θ can be obtained by integratingfY|Θ(y(m)|θ) with respect to y(m). As shown, an observation y(m) cuts aposterior pdf fΘ|Y(θ|y(m)) through the joint distribution.Example 4.2 A noisy signal vector of length N samples is modelled asy(m) = x(m)+ n(m)(4.15)Assume that the signal x(m) is Gaussian with mean vector µx and covariancematrix Σxx, and that the noise n(m) is also Gaussian with mean vector µnand covariance matrix Σnn.

The signal and noise pdfs model the prior spacesof the signal and the noise respectively. Given an observation vector y(m),the underlying signal x(m) would have a likelihood distribution with a meanvector of y(m) – µn and covariance matrix Σnn as shown in Figure 4.6.Thelikelihood function is given byfY |X=( y (m) x (m) ) = f N ( y (m) − x ( m) )1( 2π )N /2Σ nn1/ 2 1−1[ x ( m ) −( y ( m ) − µ n ) ] exp  − [ x ( m ) −( y ( m ) − µ n ) ] T Σ nn 2(4.16)where the terms in the exponential function have been rearranged toemphasize the illustration of the likelihood space in Figure 4.6.

Hence theposterior pdf can be expressed as99Basic DefinitionsNoisy signal spaceSignal priorspaceyA noisyobservationPosterior spaceNoise priorspaceLikelihood spaceFigure 4.6 Sketch of a two-dimensional signal and noise spaces, and thelikelihood and posterior spaces of a noisy observation y.f=X |Y( x ( m) y ( m) ) =f1fY( y (m) x ( m) ) f X ( x ( m) )Y |X( y ( m ) ) ( 2π ) N Σ nnfY( y (m) )11/ 2Σ xx1/ 2−1−1( x ( m ) − µ x ) }× exp  − 1 {[ x ( m ) − ( y ( m ) − µ n ) ]T Σ nn[ x ( m ) − ( y ( m ) − µ n ) ]+ ( x ( m ) − µ x )T Σ xx 2(4.17)For a two-dimensional signal and noise process, the prior spaces of thesignal, the noise, and the noisy signal are illustrated in Figure 4.6. Alsoillustrated are the likelihood and posterior spaces for a noisy observationvector y.

Note that the centre of the posterior space is obtained bysubtracting the noise mean vector from the noisy signal vector. The cleansignal is then somewhere within a subspace determined by the noisevariance.100Bayesian Estimation4.2 Bayesian EstimationThe Bayesian estimation of a parameter vector θ is based on theminimisation of a Bayesian risk function defined as an average cost-of-errorfunction:R (θˆ ) = E[ C (θˆ ,θ )]=∫∫ C (θˆ ,θ ) fY,Θ ( y,θ ) dy dθ= ∫ ∫ C (θˆ ,θ ) fΘ |Y (θ | y ) f Y ( y ) dy dθθ Yθ Y(4.18)where the cost-of-error function C (θˆ ,θ ) allows the appropriate weighting ofthe various outcomes to achieve desirable objective or subjective properties.The cost function can be chosen to associate a high cost with outcomes thatare undesirable or disastrous.

For a given observation vector y, fY(y) is aconstant and has no effect on the risk-minimisation process. Hence Equation(4.18) may be written as a conditional risk function:R (θˆ | y ) = ∫ C (θˆ ,θ ) fΘ |Y (θ | y ) dθθ(4.19)The Bayesian estimate obtained as the minimum-risk parameter vector isgiven byθˆBayesian = arg minR (θˆ | y ) = arg min  ∫ C (θˆ ,θ ) fΘ |Y (θ | y ) dθ  (4.20) θθˆθˆUsing Bayes’ rule, Equation (4.20) can be written asθˆBayesian = arg min  ∫ C (θˆ ,θ ) f Y |Θ ( y | θ ) fΘ (θ ) dθ  θθˆ(4.21)Assuming that the risk function is differentiable, and has a well-definedminimum, the Bayesian estimate can be obtained as∂∂R (θˆ | y )θˆBayesian = arg zero= arg zero  ∫ C (θˆ ,θ ) fY |Θ ( y | θ ) fΘ (θ ) dθ ∂θˆ ∂θˆ θθˆθˆ(4.22)101Bayesian EstimationfΘ |Y (θ | y )C (θˆ ,θ )θθˆMAPFigure 4.7 Illustration of the Bayesian cost function for the MAP estimate.4.2.1 Maximum A Posteriori EstimationThe maximum a posteriori (MAP) estimate θˆ MAP is obtained as theparameter vector that maximises the posterior pdf f Θ |Y ( θ | y ) .

The MAPestimate corresponds to a Bayesian estimate with a so-called uniform costfunction (in fact, as shown in Figure 4.7 the cost function is notch-shaped)defined as(4.23)C (θˆ ,θ ) = 1 − δ (θˆ ,θ )where δ(θˆ ,θ ) is the Kronecker delta function. Substitution of the costfunction in the Bayesian risk equation yieldsR MAP (θˆ | y ) = ∫ [1 − δ (θˆ ,θ )] fΘ |Y (θ | y ) dθθ=1 − fΘ |Y (θˆ | y )(4.24)From Equation (4.24), the minimum Bayesian risk estimate corresponds tothe parameter value where the posterior function attains a maximum. Hencethe MAP estimate of the parameter vector θ is obtained from a minimisationof the risk Equation (4.24) or equivalently maximisation of the posteriorfunction:θˆMAP = arg max f(θ | y )Θ |Yθ(4.25)( y|θ ) f (θ )]= arg max [ fθY |θΘ102Bayesian Estimation4.2.2 Maximum-Likelihood EstimationThe maximum-likelihood (ML) estimate θˆML is obtained as the parametervector that maximises the likelihood function f Y |Θ ( y |θ ) .

Характеристики

Список файлов книги