Chapter 3 Cramer-Rao Lower Bound - Binghamton

1 Chapter 3. Cramer-Rao Lower Bound What is the Cramer-Rao Lower Bound Abbreviated: CRLB or sometimes just CRB. CRLB is a Lower Bound on the variance of any unbiased estimator : If is an unbiased estimator of , then 2 ( ) CRLB ( ) ( ) CRLB ( ).. The CRLB tells us the best we can ever expect to be able to do (w/ an unbiased estimator ). Some Uses of the CRLB. 1. Feasibility studies ( Sensor usefulness, etc.). Can we meet our specifications? 2. Judgment of proposed estimators Estimators that don't achieve CRLB are looked down upon in the technical literature 3. Can sometimes provide form for MVU est. 4. Demonstrates importance of physical and/or signal parameters to the estimation problem We'll see that a signal's BW determines delay est.

2 Accuracy Radars should use wide BW signals Est. Accuracy Consideration Q: What determines how well you can estimate ? samples from a random Recall: Data vector is x process that depends on an . the PDF describes that dependence: p(x; ). Clearly if p(x; ) depends strongly/weakly on . we should be able to estimate well/poorly. See surface plots vs. x & for 2 cases: 1. Strong dependence on . 2. Weak dependence on . Should look at p(x; ) as a function of for fixed value of observed data x Surface Plot Examples of p(x; ). Ex. : PDF Dependence for DC Level in Noise x[0] = A + w[0] w[0] ~ N(0, 2). Then the parameter-dependent PDF of the data point x[0] is: 1 ( x[0] A) 2.

3 P (x[0]; A) = exp 2 . 2 2 2 . Say we observe x[0] = 3 . So Slice at x[0] = 3. p(x[0]=3; ). 3 A. A x[0]. Define: Likelihood Function (LF). The LF = the PDF p(x; ). but as a function of parameter w/ the data vector x fixed We will also often need the Log Likelihood Function (LLF): LLF = ln{LF} = ln{ p(x; )}. LF Characteristics that Affect Accuracy Intuitively: sharpness of the LF sets accuracy But How??? Sharpness is measured using curvature: 2 ln p (x ; ). 2. x = given data = true value Curvature PDF concentration Accuracy . But this is for a particular set of data we want in general : So Average over random vector to give the average curvature: 2 ln p (x ; ) Expected sharpness E of LF.

4 2 . = true value E{ } is p(x; ). Cramer-Rao Lower Bound Theorem CRLB for Scalar Parameter ln p( x; ) . Assume regularity condition is met: E = 0 .. Then 2 1.. Right-Hand 2 ln p (x; ) Side is E 2 . CRLB. = true value E{ } is p(x; ). 2 ln p (x; ) 2 ln p (x; ). E 2 = 2. p( x; )dx . Steps to Find the CRLB. 1. Write log 1ikelihood function as a function of : ln p(x; ). 2. Fix x and take 2nd partial of LLF: 2ln p(x; )/ 2. 3. If result still depends on x: Fix and take expected value x Otherwise skip this step 4. Result may still depend on : Evaluate at each specific value of desired. 5. Negate and form reciprocal Example CRLB for DC in AWGN. x[n] = A + w[n], n = 0, 1, , N 1.

5 W[n] ~ N(0, 2). & white Need likelihood function: N 1. 1 (x [n ] A )2 Due to p (x ; A ) = exp whiteness n =0 2 2 2 2 . N 1 . (x [n ] A ). 2. 1 Property = exp n = 0 of exp (2 ). N. 2 2 2 2 .. Now take ln to get LLF: ( . ). N N 1. 1. ln p ( x; A) = ln 2 2 2 2 . ( x [n ] A )2. 2 n =0. $!!#!!" $!!!#!!!".. (~~) =0 (~~) =? A A. sample Now take first partial A: mean N 1. 1 N. A. ln p ( x; A) =. 2. (x[n] A) = 2 (x A) (!). n =0. Now take partial again: Doesn't depend on x so we don't need to do E{ }. 2 N. 2. ln p ( x; A) = . A 2. Since the result doesn't depend on x or A all we do is negate and form reciprocal to get CRLB: 1 2. CRLB = = 2. 2 ln p (x; ) N var{ A } . E 2 N.

6 = true value CRLB. Doesn't depend on A. For fixed N & 2. Increases linearly with 2. Decreases inversely with N. A. CRLB CRLB Doubling Data Halves CRLB! For fixed N For fixed 2. 2 N. Continuation of Theorem on CRLB. There exists an unbiased estimator that attains the CRLB iff: ln p ( x; ). = I ( )[g ( x ) ] (!).. for some functions I( ) and g(x). Furthermore, the estimator that achieves the CRLB is then given by: Since no unbiased estimator can do better this = g ( x ) is the MVU estimate!! 1 This gives a possible way to find the MVU: . } =. var{with = CRLB Compute ln p(x; )/ (need to anyway). I ( ) Check to see if it can be put in form like (!). If so then g(x) is the MVU esimator Revisit Example to Find MVU Estimate For DC Level in AWGN we found in (!)

7 That: Has form of N. ln p ( x; A) = 2 (x A) I(A)[g(x) A]. A . N 1. 2 1. I ( A) =. N. 2. var{ A } = = CRLB = g ( x ) = x =. N. x[n]. N n =0. So for the DC Level in AWGN: the sample mean is the MVUE!! Definition: Efficient estimator An estimator that is: unbiased and attains the CRLB. is said to be an Efficient estimator . Notes: Not all estimators are efficient (see next example: Phase Est.). Not even all MVU estimators are efficient So there are times when our 1st partial test won't work!!!! Example : CRLB for Phase Estimation This is related to the DSB carrier estimation problem we used for motivation in the notes for Ch. 1. Except here we have a pure sinusoid and we only wish to estimate only its phase AWGN w/ zero Signal Model: x[n ] = A cos(2 f o n + o ) + w[n ].

8 $!! !#!!! " mean & 2. s[ n; o ]. Signal-to-Noise Ratio: Signal Power = A2/2 A2. SNR =. Noise Power = 2 2 2. Assumptions: 1. 0 < fo < ( fo is in cycles/sample). 2. A and fo are known (we'll remove this assumption later). Problem: Find the CRLB for estimating the phase. Exploit We need the PDF: Whiteness and Exp. N 1 Form (x [n ] A cos( 2 f o n + ) ). 2. 1 . p (x ; ) = exp n = 0 . (2 ). N. 2 2 2 2 .. Now taking the log gets rid of the exponential, then taking partial derivative gives (see book for details): ln p (x ; ) A N 1 . 2. A . = 2 x [n ]sin( 2 f o n + ) sin( 4 f o n + 2 ) . n =0 2 . Taking partial derivative again: 2 ln p (x ; ) A N 1. 2. = 2. (x [n ]cos( 2 f o n + ) A cos( 4 f o n + 2 ) ).

9 N =0. Still depends on random vector x so need E{}. Taking the expected value: 2 ln p (x ; ) A N 1 . E 2 = E 2 (x [n ]cos( 2 f o n + ) A cos( 4 f o n + 2 ) ) . n =0 . A N 1. = 2. (E {x [n ]}cos( 2 f o n + ) A cos( 4 f o n + 2 ) ). n =0. E{x[n]} = A cos(2 fon + ). So plug that in, get a cos2 term, use trig identity, and get 2 ln p (x ; ) A2 N 1 N 1 NA 2. E 2 = 1 cos( 4 f o n + 2 ) 2 2 = N SNR. 2 2 n = 0 n 0 . =N << N if fo not near 0 or . n N-1. 1. var{ } . Non-dB. Now invert to get CRLB: N SNR. CRLB Doubling Data Halves CRLB! For fixed SNR. N. CRLB Doubling SNR. Halves CRLB! For fixed N Halve CRLB. for every 3B. in SNR. SNR (non-dB). Does an efficient estimator exist for this problem?

10 The CRLB. theorem says there is only if ln p( x; ). = I ( )[g ( x ) ].. Our earlier result was: ln p (x ; ) A N 1 . 2. A . = 2 x [n ]sin( 2 f o n + ) sin( 4 f o n + 2 ) . n =0 2 . Efficient estimator does NOT exist!!! We'll see later though, an estimator for which var{ } CRLB. as N or as SNR . var{ }. CRB. N. Such an estimator is called an asymptotically efficient estimator (We'll see such a phase estimator in Ch. 7 on MLE).