1. Regression model: y = X β + u, u ∼ N(0 , σ

8 Generalized Least Squares Method (GLS, ^一般化最 小自乗法 )

1. Regression model: y = X β + u, u ∼ N(0 , σ

Ω ) 2. Heteroscedasticity (

)

σ

Ω =

 







σ

0 · · · 0 0 σ

... ...

... ... ... 0 0 · · · 0 σ

 







First-Order Autocorrelation (

)

In the case of time series data, the subscript is conventionally given by t, not i . u

= ρ u

+

,

∼ iid N(0 , σ

)

σ

Ω = σ

1 − ρ

 











1 ρ ρ

· · · ρ

ρ 1 ρ · · · ρ

ρ

ρ 1 · · · ρ

... ... ... ... ...

ρ

ρ

ρ

· · · 1

 











V(u

) = σ

= σ

1 − ρ

3. The Generalized Least Squares (GLS

) estimator of β ,

denoted by b, solves the following minimization problem:

min

b

(y − Xb)

Ω

(y − Xb)

The GLSE of β is:

b = (X

Ω

X)

X

Ω

y

4. In general, when Ω is symmetric, Ω is decomposed as follows.

Ω = A

Λ A

Λ is a diagonal matrix, where the diagonal elements of Λ are given by the eigen values.

A is a matrix consisting of eigen vectors.

When Ω is a positive definite matrix, all the diagonal elements of Λ are positive.

5. There exists P such that Ω = PP

(i.e., take P = A

Λ

). = ⇒ P

Ω P

= I

Multiply P

on both sides of y = X β + u.

We have:

y

= X

β + u

,

where y

= P

y, X

= P

X, and u

= P

u.

The variance of u

is:

V(u

) = V(P

u) = P

V(u)P

= σ

P

Ω P

= σ

I

. because Ω = PP

, i.e., P

Ω P

= I

.

Accordingly, the regression model is rewritten as:

y

= X

β + u

, u

∼ (0 , σ

I

)

Apply OLS to the above model.

Let b be as estimator of β from the above model.

That is, the minimization problem is given by:

min

b

(y

− X

b)

(y

− X

b) ,

which is equivalent to:

min

b

(y − Xb)

Ω

(y − Xb) .

Solving the minimization problem above, we have the following estimator:

b = (X

X

)

X

y

= (X

Ω

X)

X

Ω

y ,

which is called GLS (Generalized Least Squares) estimator.

b is rewritten as follows:

b = β + (X

X

)

X

u

= β + (X

Ω

X)

X

Ω

u The mean and variance of b are given by:

E(b) = β,

V(b) = σ

(X

X

)

= σ

(X

Ω

X)

. 6. Suppose that the regression model is given by:

y = X β + u , u ∼ N(0 , σ

Ω ) . In this case, when we use OLS, what happens?

β ˆ = (X

X)

X

y = β + (X

X)

X

u

V( ˆ β ) = σ

(X

X)

X

Ω X(X

X)

Compare GLS and OLS.

(a) Expectation:

E( ˆ β ) = β, and E(b) = β Thus, both ˆ β and b are unbiased estimator.

(b) Variance:

V( ˆ β ) = σ

(X

X)

X

Ω X(X

X)

V(b) = σ

(X

Ω

X)

Which is more e ffi cient, OLS or GLS?.

V( ˆ β ) − V(b) = σ

(X

X)

X

Ω X(X

X)

− σ

(X

Ω

X)

= σ

(

(X

X)

X

− (X

Ω

X)

X

Ω

) Ω

× (

(X

X)

X

− (X

Ω

X)

X

Ω

)

= σ

A Ω A

Ω is the variance-covariance matrix of u, which is a positive definite ma- trix.

Therefore, except for Ω = I

, A Ω A

is also a positive definite matrix.

This implies that V( ˆ β

) − V(b

) > 0 for the ith element of β . Accordingly, b is more e ffi cient than ˆ β .

7. If u ∼ N(0 , σ

Ω ), then b ∼ N( β, σ

(X

Ω

X)

).

Consider testing the hypothesis H

: R β = r.

R : G × k, rank(R) = G ≤ k.

Rb ∼ N(R β, σ

R(X

Ω

X)

R

).

Therefore, the following quadratic form is distributed as:

(Rb − r)

(R(X

Ω

X)

R

)

(Rb − r)

σ

∼ χ

(G)

8. Because (y

− X

b)

(y

− X

b) /σ

∼ χ

(n − k), we obtain:

(y − Xb)

Ω

(y − Xb)

σ

∼ χ

(n − k)

9. Furthermore, from the fact that b is independent of y − Xb, the following F

distribution can be derived:

(Rb − r)

(R(X

Ω

X)

R

)

(Rb − r) / G

(y − Xb)

Ω

(y − Xb) / (n − k) ∼ F(G , n − k) 10. Let b be the unrestricted GLSE and ˜b be the restricted GLSE.

Their residuals are given by e and ˜u, respectively.

e = y − Xb , ˜u = y − X ˜b

Then, the F test statistic is written as follows:

( ˜u

Ω

˜u − e

Ω

e) / G

e

Ω

e / (n − k) ∼ F(G , n − k)

8.1 Example: Mixed Estimation (Theil and Goldberger Model)

A generalization of the restricted OLS = ⇒ Stochastic linear restriction:

r = R β + v , E(v) = 0 and V(v) = σ

Ψ y = X β + u , E(u) = 0 and V(u) = σ

I

Using a matrix form,

( y

r )

=

( X

R )

β +

( u

v )

, E

( u

v )

=

( 0

0 )

and V

( u

v )

= σ

( I

0

0 Ψ

)

For estimation, we do not need normality assumption.

Applying GLS, we obtain:

b =

 

 ( X

R

)

( I

0

0 Ψ

)

( X

R ) 

−1





 ( X

R

)

( I

0

0 Ψ

)

( y r

) 

= (

X

X + R

Ψ

R )

(

X

y + R

Ψ

r )

.

Mean and Variance of b: b is rewritten as follows:

b =

 

 ( X

R

)

( I

0

0 Ψ

)

( X

R ) 

−1





 ( X

R

)

( I

0

0 Ψ

)

( y r

) 

= β +

 

 ( X

R

)

( I

0

0 Ψ

)

( X

R ) 

−1

( u v )

Therefore, the mean and variance are given by:

E(b) = β = ⇒ b is unbiased.

V(b) = σ

 

 ( X

R

)

( I

0

0 Ψ

)

( X R

) 

−1

= σ

(

X

X + R

Ψ

R )

9 Maximum Likelihood Estimation (MLE, ^最尤法 )

−→ Review

1. The distribution function of { X

}

is f (x; θ ), where x = (x

, x

, · · · , x

) and θ = ( µ, Σ ).

Note that X is a vector of random variables and x is a vector of their realizations (i.e., observed data).

Likelihood function L( · ) is defined as L( θ ; x) = f (x; θ ).

Note that f (x; θ ) = ∏

i=1

f (x

; θ ) when X

, X

, · · · , X

are mutually indepen-

dently and identically distributed.

The maximum likelihood estimator (MLE) of θ is θ such that:

max

θ

L( θ ; X) . ⇐⇒ max

θ

log L( θ ; X) .

MLE satisfies the following two conditions:

(a) ∂ log L( θ ; X)

∂θ = 0.

(b) ∂

log L( θ ; X)

∂θ∂θ

is a negative definite matrix.

2. Fisher’s information matrix (フィッシャーの情報行列) is defined as:

I( θ ) = − E ( ∂

log L( θ ; X)

∂θ∂θ

) ,

where we have the following equality:

− E ( ∂

log L( θ ; X)

∂θ∂θ

) = E ( ∂ log L( θ ; X)

∂θ

∂ log L( θ ; X)

∂θ

) = V ( ∂ log L( θ ; X)

∂θ

)