Asymptotic Normality:

Asymptotic Normality:

Assumption 1：  θ ˆ

−→ θ , Assumption 2

√

T g( θ ; W

) −→ N(0 , S ).

Then, we have the following first-order approximation:

g( θ ; W

) ≈ g(ˆ θ

; W

) + ∂ g(ˆ θ

; W

)

∂θ

( θ − θ ˆ

)

= g(ˆ θ

; W

) + D ˆ

( θ − θ ˆ

) ,

where g( θ ; W

) is linearized around θ = θ ˆ

.

The first-order condition for the minimization problem is:

( ∂ g( θ ; W

)

∂θ

)

S

(

g( θ ; W

) )

= 0 .

Substituting the approximation into the above equation, we obtain the following:

D

S

(

g( θ ; W

) )

= D

S

(

g( θ ; W

) + D ˆ

( θ − θ ˆ

) )

= D

S

g(ˆ θ

; W

) + D

S

D ˆ

( θ − θ ˆ

) . Therefore,

√ T (ˆ θ

− θ ) ≈ (D

S

D ˆ

)

D

S

√ T (

g(ˆ θ

; W

) − g( θ ; W

) )

.

Thus, GMM estimator, ˆ θ

, has the following asymptotic distribution:

√ T (ˆ θ

− θ ) −→ N (

0 , (D

S

D)

) , where ˆ D

−→ D is utilized.

From Assumption 2, we have the following asymptotic distribution:

( √

T g( θ ; W

) )

S

( √

T g( θ ; W

) )

−→ χ

(r) .

When θ is replaced by GMM estimator ˆ θ

, we have the following dis- tribution:

( √

T g(ˆ θ

; W

) )

S ˆ

( √

T g(ˆ θ

; W

) )

−→ χ

(r − k) , which is called a test of the overidentifying restrictions.

= ⇒ J test by Hansen (1982)

k linear combinations consisting of a r × 1 vector g(ˆ θ

; W

) are zeros.

Therefore, the degrees of freedom are r − k.

Some Examples:

(a) OLS:

Regression Model: y

= x

β +

, E(x

) = 0 h( θ ; w

) is taken as:

h( θ ; w

) = x

(y

− x

β ) . (b) IV (Instrumental Variable,

):

Regression Model: y

= x

β +

, E(x

) , 0, E(z

) = 0

h( θ ; w

) is taken as:

h( θ ; w

) = z

(y

− x

β ) , where z

is a vector of instrumental variables.

(c) NLS (Nonlinear Least Squares,

):

Regression Model: f (y

, x

, β ) =

, E(x

) , 0, E(z

) = 0 h( θ ; w

) is taken as:

h( θ ; w

) = z

f (y

, x

, β )

where z

is a vector of instrumental variables.

5 Bayesian Estimation (

)

Greenberg, E. (2013) Introduction to Bayesian Econometrics (2nd ed.)

安藤知寛

(2010)

(

)

豊田秀樹編

(2008)

(朝倉書店)

Dey, D.K. and Rao, C.R., (2005) Handbook of Statistics, Vol.25: Bayesian Think- ing: Modeling and Computation

繁桝・岸野・大森監訳

(2011)

(

)

5.1 Introduction

Two Events: A and B Conditional Probability:

P(A | B) = P(A ∩ B)

P(B) = P(B | A)P(A) P(B) Posterior Distribution (事後分布): f

( θ| y):

f

( θ| y) = f

(y |θ ) f

( θ )

f

(y) = f

(y |θ ) f

( θ )

∫ f

(y |θ ) f

( θ )d θ ∝ f

(y |θ ) f

( θ ) ,

where f

( θ ) is called the prior distribution (事前分布).

Example 1: Let x be the number of successes in a series of n trials with proba- bility θ of success in each.

That is, x has the binomial probability function, given θ , f

(x |θ ) =

( n x )

θ

(1 − θ )

, x = 0 , 1 , · · · , n . θ is assumed to be the beta distribution:

f

( θ ) = 1

B(p , q) θ

(1 − θ )

,

for ≤ θ ≤ 1, which corresponds to a prior distribution.

Before applying Bayes’ theorem, f

(x) is given by:

f

(x) =

∫

f

(x |θ ) f

( θ )d θ

= ( n

r

) 1

B(p , q)

∫

θ

(1 − θ )

d θ

= ( n

r

) B(p + x , q + n − x) B(p , q) . The posterior distribution of θ is:

f

( θ| x) = 1

B(p + x , q + n − x) θ

(1 − θ )

,

which is also a beta distribution with prameters p + x and q + n − x.

The posterior mean and variance are:

E( θ| x) = p + x

p + q + n , V( θ| x) = (p + x)(q + n − x) (p + q + n)

(p + q + n + 1) . Example 2: x |θ ∼ N( θ, v), where v is known.

θ ∼ N(m , w), where m and w are known. = ⇒ prior dist.

Then, the posterior distribution of θ is:

θ| x ∼ N

( wx + vm w + v , vw

w + v

) .

Example 3: x

, x

, · · · , x

are mutually independently and identically distributed as N( µ, σ

), where µ and σ

are unknown.

f

(x |θ ) =

∏

(2 πσ

)

exp (

− 1

2 σ

(x

− µ )

)

= (2 πσ

)

exp (

− 1

2 σ

(s

+ n(x − µ )

) ) , where x = (1 / n) ∑

i=1

x

and s

= ∑

i=1

(x

− x)

. The prior density is:

f

( θ ) = k(a , b , w) σ

exp (

− 1 2 σ

( a + ( µ − m)

w

)) ,

where k(a , b , w) = a

2

( π w)

Γ (

b) is a constant.

The posterior density is:

f

( θ| x) = k(a

, b

, w

) σ

exp (

− 1 2 σ

( a

+ ( µ − m

)

w

)) ,

where w

= w

1 + nw , m

= m + nwx

1 + nw , b

= b + n, a

= a + s

+ n(x − m)

1 + nw . Inference on µ : The posterior density of µ is:

f ( µ| x) =

∫

0

f ( θ| x)d σ

= k

(t

, b

) (

1 + ( µ − m

)

b

t

)

,

where t

= w

a

b

and k

(t

, b

) = 1

√ t

k

B(

,

b

) .

Thus, µ − m

√ t

has a t distribution with b

degrees of freedom.

Inference of σ

: The posterior density of σ

is:

f ( σ

| x) =

∫

−∞

f ( θ| x)d µ = k

(a

, b

) σ

exp (

− a

2 σ

) ,

where k

(a

, b

) = (

a

)

Γ (

b

) . Thus, a

σ

is chi-squared with b

degrees of freedom.

5.2 Inference

Posterior Distribution (

): f

( θ| y)

5.2.1 Point Estimate Posterior Mean (事後平均):

θ =

∫

−∞

θ f

( θ| y)d θ.

Posterior Mode (

):

θ ˆ = argmax

f

( θ| y) .

Posterior Median (事後メディアン):

θ ˜ such that

∫

−∞

f

( θ| y)d θ = 0 . 5 . 5.2.2 Interval Estimate

∫

R

f

( θ| y)d θ = 1 − α,

where R is called confidence interval.

Bayesian confidence interval (ベイズ信頼区間) or credible interval (信用区間):

P( θ

< θ < θ

) = 1 − α.

θ

and θ

lead to lower and upper bounds.

( θ

, θ

) is called Bayesian confidence interval or credible interval.

Highest posterior density interval (

):

f

( θ

| y) ≥ f

( θ

| y) , for θ

∈ R and θ

< R .

5.2.3 Marginal Likelihood (周辺尤度) Marginal Likelihood = ⇒ Fitness of the Model:

f

(y) =

∫

f

(y |θ ) f

( θ )d θ,

which corresponds to the denominator in the posterior distribution.

5.3 Example: Linear Regression

Regression Model:

y = X β + u , u ∼ N(0 , σ

I

) ,

where y and u are n × 1 vectors, X is an n × k matrix and β is a k × 1 vector.

Likelihood Function: θ = ( β, σ

)

f

(y |θ ) = (2 πσ

)

exp (

− 1

2 σ

(y − X β )

(y − X β ) )

Prior Distributions:

f

( β, σ

) = f

( β|σ

) f

( σ

) , where

f

( β|σ

) = N( β

, σ

A

) = (2 πσ

)

| A |

exp (

− 1

2 σ

( β − β

)

A( β − β

) ) , f

( σ

) = IG ( ν

2 , λ

2

) = ( λ

/ 2)

Γ ( ν

/ 2) ( σ

)

exp (

− λ

2 σ

) .

β

, A, ν

and λ

are called the hyper-parameters.

Note that Y ∼ IG(a , b) for X ∼ G(a , b) and Y = 1

X .

The posterior distribution of β and σ

is:

f

( β, σ

| y) ∝ f

(y |β, σ

) f

( β|σ

) f

( σ

)

= (2 πσ

)

exp (

− 1

2 σ

(y − X β )

(y − X β ) )

× (2 πσ

)

| A |

exp (

− 1

2 σ

( β − β

)

A( β − β

) )

× ( λ

/ 2)

Γ ( ν

/ 2) ( σ

)

exp (

− λ

2 σ

)

∝ ( σ

)

exp (

− (y − X β )

(y − X β ) + ( β − β

)

A( β − β

) + λ

2 σ

)

∝ |σ

A ˆ |

exp (

− ( β − β ˆ )

A ˆ

( β − β ˆ ) 2 σ

) × ( σ

)

exp (

− λ ˆ 2 σ

)

∝ f

( β|σ

, y) × f

( σ

| y) = N( ˆ β, σ

A) ˆ × IG( ν ˆ 2 , λ ˆ

2 ) where

β ˆ = (X

X + A)

(X

X ˆ β

+ A β

) , β ˆ

= (X

X)

X

y , A ˆ = (X

X + A)

, ν ˆ = ν

+ n ,

λ ˆ = λ

+ (y − X ˆ β )

(y − X ˆ β ) + ( β

− β ˆ

)

((X

X)

+ A

)

( β

− β ˆ

) .

The marginal posterior distribution of β is:

f

( β| y) =

∫

f

( β, σ

| y)d σ

=

∫

f

( β|σ

, y) f

( σ

| y)d σ

∝ (

1 + 1

ν ˆ ( β − β ˆ )

( λ ˆ ν ˆ A ˆ )

( β − β ˆ )

)

,

which is a k-dimensional t distribution with parameters ˆ β , λ ˆ

ν ˆ A and ˆ ˆ ν .

Note that the k-dimensional t distribution with parameters µ , Σ and ν is given by:

f (x) = Γ

Γ (

)( νπ )

|Σ|

( 1 + 1

ν (x − µ )

Σ

(x − µ ) )

.

The marginal likelihood is:

f

(y) = f

(y |θ ) f

( θ )

f

( θ| y) = | A ˆ |

| A |

( λ

/ 2)

Γ (ˆ ν/ 2) π

Γ ( ν

/ 2)( ˆ λ/ 2)

, which is utilized for model selection.

In general, how do we evaluate f

( θ| y), E( θ| y), f

(y) and so on?