Reproducing Kernel Exponential Manifold:

Reproducing Kernel Exponential Manifold:

Estimation and Geometry

Kenji Fukumizu

Institute of Statistical Mathematics, ROIS Graduate University of Advanced Studies

Mathematical Explorations in Contemporary Statistics

Outline

„

Introduction

„

Reproducing kernel exponential manifold (RKEM)

„

Statistical asymptotic theory of singular models

„

Concluding remarks

Introduction

Maximal Exponential Manifold

„

Maximal exponential manifold (P&S‘95)

‰ A Banach manifold is defined so that the cumulant generating

function is well-defined on a neighborhood of each probability density.

‰ Orlicz space L_cosh-1(f)

This space is (perhaps) the most general to guarantee the finiteness of the cumulant generating functions around a point.

, )) (

exp(u u f

f_u = − Ψ_f Ψ_f (u) = logE_f [e^u] < ∞

{

}

= u | α 0 s.t. E_f [e^αu] and E_f [e^−αu]

Estimation with Data

„

Estimation with a finite sample

‰ A finite dimensional exponential family is suitable for the maximum likelihood estimation (MLE) with a finite sample.

MLE: θ that maximizes

‰ Is MLE extendable to the maximal exponential manifold?

But, the function value u(X_i) is not a continuous functional on u in the exponential manifold.

{ }

∑ ∑

= ⁿ

i

m

a a i

a n

n u X

n ₁ ¹ ( ) ( )

) 1 X

;

(θ θ θ

l

0μ

1, , X : f

X K _n i.i.d. ～

{ }

∑

Ψ

−

= ⁿ

i

f i

n

n u X u

u n

1

) ( )

1 ( ) X

; l (

(

)

n X , , X

X = ₁ K

Reproducing kernel exponential

manifold

Reproducing Kernel Hilbert Space

„

Reproducing kernel Hilbert space (RKHS)

‰ Ω: set. A Hilbert space H consisting of functions on Ω is called a reproducing kernel Hilbert space (RKHS) if the evaluation functional

is continuous for each

‰ A Hilbert space H consisting of functions on Ω is a RKHS if and only if there exists (reproducing kernel) such that

(by Riesz’s lemma)

∈H

⋅ , ) ( x k

) ( ),

,

( x f f x

k ⋅ _H = ∀f ∈H , x∈Ω.

) ( ,

: f f x

e_x H → R a

Ω.

∈ x

Reproducing Kernel Hilbert Space II

„

Positive definite kernel and RKHS

A symmetric kernel k: Ω x Ω Æ R is said to be positive definite, if for any and

Theorem (construction of RKHS)

If k: Ω x Ω Æ R is positive definite, there uniquely exists a RKHS H_k on Ω such that

(1) for all

(2) the linear hull of is dense in H_k , (3) is a reproducing kernel of H_k, i.e.,

Ω

n ∈ x x₁,_K,

, 0 ) ,

1 (

, ≥

∑

j

i cicjk xi xj

, ,

1, c_n ∈R c _K

}

| ) , (

{k ⋅ x x∈Ω Ω,

∈ H x

∈

⋅, ) ( x k

) , ( x k ⋅

) ( ),

,

( x f f x

k

k

=

⋅ _H ∀f ∈H _k, x∈Ω.

Reproducing Kernel Hilbert Space III

„

Some properties

‰ If the pos. def. kernel k is of C^r, so is every function in H_k.

‰ If the pos. def. kernel k is bounded, so is every function in H_k.

„

Examples:

„ Euclidean inner product

„ Gaussian RBF kernel

„ Polynomial kernel

y x y

x

k( , ) = ^T

d

T y c

x y

x

k( , ) = ( + ) (c ≥ 0, d ∈N)

(

)

exp )

,

(x y x y σ

k = − −

H = {polyn. deg ^≦d}

dim _Hk = ^∞

m

k ≅ R

H

Exponential Manifold by RKHS

„

Definitions

Ω: topological space. μ: Borel probability measure on Ω s.t. suppμ = Ω.

k : continuous pos. def. kernel on Ω such that H_k contains 1 (constants).

Note: If || u || < δ,

‰ Tangent space

{

= : : , ( ) 0 ( ), 1,

: )

( μ

μ k f | f f x x fd

M R continuous

}

∞

<

>

∃^{δ 0,} ∫^{eδ k(x,x) f (x)dμ(x)}

{

}

:= u∈ E u X =

T_f H _{k f} closed subspace of H_k

M_μ(k) is provided with a Hilbert manifold structure.

. ]

[ ]

[ ]

[ ^u(X) = _f ^u,k(⋅,X) ≤ _f ^{||u|| k(X,X)} < ∞

f e E e E e

E

Exponential Manifold by RKHS II

„

Local coordinate

For

Then, for any

Define

Lemma

(1) W_f is an open subset of T_f.

{ }

X X k X

u f f

f u T E e T

W := ∈ | ∃δ > 0, [ ^{( )+δ ( , )}]< ∞ ⊂ ),

(k M f ∈ _μ

Wf

u∈

).

( ))

( exp(

: u u f M k

f_u = − Ψ_f ∈ _μ

u f

f ^:W Mμ⁽k^), u a f

ξ → (one-to-one)

, 1

: _f → _{f f} = _f ⁻

f S W ϕ ξ

ϕ

) (

: _{f f}

f = ξ W

E

Æ works as a local coordinate

. g^∈E ^⇔ E ⁼ E

„

Reproducing Kernel Exponential Manifold (RKEM)

Theorem

Exponential Manifold by RKHS III

. The system is a -atlas of M_μ(k).

‰ A structure of Hilbert manifold is defined on M_μ(k) with Riemannian metric E_f[uv].

‰ Likelihood functional is continuous.

‰ The function u(x) is decoupled in the inner product

u: natural coordinate, : sufficient statistics

‰ The manifold depends on the choice of k.

e.g. Ω μ . Æ ≦

{

}

ϕ _∈ μ

E _C^∞

⎥⎦

⎢ ⎤

⎣

⎡ −Ψ

Ψ −

= −

−

g

f u E u

g

f u

u u ^f _g ^f

f g

)) ( logexp(

)) ( logexp(

)

1( ϕ ϕ _o

⎥⎦⎤

⎢⎣⎡ +

− +

= g

u f g E

u log f _g log coordinate

transform

) , ( ,k x u ⋅ )

, ( x k ⋅

„

Mean parameter

‰ For any there uniquely exists such that

‰ The mean parameter does not necessarily give a coordinate, as in the case of the maximal exponential manifold.

„

Empirical mean parameter

‰ X₁, …, X_n: i.i.d. sample ~ fμ. ),

(k M

f ∈ _μ m_f H _k

Mean parameter of RKEM

∈

f k

f u X u m

E [ ( )]= , H for all u^∈H _k.

∑

⋅

= ⁿ

i

i

n k X

m n

1

) ,

1 ( : ˆ

Fact 1.

Empirical mean parameter:

) (

) 1 (

, ˆ

n

f X

f f

m = ∑ ∀ ∈H

Applications of RKEM

‰ Maximum likelihood estimation (IGAIA2005)

„ Maximum likelihood estimation with regularization is possible.

„ The consistency of the estimator is proved.

‰ Statistical asymptotic theory of singular models

„ There are examples of statistical model which is a submodel of an infinite dimensional exponential family, but not

embeddable into a finite dimensional exponential family.

„ For a submodel of RKEM, developing asymptotic theory of the maximum likelihood estimator is easy.

‰ Geometry of RKEM

„ Dual connections can be introduced on the tangent bundle in some cases.

) (±1

Statistical asymptotic theory of

singular models

„

Standard asymptotic theory

Statistical model on a measure space (Ω,B,μ).

Θ: (finite dimensional) manifold.

“True” density: f₀(x) = f(x ;θ₀)

Maximum likelihood estimator (MLE)

Under some regularity conditions,

Likelihood ratio

Singular Submodel of exponential family

0μ

1, , X : f

X K _n i.i.d. ～ }

| )

; (

{f x θ θ ∈Θ

∑= Θ

∈

= ⁿ

i

i

n f X

1

)

; ( log max

ˆ arg θ

θ θ

) (θ₀ ∈Θ

) ) ( (0, ˆ )

(θ −θ₀ ⇒ N I θ₀ ⁻¹

n _n in law (n → ∞) f₀ MLE

Asymptotically normal

) 2

; ˆ log (

2 ˆ )

(

2 _d

n

n i n

n

X

f χ

θ

θ =

∑

l

Singular Submodel of exponential family II

„

Singular submodel in ordinary exponential family

Finite dimensional exponential family M : Submodel

Tangent cone:

Under some regularity conditions, }

| )

; (

{ f x M _S

S = θ ∈ θ ∈Θ

f₀

S M

S C_f

0

)) ( )

( exp(

)

;

(x θ = θ u x − Ψ θ

f ^T

) (θ ∈Θ

)}

( )

( s.t.

0 ,

} {

| )

(

{ ₀

0

0S = u x ∈T M ∃ ⊂ Θ ∃ > − → n →∞

C_f ξ^T _f θ_{n S} λ_n λ_n θ_n θ ξ

∑

= ⁿ

i i

n i n

n f X

X f

1 ( ; 0)

ˆ )

; log (

ˆ )

( θ

θ θ l

( )

{

}

2 sup

1 ²

1 1

1

|

|

, ²

0 0

p n

i i

n T u

E S C u

o X

u

T f f

T

+

=

∑

=

∈

ξ

ξ

ξ projection of empirical (n → ∞)

Singular submodel in RKEM

„

Submodel of an infinite dimensional exponential family

‰ There are some models, which are not embeddable into a finite

dimensional exponential family, but can be embedded into an infinite dimensional RKEM.

„

Example:

Mixture of Beta distributions (on [0,1])

‰ Singularity at

), 1 , 1

; ( ) 1

( ) 1 ,

; ( )

,

;

(x B x B x

f α β =α β + −α

1 1

) ( ) (

)

( (1 )

) ,

;

(x β γ = _Γ^Γ_β^β_Γ^+γ_γ x^β− − x ^γ− B

0 1

0 0.5 1 1.5 2 2.5 3

B(x;1,1)

B(x;3,1)

B(x;3,2) where

) 1 , 1

; ( )

, 0

; ( )

0(x f x B x

f = β =

Singular submodel in RKEM II

‰ H_k = Sobolev space H¹(0,1)

‰ Submodel of E_f0

‰ Tangent cone at f₀ is not finite dimensional.

⋅ α β

} 2 / 3 ,

1 0

| )) (

exp(

) ,

; (

{ ⋅ = _, − Ψ _{, 0} ≤ < >

= f α β u_{α β} u_{α β} f α β

S _f

)]

,

; ( [log )

,

; ( log :

)

( 0

,_β α β α β

α x f x E f x

u = − _f

S is a submodel of E_f0, and f₀ is a singularity of S.

|),

| exp(

) ,

(x y x y

k = − − ⁼

(

)

∫

(

)

2 2

2 2

2 | '( )| | ( )|

2 ) 1

1 ( )

0 2 (

|| 1

||u u u u x u x dx

Hk

) 1 , 0 ( )

,

; (

log f x α β ∈H¹ for 0 ≤α <1, β > 3/2. Fact:

„

General theory of singular submodel

M_μ(k): RKEM.

Submodel defined by ),

(k M

f _μ

Singular submodel in RKEM III

∈ Ef

S ⊂ ϕ : K ×[0,1]→T_f

f

Singularity such that

(1) K: compact set (2)

(3) ϕ(a,t): Frechet differentiable w.r.t. t and

(4)

0 0

) ,

(a t = ⇔ t = ϕ

) , (a t

∂t

∂ϕ

is continuous on K ×[0,1] 0

) , (

min 0 >

∂ =

∂

∈ t t

K

a ^ϕ a t

S E_f

Singular submodel in RKEM IV

Lemma (tangent cone)

⎭⎬

⎫

⎩⎨

⎧ ∈

= _{≥ =}

∂

∂ a t a K

S

C_{f t}

t ( , )| ⁰ R ϕ

Theorem

∑=

∈ n

i i

i S

g f X

X g

1 ( )

) log (

sup sup , ˆ (1)

2

1 ²

1

|

|

, ²

p n

w E S C w

o m

w

f f

+

=

=

∈

) (n →∞

• Analogue to the asymptotic theory on submodel in a finite dimensional exponential family.

projection of empirical mean parameter

2 1

|

|

, ²

2 sup 1

w w

E S C w

G

f

f =

∈

⇒

in law

G_w: Gaussian process

Summary

‰ Exponential Hilbert manifolds, which can be infinite dimensional, is defined using reproducing kernel Hilbert spaces.

‰ From the estimation viewpoint, an interesting class is submodels of infinite dimensional exponential manifolds, which are not

embeddable into a finite dimensional exponential family.

‰ The asymptotic behavior of MLE is analyzed for singular submodels of infinite dimensional exponential manifolds.