Likelihood Ratio of UnidentiÞable Models and Multilayer Neural Networks

Likelihood Ratio of UnidentiÞable Models and Multilayer Neural Networks

Kenji Fukumizu

Institute of Statistical Mathematics

4-6-7 Minami-Azabu, Minato-ku, Tokyo 106-8569, Japan E-mail: [email protected]

February 19, 2001

Abstract

This paper discusses the behavior of the maximum likelihood esti- mator, when the true parameter cannot be identiÞed uniquely. Among many statistical models with unidentiÞability, neural network models are the main concern of this paper. The set of unidentiÞable true pa- rameters is formulated as a conic singularity of the model, which is embedded in an inÞnite dimensional space of probability density func- tions. It has been known in some models with unidentiÞability the asymptotics of the likelihood ratio of MLE has an unusually larger or- der. Following Hartigan’s idea, the likelihood ratio of MLE is described by the supremum of an empirical process over a set of functions, and a useful sufficient condition of such larger orders is derived. This result is applied to neural network models, and a larger order is observed if the true function is realized by a network with a smaller number of hidden units than the model. A stronger lower bound of the order of likelihood ratio is also derived on condition that there are at least two redundant hidden units to realize the true function.

1 Introduction

This paper discusses the asymptotic behavior of the maximum likelihood estimator (MLE) under the condition that the true parameter is uniden- tiÞable. The asymptotics of MLE is an important problem in statistical estimation theory, and the asymptotic normality under some regularization conditions is well known ([1]). However, if the dimensionality of the set of true parameters is larger than zero, the Fisher information matrix at a true

1

parameter is singular, and the asymptotic normality is no longer satisÞed.

The behavior of MLE in such unidentiÞable situations has not been clariÞed completely.

There are many statistical models that have unidentiÞability. Finite mix- ture models, ARMA, reduced rank regression, and change point problems are typical examples of such models. Because the asymptotics of the MLE is not simple, model selection needs special consideration on such models. It is known that feed-forward neural networks have also the problem of uniden- tiÞability. The true parameter of a feed-forward neural network model is unidentiÞable, if the true function is realized by a network with smaller number of hidden units than the model. In this paper, we mainly discuss the neural network model in investigating the behavior of MLE closely.

We formulate the problem of unidentiÞability as a conic singularity ([2]) in the set of a statistical model, which is embedded in the space of all the probability density functions. In this formulation, the likelihood ratio of the MLE, with the true probability at the singularity, can be well described by the supremum of an empirical process over the unit vectors in the tangent cone. This empirical process shows very different behavior depending on the functional property of the tangent cone, while each marginal variable converge to a Gaussian distribution.

One of the interesting features is the order of the likelihood ratio of MLE, as the sample-size n goes to inÞnity. A model satisfying the regularity con- dition of the usual asymptotic theory has the likelihood ratio of the order O

(1). However, larger orders have been reported in some unidentiÞable models. Hartigan ([3]) discusses the normal mixture models with two com- ponents, and shows the likelihood ratio test statistics, under the hypothesis of one component, has a larger order than O

(1). In neural networks, the lower bound O

(log n) has been derived in unidentiÞable cases ([4]). In this paper, a useful sufficient condition of such larger orders than O

(1) will be given in the term of functional properties of the tangent cone. This result covers many models of a larger order of the likelihood ratio. Furthermore, a stronger lower bound of the order for some neural network models will be derived, by the analysis of the functional properties of the tangent cone.

2 UnidentiÞability and Locally Conic Models

2.1 Preliminaries

Let ( Z , B , µ) be a measure space, and S be a set of probability density

functions on ( Z , B , µ). The set S is called a statistical model if there is

a differentiable manifold (with boundary) Θ such that S is given by S = { f (z; θ) | θ ∈ Θ } . We call Θ as the parameter space. We assume through- out this paper that Suppf (z; θ) is invariant for all θ ∈ Θ, and f (z; θ) is differentiable on θ for each z ∈ Z .

Suppose that the probability distribution of i.i.d. random variables Z

, Z

, . . . , Z

is f

(z)µ with the probability density function f

(z), which has the same support as the model S. The function f

is called the true probability density. Given the random variables, the likelihood ratio of the model S with respect to { Z

}

is deÞned by

L

(θ) = X

log f(Z

; θ)

f

(Z

) . (1)

We consider the maximum likelihood estimator (MLE) ˆ θ that attains the maximum of the likelihood ratio, if it exists. From the deÞnition, we have

L

(ˆ θ) = sup

θ∈Θ

L

(θ) = sup

θ∈Θ

X

log f (Z

; θ)

f

(Z

) . (2) The main topic of this paper is the behavior of the likelihood ratio of MLE under the asymptotic assumption, where the number of samples goes to inÞnity.

2.2 UnidentiÞability of the true parameter

Throughout this paper, the true probability density f

(z) is assumed to be included in the model { f (z; θ) | θ ∈ Θ } . Then, there exists θ

∈ Θ such that f (z; θ

) = f

(z). We do not assume the uniqueness of θ

, and denote the set of true parameters by Θ

; that is, Θ

= { θ ∈ Θ | f(z; θ

)µ = f

(z)µ } . Unless Θ

is a single point, the usual view of asymptotic convergence to a single true parameter does not hold.

We say that the true parameter is unidentiÞable, if the set of true param- eters Θ

is a union of Þnitely many submanifolds of Θ, and the dimension of at least one of the submanifolds is larger than zero. There are many important statistical models in which the true parameter can be unidenti- Þable. One of the most famous examples is a Þnite mixture model. Let g(z; a) be a probability density function on Z with a variable parameter a, and f(z; a

, a

, b) be a mixture model deÞned by

f (z; a

, a

, b) = b g(z; a

) + (1 − b) g(z; a

), (3)

3

where b ∈ [0, 1]. Suppose that the true density f

(z) is given by g(z; a

) for some a

. Then, the set of parameters to give f

(z) contains { (a

, a

, b) | a

= a

= a

, b : arbitrary } ∪ { (a

, a

, b) | b = 0, a

= a

, a

: arbitrary } ∪ { (a

, a

, b) | b = 1, a

= a

, a

: arbitrary } , which is high dimensional.

The reduced rank problems ([5]), ARMA model ([6]), and the change point problem ([7]) are other examples of models with unidentiÞability. Feed- forward neural network models, such as multilayer perceptrons ([8]), are also among such models. We will mainly discuss the multilayer perceptron model in this paper.

Our main concern is to investigate how the likelihood ratio of MLE behaves on condition that the true parameter is unidentiÞable. If the true parameter is identiÞable, under some regularity conditions, the asymptotic distribution of the likelihood ratio of MLE converges in law to the chi-square distribution of freedom d. On the other hand, in unidentiÞable cases, even the order of the likelihood ratio of MLE can be different from O

(1), as shown later.

2.3 Locally conic model

In the previous subsection, the unidentiÞability was deÞned in terms of the parameters. However, if the space of probability density functions is considered, the set of true parameters corresponds to a single point in the space. The point is a singularity in the set of density functions deÞned by the model, if the dimensionality shrinks only at the point. The property of the set of density functions around the singularity will be better understood, if more convenient parameterization can be introduced than the original one.

Following Dacunha-Castelle & Gassiat ([2]), with some modiÞcation, a conic singularity is utilized for describing the unidentiÞability.

Let A

be a (d − 1)-dimensional differentiable manifold (with boundary), Θ an open set in A

× R , and S = { f (z; θ) | θ ∈ Θ } be a statistical model.

The parameter θ ∈ Θ is decomposed as θ = (α, β) for α ∈ A

and β ∈ R . Let a function f

(z) be an element in S. The statistical model S is called locally conic at f

if the following conditions are satisÞed;

1. f(z; (α, β )) is differentiable on β for each α ∈ A

and f

µ-almost every z.

2. Let Θ

and Θ(α) be subsets deÞned by Θ

= Θ ∩ (A

× { 0 } ) and

Θ(α) = Θ ∩ ( { α } × R ) for α ∈ A

, respectively. Then, Θ = [

α∈A0

Θ(α). (4)

3. The set of the parameters to give f

is Θ

; that is,

f(z; (α, β ))µ = f

(z)µ ⇐⇒ β = 0. (5) 4. For all α ∈ A

,

° °

° °

∂ log f (z; α, 0)

∂β

° °

° °

0µ)

= 1. (6)

If the dimension of A

is larger than zero, the parameter giving f

is not identiÞable. Intuitively, a locally conic model S is a d-dimensional set with a singularity at f

in the space of probability density functions. For each α ∈ A

, the submodel S

= { f (z; θ) | θ ∈ Θ(α) } is a one-dimensional, identiÞable statistical model. The score function of S

at the origin,

v

(z) = ∂ log f (z; (α, 0))

∂β , (7)

can be looked as a unit tangent vector in the direction of S

(see Þg.1). The family of score functions C = { v

| α ∈ A

} generates the tangent cone at the singularity f

. We call the set C the basis of the tangent cone, which has a key importance in the following discussion.

The view of tangent vectors can be rigorously formulated if S is included in a maximal exponential model ([9]), which is an inÞnite dimensional Ba- nach manifold. In the deÞnition, we only require that the functions in C are in L

(f

µ). They are not necessarily tangent vectors of the Banach manifold in the sense of Pistone and Sempi ([9]).

2.4 Neural network as a locally conic model

A feed-forward neural network model is an example of a locally conic model.

This paper mainly discusses multilayer perceptrons ([8]). The multilayer perceptron model with H hidden units is deÞned by a family of functions

ϕ(x; θ) = X

b

s(a

x + c

) + d, (8)

5

Figure 1: Locally conic model

where x ∈ X = R , s(t) = tanh(t), and θ = (a

, . . . , a

, b

, . . . , b

, c

, . . . , c

, d)

∈ Θ

= R

. Only models with one-dimensional input and output is dis- cussed for simplicity.

Learning in neural networks can be regarded as statistical estimation.

Assume that the distribution of an input sample X

is a probability Q on X = R . When the multilayer perceptron model is discussed, it is always assumed that Q is absolutely continuous with respect to the Lebesgue mea- sure on R , which is written by µ

, with the density function q(x), and that the integral E

| log q(x) | is Þnite. Let Y be a subset of R , and ( Y , B

, µ

) be a measure space. Let r(y | u) be a conditional probability density function of y ∈ Y given u ∈ R . This is used for a noise model. Throughout this paper, we put the following assumptions;

[Conditions on noise model (NM1)]

1. The conditional density r(y | u) is of class C

on u for all y ∈ Y . 2. For different u

and u

, we have r(y | u

)µ

6 = r(y | u

)µ

.

3. The Fisher information G(u) of r(y | u), deÞned by G(u) = Z ³ ∂ log r(y | u)

∂u

´

r(y | u)dµ

, (9)

is positive, Þnite, and continuous for all u ∈ R .

4. For all u ∈ R

lim

E

h

sup

|u⁰−u|≤ρ

¯ ¯

¯ ∂ log r(y | u

)

∂u

¯ ¯

¯ i

< ∞ . (10)

The condition 4 assures the famous relation E

[

] = − G(u) by Lebesgue’s dominated convergence theorem.

Given the function ϕ(x; θ), the statistical model of multilayer perceptron is deÞned by

f (z; θ) = r(y | ϕ(x; θ))q(x), (11) where z = (x, y) ∈ Z = X × Y , with respect to the measure µ

× µ

.

Popular choices of r(y | u) are the additive Gaussian noise model r(y | u) = 1

√ 2πσ exp ©

− 1

2σ

(y − u)

ª

(12) for continuous y, and the binomial distribution model

r(y | u) = e

1 + e

(13)

for binary output y ∈ Y = { 0, 1 } , which often appears in classiÞcation problems.

The true parameter can be unidentiÞable in the multilayer perceptron model. It can be seen in the simplest case as follows. Suppose we have the multilayer perceptron model with 2 hidden units, and the true function ϕ

(x) is given by a perceptron with only one hidden unit. If ϕ

(x) = b

tanh(a

x), then for any parameter θ in the set { θ ∈ Θ

| a

= a

, b

= b

, c

= 0, b

= 0, d = 0, a

, c

: arbitrary } ∪ { θ ∈ Θ

| a

= a

, b

= b

, c

= 0, a

= 0, b

tanh(c

) + d = 0 } the function ϕ(x; θ) equals to the true function

. We can see that the set of true parameters is a high dimensional subset in the parameter space. It is known that the true parameter is unidentiÞable if and only if the true function can be realized by a network with smaller number of hidden units than the model ([10],[11],[12]).

This unidentiÞability of multilayer perceptrons can be formulated as a locally conic model. Suppose we have the multilayer perceptrons with H

1These two subsets do not give all the parameters to realizeϕ0(x). The whole set of the true parameters is shown in [12].

7

hidden units. Let K be an integer such that 0 ≤ K < H, and ϕ

(x) be a function realizable by a multilayer perceptron with K hidden units.

A slightly restricted parameter space Θ

is deÞned by Θ

= { θ = (a

, . . . , a

, b

, . . . , b

, c

, . . . , c

, d) ∈ Θ

| a

6 = 0, b

6 = 0 (1 ≤ j ≤ H), (a

, c

) 6 = ± (a

, c

) (1 ≤ j < h ≤ H) } . Note that in Θ

the parameters that correspond to the functions realizable by a smaller-sized network are eliminated (see [10]). For a parameter in Θ

, it is known ([13]) that the functions { 1, s(a

x + c

), s

(a

x + c

)x, s

(a

x + c

) | 1 ≤ j ≤ H } are linearly independent.

Given a function

ϕ

(x) = X

k=1

b

s(a

x + c

) + d

(14) for θ

= (a

, . . . , a

, b

, . . . , b

, c

, . . . , c

, d

) ∈ Θ

, the parameter space is again restricted slightly to Θ

by Θ

= { θ ∈ Θ

| (a

, c

) 6 = ± (a

, c

) (1 ≤ k ≤ K, K + 1 ≤ j ≤ H) } . This reduction does not matter in discussing the maximum likelihood estimation, because MLE lies in Θ

with probability one. Introduce a new parameterization by

β = sgn(b

) q

b

+ · · · + b

, ξ

= a

− a

β , (1 ≤ k ≤ K ), ξ

= a

, (K + 1 ≤ j ≤ H), η

= b

− b

β , (1 ≤ k ≤ K), η

= b

β , (K + 1 ≤ j ≤ H), ζ

= c

− c

β , (1 ≤ k ≤ K), ζ

= c

, (K + 1 ≤ j ≤ H), δ = d − d

β . (15)

for θ ∈ Θ

, and deÞne new parameter spaces Π

and Π

by Π

= { ω = (ξ

, . . . , ξ

, η

, . . . , η

, ζ

, . . . , ζ

, δ, β) |

a

+ βξ

6 = 0 (1 ≤ k ≤ K), ξ

6 = 0 (K + 1 ≤ j ≤ H),

(a

+ βξ

, c

+ βζ

) 6 = ± (a

+ βξ

, c

+ βζ

) (1 ≤ k < h ≤ K), (a

+ βξ

, c

+ βζ

) 6 = ± (ξ

, ζ

) (1 ≤ k ≤ K, K + 1 ≤ j ≤ H), (ξ

, ζ

) 6 = ± (ξ

, ζ

) (K + 1 ≤ j < i ≤ H),

(ξ

, ζ

) 6 = ± (a

, c

) (1 ≤ k ≤ K, K + 1 ≤ j ≤ H), b

+ βη

6 = 0 (1 ≤ k ≤ K),

X

η

= 1, η

6 = 0 (K + 1 ≤ j ≤ H),

η

> 0, β ∈ R} (16)

and Π

= { ω ∈ Π

| β 6 = 0 } , respectively. The multilayer perceptron can be rewritten using this parameterization:

ψ(x; ω) = X

k=1

(b

+ βη

) s ¡

(a

+ βξ

)x + (c

+ βζ

) ¢

+ X

βη

s(ξ

x + ζ

) + βδ. (17) It is easy to see that the Π

and Θ

are diffeomorphic by the transform (15), and ϕ(x; θ) = ψ(x; ω) holds for the corresponding θ ∈ Θ

and ω ∈ Π

. Thus, it suffice to consider { ψ(x; ω) | ω ∈ Π

} , when the maximum likelihood estimation is discussed.

Let S

= { f(x, y; ω) | ω ∈ Π

} be a statistical model deÞned by f (x, y; ω) = r(y | ψ(x; ω))q(x). (18) The model S

consists of probability density functions corresponding to ϕ

(x) and the functions given by ϕ(x; θ) for θ ∈ Θ

. The function f

(x, y) be a density function deÞned by ϕ

(x), that is, f

(x, y) = r(y | ϕ

(x))q(x).

The model S

is a locally conic model, if α summarizes (ξ

, . . . , ζ

, δ) and ω = (α, β).

Theorem 1. Let S

be the statistical model of multilayer perceptrons with H hidden units deÞned by eqs. (17) and (18), and f

be a density function given by (14). Then, under the assumption [NM1], S

is locally conic at f

.

9

Proof. Let A

be a set given by A

= { α | (α, 0) } , and Π

(α) by Π

(α) = { (α, β) | β ∈ R} for α ∈ A

. We can see Π

= ∪

Π

(α), because for all (α, β) ∈ Π

, the point (α, 0) is also contained in Π

by the fact θ

∈ Θ

and (ξ

, ζ

) 6 = ± (a

, c

) for K +1 ≤ j ≤ H. We can also prove that ψ(x; ω) = ϕ

(x) for all x if and only if ω ∈ Π

. The sufficiency is trivial.

For the necessity, because s(ξ

x + ζ

) (K + 1 ≤ j ≤ H) is not contained in the linear hull of the functions { 1, s(a

x + c

), s(ξ

x + ζ

), s((a

+ βξ

)x + (c

+ βζ

)) | 1 ≤ k ≤ K, K + 1 ≤ i ≤ H, i 6 = j } by the deÞnition of Π

, the coefficients of s(ξ

x + ζ

) in eq.(17) must be zero to realize ψ(x; ω) = ϕ

(x).

This implies β = 0. Thus, the model S

satisÞes the conditions 1, 2, and 3 in the deÞnition of a locally conic model.

For the condition 4, let N (α) be the L

(f

(x, y)µ

µ

)-norm of a tangent vector

log f (x, y; (α, 0)). This is essentially determined by the partial derivative:

∂ψ(x; (α, 0))

∂β =

X

j=K+1

η

s(ξ

x + ζ

) + δ

+ X

k=1

η

s(a

x + c

) + X

k=1

b

ξ

s

(a

x + c

)x + X

k=1

b

ζ

s

(a

x + c

). (19) The L

norm is calculated as

N (α)

= Z Z

r(y | ϕ

(x))q(x) n ∂r(y | ϕ

(x))

∂u

∂ψ(x; (α, 0))

∂β

o

dxdµ

= Z

G(ϕ

(x))

n ∂ψ(x; (α, 0))

∂β

o

q(x)dx. (20)

Since ϕ

(x) is bounded, so is G(ϕ

(x)) by the continuity of G(u). From eq.(19), the function ©

∂β

ψ(x; (α, 0)) ª

is also bounded. Thus, N (α) is Þnite.

Because the functions 1, s(ξ

x+ζ

), s(a

x+c

), s

(a

x+c

)x, and s

(a

x+c

) (1 ≤ k ≤ K, K + 1 ≤ j ≤ H) are linearly independent (see [13]), the partial derivative

ψ(x; (α, 0)) is not constant zero. Hence, the zero points of

∂

∂β

ψ(x; (α, 0)) has no accumulation points, and the probability of the set by

Q is zero. Therefore, 0 < N (α) < ∞ for all α ∈ A

. Using N (α)β instead

of β, we have the normalized tangent vectors at f

(x, y).

3 Maximum likelihood estimation in locally conic models

3.1 MLE and supremum of a random process

Let S = { f (z; (α, β )) | (α, β) ∈ Θ } be a statistical model, which is locally conic at f

∈ S. Suppose Z

, Z

, . . . , Z

are i.i.d. random variables with the law f

µ. For each α ∈ A

, the submodel S

= { f (z; (α, β)) | β ∈ Θ(α) } is a smooth, one-dimensional model with a variable parameter β. If the maximum likelihood estimator ˆ β

in S

exists for each α ∈ A

, the likelihood ratio of the MLE in S is given by

sup

θ∈Θ

L

(θ) = sup

α

L

(α, β ˆ

). (21) Assume that each submodel S

satisÞes some regularity conditions of the asymptotic normality. A set of conditions, which is essentially from Wald ([16]) and Cram´ er ([1]), is given as follows

. For simplicity, we write each submodel by { g(z; β) | β ∈ V } , neglecting the index α. The parameter set V is an open set in R , and we write a

= inf { β | β ∈ V } ∈ R ∪ {−∞} and b

= sup { β | β ∈ V } ∈ R ∪ {∞} .

[Conditions on asymptotic normality (AN)]

1. For any β ∈ V , the integral E

[ | log g(z; β) | ] is Þnite.

2. Let H

(z; t) and H

(z; s) be functions deÞned by H

(z; t) = sup

β≥t

log g(z; β) and H

(z; s) = sup

β≤s

log g(z; β), (22) respectively. Then,

lim

E

[H

(z; t)] < ∞ and lim

s↓a0

E

[H

(z; s)] < ∞ . (23) 3. There exist ∆

and ∆

such that R

∆_±

f

(z)dµ > 0 and lim

H

(z; t) = −∞ for all z ∈ ∆

, (24)

s

lim

H

(z; s) = −∞ for all z ∈ ∆

. (25)

2Another set of conditions is found in van der Vaart ([14], Section 5.3), which is more reÞned than the famous ones by Cram´er ([1]).

11

4. For all β ∈ V , lim

ρ↓0

E

£ sup

|β⁰−β|≤ρ

log g(z; β

) ¤

< ∞ . (26)

5. The density g(z; β) is three-times differentiable on β for all z, and lim

E

h

sup

|β|≤ρ

¯ ¯

¯ ∂

log g(z; β)

∂β

¯ ¯

¯ i

< ∞ . (27)

The conditions 1—4 are slight modiÞcation of Wald’s regularity conditions for the consistency of MLE β b

([16]). The condition 5 assures asymptotic efficiency of β b

under the consistency assumption. If each submodel S

satisÞes the conditions [AN], the standard argument using Taylor expansion leads to

L

(α, β ˆ

) = 1

2 U

(α)

+ o

(1), (28) where U

(α) is a random variable deÞned by

U

(α) =

X

v

(Z

), (29)

and v

(z) is a function in the basis of the tangent cone C, deÞned by v

(z) = ∂

∂β log f (z; (α, 0)). (30)

The variable U

(α) converges in law to the standard normal distribution for each α ∈ A

. If we consider the behavior of U

(α) over all α, it can be looked as an empirical process over α or C, and every marginal distribution on Þnite points converges to a multidimensional normal distribution. The likelihood ratio of MLE is given by

sup

θ∈Θ

L

(θ) = sup

α∈A0

½ 1

2 U

(α)

+ o

(1)

¾

. (31)

Dacunha-Castelle and Gassiat ([2]) discuss the convergence of U

, as- suming the uniform convergence in the asymptotic normality and the em- pirical process. More precisely, if the higher order term of o

(1) in eq.(31) is bounded uniformly over α, the term can be eliminated from the supremum;

sup

θ∈Θ

L

(θ) = sup

α

½ 1

2 U

(α)

¾

+ o

(1). (32)

Furthermore, if the stochastic process U

converges ”nicely” to a Gaussian process W over C, the limit of the supremum of | U

| can be replaced by the the supremum of | W | (see Wellner & van der Vaart ([15]) and van der Vaart ([14]) for the detail). Then, we obtain

sup

θ∈Θ

L

(θ) = sup

α

1

2 W

+ o

(1). (33)

Dacunha-Castelle & Gassiat propose a likelihood ratio test based on the supremum of the Gaussian process W .

Unlike Dacunha-Castelle & Gassiat ([2]), when discussing the stochastic process U

in eq.(28), this paper will investigate non-uniform cases, in which the simpliÞcation in eqs. (32) and (33) does not hold. In non-uniform cases, the behavior of MLE is complex, and even the order of the likelihood ratio can be different from the usual O

(1), as I mentioned in Section 1.

3.2 Slower convergence in non-uniform cases

The likelihood ratio of MLE can have a larger order than O

(1), if the function class of the tangent cone is ”rich” enough, as the cone in the normal mixture and multilayer perceptrons.

In this subsection, a useful sufficient condition of such an unusually larger order is derived, as an extension of Hartigan’s idea ([3]). Note that the marginal distribution of U

on Þnite points v

, . . . , v

in C always converges to a multi-dimensional normal distribution with the covariance E

[v

v

].

Thus, two components of the limit are independent on condition that their covariance is zero. Suppose we can Þnd an arbitrary number of ”almost”

uncorrelated random variables in C. Then, the supremum of U

(α) on such variables can take an arbitrary large value, since the maximum of m in- dependent samples from the standard normal distribution is approximately

√ 2 log m for large m. Hartigan ([3]) applied this idea to a normal mix- ture model with two components, calculating the covariance explicitly. An extension of this idea leads us to the following theorem;

Theorem 2. Let a statistical model S = { f (z; (α, β)) } be locally conic at f

∈ S, and C = { v

(z) =

f (z; (α, 0)) } be the basis of the tangent cone.

Assume that for each α ∈ A

the submodel { f(z; α, β) | β } satisÞes the conditions of asymptotic normality [AN]. If there exists a sequence { v

}

in C such that v

→ 0 in probability, then, for arbitrary M > 0, we have

n

lim

Prob

³ sup

(α,β)

L

(α, β) ≤ M

´

= 0. (34)

13

Proof. From Proposition 1 below, for arbitrary ε > 0 and K ∈ N , there exist v(α

), . . . , v(α

) ∈ C such that | E[v(α

)v(α

)] | < ε for different i and j. The rest of the proof is accomplished in the same way as Hartigan ([3]), which will be shown below.

Let W = (W

, . . . , W

) be a random vector following the limiting nor- mal distribution of (U

(v

), . . . , U

(v

)), and Σ be the variance-covariance matrix of W . Because the absolute value of every off-diagonal element in Σ is less than ε, by Gerÿ sgorin’s inequality ([17]), we have (1 + (K − 1)ε)I

≤ Σ ≤ (1 − (K + 1)ε)I

. Then, for arbitrary M > 0, the inequality

P ¡ max

1≤i≤K

| W

| ≤ M ¢

≤ Z

[−M,M]^K

√

(2π)^K|Σ|

e

dW

≤

Z

[−M,M]^K 1

(2π)^K/2

e

du

≤ ³

1+(K−1)ε 1−(K−1)ε

´

{ Φ(M ) − Φ( − M ) }

(35) holds, where Φ(t) is the cumulative distribution function of the standard normal distribution. For any δ > 0 and M > 0, there exists K ∈ N such that { Φ(M) − Φ( − M) }

<

. For such K, we can Þnd ε > 0 that satisÞes

¡

¢

< 2. Then, eq.(35) leads P ¡

1

max

| W

| ≤ M ¢

< δ. (36)

The convergence of (U

(α

), . . . , U

(α

)) to W means lim

P(max

| U

(α

) | ≤ M) = P(W ∈ [ − M, M]

). This completes the proof.

On the covariance of the random variables with bounded L

norm, we have the following proposition, which is used in the above proof.

Proposition 1. Let { v

}

be a sequence in L

(P ) such that k v

k

= 1 for all n, and v

→ 0 in probability. Then, there exists a subsequence { v

}

that satisÞes

E

| v

v

| < ε (37) for all different k and h.

This is a direct consequence of the following proposition.

Proposition 2. Let (Ω, B , P ) be a probability space, and Y, X

, X

, . . . be random variables. Suppose there exists K > 0 such that R

Y

dP ≤ K and R X

dP ≤ K , and X

converges to 0 in probability. Then, we have

n

lim

E | Y X

| = 0. (38) Proof. Let ε be any positive number. Because R

Y

dP < ∞ , there exists δ > 0 such that R

∆

Y

dP <

for any measurable set ∆ with P (∆) < δ.

For each n ∈ N , a measurable set A

is deÞned by A

= { ω ∈ Ω | | Y | > ε

3 √

K and | X

| > ε

3K | Y |} . (39) Because X

→ 0 in probability and A

⊂ {| X

| >

} , we can Þnd n

∈ N such that for all n ≥ n

we have P(A

) < δ, hence R

An

Y

dP <

. Since A

⊂ { ω | | Y | ≤

} ∪ { ω | | X

| ≤

| Y |} , we obtain for all n ≥ n

Z

| Y X

| dP = Z

An

| Y X

| dP + Z

A^c_n

| Y X

| dP

≤ ³Z

An

Y

dP ´

³Z

An

X

dP ´

+ Z

{|Y|≤₃^√ε_K}

| Y X

| dP + Z

{|Xn|≤_3K^ε |Y|}

| Y X

| dP

< ε 3 √

K

√ K + ε 3 √

K Z

| X

| dP + ε 3K

Z

| Y |

dP

≤ ε 3 + ε

3 √ K · √

K + ε

3K · K = ε (40)

In the last line, we use the fact R

| X

| dP ≤ ( R

| X

|

dP )

≤ √ K .

4 Likelihood Ratio of Multilayer Perceptrons

We apply the results in the previous section to the multilayer perceptron model, which is deÞned by eq.(8). We use the same notations as Section 2.4, giving the true function ϕ

(x) by eq.(14) and the locally conic parame- terization by eq.(17).

We need additional assumptions on the noise model r(y | u) to ensure the asymptotic normality conditions [AN] on the one-dimensional models.

15

Conditions on noise model (NM2)

1. For any compact set K ⊂ R , sup

E

| log r(y | u) | is Þnite.

2. Let h

(y | s) and h

(y | s) be functions deÞned by h

(y | s) = sup

u≥s

log r(y | u) and h

(y | s) = sup

u≤−s

log r(y | u), (41) respectively. For any compact set K ⊂ R and s ∈ R , sup

E

[h

(y | s)]

is Þnite.

3. For an arbitrary compact set K ⊂ R , there exist ∆

, ∆

⊂ Y and B > 0 such that

s

lim

h

(y | s) = −∞ for all y ∈ ∆

, (42)

s

lim

h

(y | s) = −∞ for all y ∈ ∆

, (43) and

Z

∆_±

r(y | ξ)dy ≥ B for

ξ ∈ K. (44) 4. For any compact set K ⊂ R ,

lim

sup

ξ∈K u∈K

E

£ sup

|u⁰−u|≤ρ

log r(y | u

) ¤

< ∞ . (45)

5. The density r(y | u) is three-times differentiable on u for all y ∈ Y , and for any compact set K ⊂ R ,

lim

sup

ξ∈K

E

h

sup

|ξ⁰−ξ|≤ρ

¯ ¯

¯ ∂

log r(y | ξ

)

∂

u

¯ ¯

¯ i

< ∞ . (46)

The above conditions are satisÞed by many important noise models. In the case of the Gaussian noise model and binary output model, they can be checked easily. In fact, the conditions 1, 4, and 5 are easy. On the conditions 2 and 3, stronger conditions will be checked in Section 4.

The next lemma shows that the conditions [NM2] implies the asymptotic

normality [AN] in some type of submodel in S

.

Lemma 1. Let w

(x) be a bounded function, w(x) be a positive, bounded function, and r(y | u) be a density function on Y which satisÞes [NM1] and [NM2]. Then, the statistical model { g(z; β) | β ∈ R} , which is deÞned by g(z; β) = r(y | w

(x) + βw(x))q(x), satisÞes the conditions [AN].

Proof. From [NM2]-1 and boundedness of w(x) and w

(x), for each β there is A > 0 such that E

| log r(y | w

(x) + β w(x)) | ≤ A for all x ∈ R . The fact E

| log q(x) | < ∞ implies the condition [AN]-1.

Since H

(z; t) = h

(y | w

(x) + tw(x)) + log q(x) and for any t there exists s

such that w

(x) + tw(x) ≥ s

for all x , we have E

[H

(z; t)] ≤ E

[E

[h

(y | s

)] + log q(x)]. The compactness of the range of w

(x) and the condition [NM2]-2 show the Þrst assertion of [AN]-2. The second one is similar.

We will show only on H

for the assumption [AS]-3, because the proof on H

is exactly the same. There exists M > 0 such that | w

(x) | ≤ M . Take

∆

⊂ Y and B > 0 in the assumption [NM2]-3 for a compact set [ − M, M ].

Then, for any z ∈ X × ∆

, we have lim

H

(z; t) = lim

h

(y | w

(x)+

tw(x)) + log q(x) = −∞ , and R

X ×∆+

f

(z)dµ = E

[ R

∆+

r(y | w

(x))] ≥ B.

From [NM2]-4 and the boundedness of w(x), for any β there exists ρ

> 0 and C such that E

[sup

log r(y | w

(x) + β

w(x))] ≤ C holds for all ρ ∈ (0, ρ

] and x ∈ R . This shows the condition [AN]-4. By a similar argument, [NM]-5 implies [AN]-5.

Theorem 3. Assume that the model is the multilayer perceptron model (8) with H hidden units, and the true function is given by a network with K hidden units for K < H. Under the assumptions [NM1] and [NM2] on the noise model r(y | u), we have for arbitrary M > 0,

n

lim

Prob

³ sup

θ

L

(θ) ≤ M

´

= 0. (47)

Remark. This theorem means that the order of the likelihood ratio of MLE is strictly larger than O

(1).

Proof. For the lower bound, it suffice to consider a submodel in the locally conic parameterization eq.(17). Let σ(x; ξ, h) be a bounded, monotone de- creasing function given by

σ(x; ξ, h) = 1

2 { 1 + s( − 1

2 ξ(x − h)) } = 1

1 + exp { ξ(x − h) } , (48) and { g(z; t, c) } be a submodel deÞned by

g(z; t, c, β) = r(y | ϕ

(x) + βw(x; t, c))q(x), (49)

17

where

w(x; t, c) = 1

p B(t, c) σ(x; c

, t +

), (50) and B(t, c) is a normalizing constant of L

(f

µ) norm given by

B (t, c) = Z

G(ϕ

(x))σ(x; c

, t +

)

dQ(x). (51) Because ϕ

(x) and w(x; t, c) are bounded functions, from Theorem 2 and Lemma 1, we have only to show there is a sequence in the basis of the tangent cone C, which converges to zero in probability. The set C consists of the functions

v(x, y; t, c) = 1 p B (t, c)

∂ log r(y | ϕ

(x))

∂u σ(x; c

, t +

). (52) Let a be a positive number that satisÞes G(ϕ

(x)) ≥ a for all x ∈ R . Such a exists because of the continuity of G(u) and the boundedness of ϕ

. Let F

(t) be a distribution function of the input probability Q. From the assumption that Q is absolute continuous with respect to the Lebesgue measure, F

is continuous on R . If we deÞne t

= inf { t ∈ R | F

(t) > 0 } ∈ R ∪ {−∞} , we have F

(t) > 0 for all t > t

, and lim

F

(t) = 0.

Since σ(x; c

, t +

) is bounded and converges to χ

(x) at every x for c → + ∞ , by Lebesgue’s dominated convergence theorem, we have lim

B(t, c) = R

−∞

G(ϕ

(x))dQ(x) ≥ aF

(t). Hence, for each t we can Þnd c

such that p

B(t, c) ≥

p

aF

(t) for all c ≥ c

.

For any t > t

and δ > 0, there exists c

(δ) > 0 such that σ(x; c

, t +

1

c

) ≤ F

(t) for all x ≥ t + δ and c ≥ c

(δ). Then, if a sequence (t

, δ

, c

) is chosen so that t

↓ t

, δ

↓ 0, and c

≥ max { c

, c

(δ

) } , the inequality

| v(x, y; t

, c

) | ≤ 2

√ a

¯ ¯

¯ ∂ log r(y | ϕ

(x))

∂u

¯ ¯

¯ q

F

(t

) (53) holds for all x ≥ t

+ δ

and y. Because F

(t

) → 0 and t

+ δ

↓ t

for n → ∞ , the sequence v(x, y; t

, c

) converges to zero for all x > t

and y, which means almost everywhere convergence.

If K ≤ H − 2, a different type of sequence can work for the proof of Theorem 3. Let W = { w(x; ξ, h, t) } be a family of functions deÞned by

w(x; ξ, h, t) = 1 p A(ξ, h, t)

1

2 { s(ξ(x − t + h)) − s(ξ(x − t − h)) } , (54)

where A(ξ, h, t) is a normalization constant of L

(f

µ) norm given by A(ξ, h, t) = E

h³ ∂ log r(y | ϕ

(x))

∂u

s(ξ(x − t + h)) − s(ξ(x − t − h)) 2

´

i

= E

£ G(ϕ

(x))

{ s(ξ(x − t + h)) − s(ξ(x − t − h)) }

¤

. (55) A subfamily of the multilayer perceptron in the locally conic parameteriza- tion is deÞned by

ψ(x; ξ, h, t, β) = ϕ

(x) + β w(x; ξ, h, t). (56) This is obtained by setting η

= ξ

= ζ

= δ = 0 (1 ≤ i ≤ k and i ≥ K + 3), ξ

= ξ

= ξ, ζ

= − ζ

= h, and η

= η

=

in eq.

(17). The basis of the tangent cone of the submodel { r(y | ψ(x; ξ, h, t, β))q(x) } consists of the functions of the form

v(z; ξ, h, t) = ∂ log r(y | ϕ

(x))

∂u w(x; ξ, h, t). (57) From the fact that G(u) is positive and continuous, and that ϕ

is bounded, there exist a, b > 0 such that a ≤ G(ϕ

(x)) ≤ b for all x ∈ R . For arbitrary h > 0 we can Þnd δ(h) > 0 so that for any ξ ≥ δ(h),

1

2

{ s(ξ(x + h)) − s(ξ(x − h)) } is larger than

on x ∈ [ −

,

] and less than h on x / ∈ [ −

h,

h]. Let h

> 0 be a decreasing sequence which converges to zero. If ξ

is taken so that ξ

≥ δ(h

), the normalization constant satisÞes

A(ξ

, h

, 0) ≥ Z

2hn

−¹₂hn

G(ϕ

(x)) ¡

2

¢

q(x)dx ≥ a

4 h

. (58) Thus, for all x with | x | ≥

h

, we have

| v(z; ξ

, h

, 0) | =

¯ ¯

¯ ∂ log r(y | ϕ

(x))

∂u

¯ ¯

¯ 1

p A(ξ

, h

, 0) 1

2 { s(ξ(x + h)) − s(ξ(x − h)) }

≤ 2 √ h

√ a

¯ ¯

¯ ∂ log r(y | ϕ

(x))

∂u

¯ ¯

¯ . (59)

For all x 6 = 0, the sequence v(z; ξ

, h

, 0) converges to zero from the fact h

↓ 0,. This means almost everywhere convergence, since Q is absolutely continuous with respect to the Lebesgue measure.

The next lemma on the functional space W will be used in Corollary 1 after Theorem 4.

19