Introduction to Automatic Speech Recognition

(1)

Spee h a d La guage Pro essi g

Le ture

Ba esia et ork a d Ba esia i fere e

I fo atio a d Co u i atio s E gi ee i g Cou se

Takahi o Shi ozaki

(2)

Le ture Pla Shi ozaki’s part

. / e ote

Spee h e og itio ased o GMM, HMM, a d N‐g a . / e ote

Ma i u likelihood esti atio a d EM algo ith

. / e ote

Ba esia et o k a d Ba esia i fe e e

. / @TAIST

Va iatio al i fe e e a d sa pli g . / @TAIST

Neu al et o k ased a ousti a d la guage odels . / @TAIST

Weighted fi ite state t a sdu e WFST a d spee h de odi g

(3)

Toda ’s Topi

• A s e s fo the p e ious e e ises

• Ba esia et o k

(4)

(5)

E er ise .

• Sho the de i atio p o ess of o tai i g

 













_







  K k k k K k k

m

L

1 1

1 log

,









μ

n

m

_k k





a i izi g

 

(6)

E er ise .

• De i e the ML solutio { , } of the Gaussia

dist i utio . The de i atio p o ess ust e

des i ed







| ,



0

(7)

E er ise .

• Gi e a t ai i g data

D

ith

n

t ai i g sa ples

D={x

₁

, x

₂

,…,x

_n

},

o tai ML esti atio fo GMM

ith

M

i tu es

You a assu e the a ia e is fo si pli it









                      _                         _  



 

    N i M m m i m N i M m m i m X w X w M M M 1 1 2 , , 1 1 2 , , 2 exp 2 1 log max arg 2 exp 2 1 max arg ˆ 2 1 2 1             GMM

(8)

E er ise .

• Assu e ou ha e a i itial odel pa a ete

Θ

₀

.

P o e that

if you take

₀ ₀

,

then the lower bound

₀ ₀

is equal to the log

likelihood

₀



|

₀





|

,

₀



,

₀



log

P

X





J

P

H

X









P

H

|

X

,



₀

,



₀





log

P



X

|



₀



J







 



_{ }



H

q

H

X

P

H

q

J

H









_log

,

|

(9)

E er ise .

• Co side the ‐ i GMM of the p e ious page.

Let

Θ . O tai s the follo i gs











₀



(10)

(11)

Graphs

• U di e ted g aph

• A g aph defi ed odes a d u di e ted a s

• Di e ted g aph

• A g aph defi ed odes a d a di e ted a s

• Di e ted A li G aph: DAG

• Di e ted g aph that does ot o tai a di e ted le

E a ples:

U di e ted g aph Di e ted g aph

(12)

Pare t, Child, A estor, Des e da t

B A

Node A is a pa e t of ode B

Node B is a hild of ode A

B A

C

D

Node B, C, a d D a e des e da t of ode A Node A, B, a d C a e

(13)

Bipartite

Whe odes of a g aph a e sepa ated to t o g oups a d

the e is o a i side the g oups, it is alled a ipa tite

(14)

Dire ted Graph a d Node Orderi g

A di e ted g aph is a DAG

The e is a o de i g of odes he e all a s fa e the sa e di e tio =The e is a u e i g of odes he e all a s go f o a lo e

u e ed to highe u e ed odes E

１

(15)

Outli e of the Proof

• State e t A:

• The e is a o de i g of odes he e all the a s fa e the sa e di e tio

• State e t B:

• A g aph does ot o tai a di e ted le

A

B

A

B

…Eas

(16)

(17)

E er ise .

• Is the di e ted g aph a DAG?

(18)

Ba esia Net ork BN

• BN is a g aphi al odel that ep ese ts a set of

a do a ia les a d thei o ditio al i depe de e

DAG

A

C B

(19)

De o positio of Joi t Pro a ilit a d BN

• B the p odu t ule, a it a joi t p o a ilit is

de o posed to a p odu t of o ditio al p o a ilities

• A DAG is ade

• Rep ese ti g the a ia les as odes

• Co e ti g the odes di e ted a s a o di g the o ditio al p o a ilities

  

A P B A

 

P C A B

 

P D A B C



P D

C B A

P( , , , )  | | , | , ,

A

C B

(20)

Co ditio al I depe de e a d Ar s

• Co ditio al i depe de e is ep ese ted

a se e

of a s

  

A P B A

 

P C A B

 

P D A B C



P D

C B A

P( , , , )  | | , | , ,

) | ( ) , , | ( ) ( ) | ( B D P C B A D P B P A B P  

    

A P B P C A B

 

P D B



P D

C B A

P( , , , )  | , |

(21)

Joi t Pro a ilit Defi ed BN

• P odu t of o ditio al p o a ilities asso iated ith

DAG al a s satisf the su ‐to‐o e o st ai t

Si e a Ba esia et o k is a DAG, ith a p ope o de i g of the a ia les, the p odu t has the follo i g fo

P oof:



|



, { ₁, ₂, , ₁}

1 







i i

N

i

i C C X X X

X

P 

That is, X_i does ot appea i the o ditio al pa t of X ,…, X_i_‐ . B thi ki g the su atio of the follo i g o de , e ha e:





_{ }





_





 





 _ _

 ₁ ₁

1 2

1 |

|

| ₁ ₁

1 X X X

N N

X X X

(22)

E er ise .

• Rep ese t the follo i g joi t p o a ilit a BN

  

|

 

|

 

|



( | ) )

, , , ,

(A B C D E P A P B A P C B P D B P E D

(23)

BN Represe tatio of a Categori al Distri utio

X

X … K

(24)

BN Represe tatio of a Gaussia Distri utio





















_



₂ 2

2 2

2

1 exp

2

1 ,

|











X

N

(25)

BN Represe tatio of a GMM

Gauss１

μ ,σ

Gauss μ ,σ

 





 

(

|

)

,

|

i

X

P

i

P

X

Gauss

i

P

X

P

i

i i

i









i

X

Mi tu e eight P o a ilit of i de

Gaussia dist i utio

(26)

Ba esia Net ork Represe tatio of a HMM

HMM

S

X

Ti e

• The et o k has u olled st u tu e

• The le gth depe ds o the i put se ue e

Ba esia et o k

X X X_T

S S S_T

T a sitio P o a ilit e.g. P(S_t=a|S_t-1=b)

E issio dist i utio o ditio ed the state

e.g. P(X_t|S_t=a)

(27)

E a ple of Alig

e t

Featu e se ue e: , , , , , , State se ue e: a, ,a,a,a, ,

S

X X X X

S S S X = , X = , X = , X = , _{X = , X = , X =}

S =a, S = , S =a, S =a, S =a, S = , S =

(28)

Represe tatio of a Repeated Stru ture

X

_T

V

X

_t

(29)

Represe tatio of Para eters

s all i les ep ese t pa a ete s

i

X

 

_

 







N

i

i i

X

Gauss

i

P

X

P

1

,

|







₁,₂,₃_N



 



_N



S 



₁,



₂,



₃



(30)

E er ise .

• Fill the la ks so that the follo i g HMM a d the

BN e o e e ui ale t

a

S

X X X X_T

S S S_T

   

 

b b

a a

 

, , ,



Xt s_t s_t



N |  ,

X _a _a

N |  ,

P a P . . .

. . _.

.

I itial state

p o a ilit P S_S t|St‐ St =a St =

t‐ =a

S_t‐= . I itial

state

HMM

BN

X _b _b

(31)

Fa tor Graph

• A ipa tite g aph he e o e side of a ia les ep ese t a do a ia les a d the othe s ep ese t fu tio s

• The a s ep ese t depe de ies of the fu tio s to the a ia les

• A fa to g aph defi es a joi t p o a ilit



X₁, X₂, X₃, X₄



f₁



X₁, X₂, X₃

   

f₂ X₂ f₃ X₃, X₄



P 

X X X X

f f f

E a ple:

Va ia le odes Fa to odes





 

_s

iables of

subsets s

s

N

f

X

P







var 2

(32)

Fa tor Graph Represe tatio of Ba esia Net ork

Ea h o ditio al p o a ilit a e ega ded as a fa to

A

C B

D E a ple

  

A P B A

 

P C A

 

P D A C



P | | | ,

Ba esia et o k

A B C D

P A P B|A P C|A P D|A,C

(33)

Pro a ilisti I fere e

• Ma gi al a d o ditio al p o a ilities a e o tai ed

f o a joi t p o a ilit appl i g the su a d

p odu t ules



A

B

C



P

,

 





A C





C B A P B P , , ,

 



_





C B C B A P A P , , ,







_





C C B A P B A

P , , ,

(34)

Distri utio Propert a d Co putatio al Cost

• P odu t is dist i uti e o e additio

• The sa e p ope t holds fo su a d a , a d

p odu t a d a

 

_

 



 



N

x N

x

f

a

x

af

1 1

 



af

x



a



f

 

x



x

max



 



a

f

x



a



f

 

x



x

max







Nu e of p odu ts：N

Nu e of su atio ：N‐

Nu e of p odu ts：

(35)

Co putatio al Cost of Margi alizatio

 

_





    1000 1 1000 1 1000 1 , , ,

B C D

D C B A P A P

Suppose A, B, C, a d D take

possi le alues

# su

atio =

=

If the joi t p o a ilit is de o posed to:



A,B,C,D



P(A|C)P(B)P(C)P(D)

P 

 







  

 

                   





      1000 1 1000 1 1000 1 1000 1 1000 1 1000 1 | , , , D B C B C D

D P B P C P C A P D C B A P A P

# su

atio =

(36)

Whe the Fa tor Graph is Li ear

X X X X

f f f

Suppose e a t P X

 



 





 





 



























                                            







4 5 2 1 4 5 1 2

1 2 4 5

1 , , 1 , , , , , , , , , , 5 4 4 4 3 3 2 1 1 3 2 2 5 4 4 4 3 3 3 2 2 2 1 1 5 4 4 4 3 3 3 2 2 2 1 1 3 X X X X X X X X

X X X X

(37)

Message Passi g Vie of the I fere e

X X X

f f f

X X f 3 3 X f   2 1 X f 

  _f₂__X₃

4 4 X f  

 

























                              



4 5 2 1 1 , , 1 ,

, ₃ ₁ ₁ ₂ ₃ ₃ ₄ ₄ ₄ ₅

2 2 3 X X X X X X f X X f X X f X X f X P 2 2 f X 

 _X₄_ _f₃



 ₂

1 X

f

 _X₂__f₂

(38)

(39)

Pro a ilisti Models a d Their Para eters

x

  

x

w

p

x

w

_AM

 

p

w

_LM



p

,



|

,



|



x w

LM



AM





x

_|





p

Gaussia dist i utio ,

(40)

ML Trai i g a d Predi tio

x



x



x x x_N



_x

_|

_

*



p

T ai i g set D

Test sa ple









_











 _n n

x p D

p | argmax |

max arg

*

(41)

Ba esia Approa h

• T eat pa a ete s as a do a ia les

x

x x x_N

Λ







_{ }





 

_{ }

  

p



x

 

p D



D p

p D

p x

p D

p D x p

D x

p | | | |

, ,

|       







 



P edi tio of a e sa ple is fo ulated as a e aluatio of o ditio al p o a ilit gi e a t ai i g set



_,D_, X

   

 P  P D _| 

 

P X _| 



p

(42)

Defi itio s of Ter s

• A p io i dist i utio of pa a ete s

• P o a ilisti odel

• A poste io i dist i utio of pa a ete s

• P edi ti e dist i utio







_{ }

  

D p

p D

p  |  |  

 



p



x

D



p



x

 

p

D



p

|





|



|





x

_|





(43)

E aluatio of A Posteriori Distri utio

• E ept fo e si ple odels, ho to e aluate the

a poste io i dist i utio is a ig issue si e it

e ui es i teg atio s o e a a ia les







_{ }

  

D

p

D

p

D

(44)

Approa hes

• A al ti al e aluatio

• Ideal, ut o l appli a le fo e si ple odels

• Fo p a ti al odels, losed fo solutio is usuall

ot o tai ed. Nu e i al i teg atio is also ot

feasi le he the e a e a a ia les

• Va iatio al Ba es

• Ca e applied to la ge odels if p ope a al ti al

app o i atio is i t odu ed

• Sa pli g

(45)

Co jugate Prior

• Fo so e o i atio s of p io a d p o a ilisti odel,

poste io takes the sa e fu tio al fo as the p io

Pro a ilisti odel Co jugate prior

Bi o ial dist i utio Beta dist i utio

Multi o ial dist i utio Di i hlet dist i utio

(46)

E er ise .

• Assu es a p o a ilisti odel P x|μ , a t ai i g sa ple x , a d

a p io dist i utio of a pa a ete P μ a e gi e as follo s.

x x

μ

 

       2 exp 2

1 _2

  P









_      _   2 exp 2 1

|  2

  x x P







_{ }

  

1 1 1 | | x P P x P x

P    



x x



P



x 

 

P  x



d

P | ₁



 | | ₁

 



Esti ate poste io dist i utio

Esti ate p edi ti e dist i utio





c dx

cx  







2

exp

Note:

Gaussia dist i utio ith ea μ a d a ia e

Gaussia dist i utio ith ea a d

(47)

(48)

Le

a .

If a g aph does ot o tai a di e ted le, the the e e ist

at least o e ode that has o i o i g a

(49)

Notatio for Co ditio al I depe de e

• Let A,B, a d C e disjoi t sets of a do a ia les.

Whe the follo i g e uatio holds, e sa that A is

i depe de t of B gi e C, a d de ote it as A

╨

B|C



A

B

C

 

P

A

C



P

|

,



|



A

B

C

 

P

A

C

 

P

B

C



P

,

|



|



A B C

 

P A B C

 

P B C



P , |  | , |



A

╨

B|C

A

╨

B|C

(50)

Graph Stru ture a d Co ditio al I depe de e

B i estigati g the g aph st u tu e, e a ead

elatio ships et ee a do a ia les

A

C B

D

A

╨

B|C

A

╨

D|C,B

?

(51)

Tail‐To‐Tail

C A B C A B







_







_



 

  

C C C P C B P C A P C B A P B A

P , , , | |

I ge e al, P A,B is ot e p essed as

P A P B . The efo , A╨B|Φ does ot hold.

Φ is a e pt set

P A,B|C is e p essed as P A|C P B|C . The efo e A╨B|C holds.







_{ }





 

_{ }

  



A C

 

P B C



(52)

Head‐To‐Tail

C A B C A B Head Tail







_







_

  

 



C C A C P C B P A P C B A P B A

P , , , | |







_{ }





  

_{ }

 





A C

 

P B C



P C P C B P A P A C P C P C B A P C B A P | | | | , , | ,   

I ge e al, P A,B is ot e p essed as

P A P B . The efo , A╨B|Φ does ot hold.

(53)

Head‐To‐Head

C

A B

C

A B

Head Head

A B PA B C P    A P B P C A B    P A P B P

C C

 





, ,



| ,

,







_{ }



    

_{ }



C P

A C P B P A P C

P

C B A P C

B A

P , |  , ,  |

I ge e al, P A,B is e p essed as P A P B . The efo , A╨B|Φ holds.

(54)

Blo ki g a Path

• Fo a Ba esia et o k, let A a d B e a ode, a d C e a

set of odes that does ot i lude A a d B. We sa a path

f o A to B is lo ked he eithe of the follo i gs holds

• O the path f o A to B, the e is a ode i C a d the o e tio of the a s is tail‐to‐tail o head‐to‐tail

• At o e of the odes o the path f o A to B, the o e tio of the a s is head‐to‐head. I additio , the ode a d its all

des e da ts a e ot i luded i C

D A

B

C

C F

E

Blo k

(55)

d‐separatio

Fo a Ba esia et o k, let A, B, a d C e e lusi e

sets of odes

• We sa A is d‐sepa ated f o B C if all the paths

sta ti g f o a ode i A a d e di g at a ode i B is

lo ked

• Whe A is d‐sepa ated f o B C, A

╨

B|C holds fo

(56)

Ma i izatio of Joi t Pro a ilit

• O tai ed epla i g Σ i the su ‐p odu t

algo ith ith a

 Ma ‐p odu t Algo ith



_N



X X

X

N

P

X

N



,

max

arg

,

₁ ₂

, , , 2

1

(57)

Su ‐Produ t Algorith for Tree

• Message passi g

• Leaf odes

• Va ia le ode：

• Fa to ode：

• Va ia le ode to fa to ode:

• Fa to ode to a ia le ode:

• Ma gi al p o a ilit

 

1

f x x



 

x

f

 

x

f 





 

_

_

 





i x f f

x



_i

x



 

_{ }





_

 

    M m m f x x x M x

f x f x x x _m x

(58)

EM for HMM a d Effi ie



_,₀







HMM



K _| X _,₀



_logHMM



K_, X _| 



Q

K

a

•_• TT x K

_X

•

Di e tl e u e ati g all the paths is i possi le

Q‐fu tio a e effi ie tl e aluated if poste io s a d a e o tai ed, he e s’ a d s a e HMM state ID

k  s | X,₀

P _t

k _₁  s ,'k  s| X,₀

P _t _t

T

k k

k

K  ₁, ₂,, X  x₁,x₂,,x_T