STUDIES ON THE 0PTIMAL STOCHASTIC CONTROL 0F "NEAR ECONOMIC SYSTEM

STUDIES ON THE 0PTIMAL STOCHASTIC CONTROL 0F "NEAR ECONOMIC SYSTEM

by

Koukyu KAWABATA'

(Received October 5 , 1976)

SYNOPSIS

Recently many of articles have appeared concentrating upon the application of modern control theory to the short-term economic stabilization policy. The optimal control problem is defined as a discrete-time tracking problem (nominal state 4nd nominal control trajectories are tracked) for a linear stochastic economic models with a quadratic cost functional. This problem is solved analytically by using of the Kalman filter and dynamic programming. The general recursive formulas of estimation and control are independently deduced and furL thermore the character of the optimal control policy is analized.

1. INTRODUCTION

A major concern of economic policy is to control the economic activity, which fluctu- ates dynamically, into a desired direction(e.g.,optimal economic growth path). In fact, it is fresh

in our memory that the great depression in 1930's and consecutive recession at that time might lead a secular stagnation thesis or a breakdown thesis of capitalism. Thus even with the medium size fluctuations may not be desirable from a social economic activity. From the point of view of these consideration it will be planned for.us to apply the modern control theory into the economic planning theory especially short--term economic stabilization policy.

N6w let us consider that the system equation of the linear economic model is given as follows:

yt = Hoyt+Hiyt-i+"""+Hh-iyt-h+i+A'hyt-h+Giut-i+"""+Gr-iut- r+i

+GrUt- r+ Fi zt-i+"'"'+Fp-izt-p+i+ Fp zt-p+et-i (1)

where yt-ii's a (niÅ~1) vector of endogenous variables appearing with lag i, ut-i is a (mi Å~1) vector of control variables, zt-i js a (2i Å~1) vector of exogenous (uncontrollable) variables, et-i is a (niÅ~1) random error vector, Hi, Gi and Fi are (niÅ~ni), (niÅ~mi) a'nd(niÅ~ei) non-sto- chastic matrices of parameters, respectively. Furthermore we can transform the eq.(1) into the 1-st order linear simultaneous difference equation as :

yt Yt-1 l Yt-k+1 Ut-1

:

l

:

Ut- r+1 Zt-1

: :

Zt-P+1

H, o •••••o

o o•--o

-- - --- --- --t

-- - ---

-- - --- -i- -{- }-- --- _{--- ---}

o -• --•o

yt Yt-1

:•

Yt-h+1 Ut-1

l l

Ut-r+1 Zt-1

:

Zt-P+1

+

Hi H2 "' Hk G2 "' Gr F2 "' Fp I O ••• O O ••• O O -• O

-t---}-e---}--- ^---

O O •••IO O ••• O O ••• O o o••- o o -• o o •- o 0 O -• O I -• O O ••• O O O •- O O -•IO O -• O

oo-•oo-•oo•-o

O O •- O O-O I -• O

}---e-e---}---t---e---

O O -• O O •- O O -•IO

Yt-1 Yt-2

l

Yt-h Ut-2

I

Ut-r Zt-2

:

Zt-p

*The Department of Computer Science

+

G, o

o I o

o o

:.

o

Ut-1+

F, o

o o

o I o

o

Zt-1+

et-1 o

o o I : o o

o

or equivalently

x, = Aoxt+A,x,-,+B,u,-,+Clz,-i+vt-i

where xt and vt-i are ((kni+(r-1)mi+(p-1)2i)Å~1) vector, Ao and Ai are ((leni+(r-1)mi +(p-1)Vi)Å~(leni+(r-1)mi+(P-1)Vi)) matrices, Bi is ((leni+(r-1)mi+(p-1)2i)Å~mi) matrix and Ci is ((leni+(r-1)mi+(p-1)Vi)Å~ei) matrix. Furthermore we can rewrite the above equa-

tlon as :

or

xt = (I-Ao)-iAixt-i+(I'Ao)-iBiut-i+(I-Ao)-iCizt-i+(I-Ao)-ivt-i = Axt-i+But-i+ Czt-i+et-i

xt+i=Axt+Bttt+czt+6t (2)

Eq.(2) is a so-called state variable form of system equation of linear economic model. Next we employ the observation equation of this system :

where ot and nt are ((leni+(r-1)mi+(p-1)gi)Å~1) vectors of observed variable and measure- ment error and D is a ((imi+(r-1)mi+(p-1)gi)Å~(leni+(r-1)mi+(p-1)ei))matrix. And furthermore we introduce the minimizing of quadratic cost functional :

N

.,,I 1, 1'I., E..(1) =.,,III,,il .E.l,(pu., ( (Xt -'Et )' Vt (Xt -•i}t)+(ut-i- tit-i)TJVt-i(ut-i- u"t"i) )) (4)

where hit is a ((feni+(r-1)mi+(p-1)ei)Å~1)nominal state vector, tit iS a (miÅ~1)nominal control vector, Vt and VVt-i are ((kni+(r-1)mi+(p-1)4)Å~(kni+(r-1)mi+(P-1)2i)) and (mi Å~ mi) symmetric and positive semi-definite matrices, respectively. Thus we have get the general discrete-time stochastic control problem.

In the next section we derive the recursive formula on estimating the system state by using of the Kalman filter.' Section 3 and 4 deal with the algorithm of optimal control policy through dynamic programming and the character of optimal control policy. Section 5 includes some concluding remarks.

2. RECURSIVE ESTIMATION OF THE SYSTEM STATE

Suppose that the equation error 6t and the measurement error vt follow the normal distri- bution with

Vt:E(e,)=E(n,)= O, E(etet')= Q, E(vtnt')=K E(6tnZ)=O (s)

Then, by straightforward application of the Kalman filter, the minimum-variance, unbiased

estimates of the system state are obtained as :

M,.,=P,.,DT(R+DP,.,DT)-i (7)

S,.,=(I- M,.,D )P,., (s)

it+i=Ax2 +But+ Czt (9)

x2.,= X,.,+M,.,[o,.,-Di,.,] (lo)

3 . DERIVATION OF THE OPTIMAL CONTROL POLICY

On the LQG problem we can apply the well-known certain equivalent principle. Conse- quently we can separately deal with the estimation of the system state and the optimal control. The optimal control policy by using of dynamic programming are described below.

Now we define :

rt(xt, ut.i) -A` (,xt-Xt)' Vt(,xt-it)+(ut-i- tit-i)'Wt-i(ut.i- uAt-i) (13) From eq.(13) we can rewrite the eq.,(4) as :

N

min E(J)= min E(2rt(xt,ut-i)) (14)

zao, Ul, .", UN.l Uo, Ul, "•, UN-1 t=1

Hence the optimal control policy at the last stage are shown below.

a. The last stage Now define

E(rN) 4. f(E(rNlo"-i, u"-2)]p( o";' u"-2)d(o";' u"-2) as)

and, for the simplicity of the notation, define

AN A. E(rNlo";'uN'2) (16)

itN A-. f(rN(xN, uN-i))p(,cNlo'V-i, u"-2)d,vN a7) or

The probability density function can be rewritten as :

p(xNlo"-', u"-2) = fp(xNlxN"i, uN-i, 6N-i)p(xN.i, uN-i, eN-ilo"-i, u"-2)

•d(XN.,, UN-,, eN-,) (18)

The equation error eN-i is assumed to be statistically independent of other processes. There- fore,

p(xN-,, uN",, e.-,loN-i, uN-2) = p(x.-,, u..,loN-i, uM2)p(e.-,) (lg) On the other hand rN(xN, uN-i) are

rN(,cN, uN-i) = (xN-,i}N)T iVN(xN- iN)+(uN.i- uAN-.i)TWN-i(uN-i- uAN.i) (20) Substituting eq.(2) into eq.(20), we get

rN(xN, uN-i) = {(AxN-i+BuN-i+ CzN-i+ eN-i-XN)' VN(AxN-i+BuN-i+ CzN"i+ eN-!

-`i}.)+(u.J,- u'h.-,)TJV.-,(u.-,- u".",)} = .xff-,AT V.Ax.-,+2uk-,BT iV. (Ax.-, + CzN- , - X. )-2uS-,W.-, u-.-, + ufi.,(B T V.B+ W.-, )u.-, + 2x.T AT V.( Cz.- , - i.) +( CzN-, -X. )T V.( Cz.-,-iN )+ uA.T-,WN-, uA.-,+ 6S-, VN eN-i

1)' pto and .Xo are initially given.

Consequently we have

A. = f (x.T-,AT V.Ax.-,+2uff-,BT V.(Ax.-,+ Cz.-,-X.)-2uS-,WN.,uAN-,

+uff-,(BTV.B+W.-,)u.-,+2x.T-,ATV.(Cz.-,-.ii.)+(Cz.-,-,i}.)T

. V,,(CzN",-i.)+ uA.T-,WN-,ti,,-,]p(xN-,, u.-,l oN-1, uN-2)d(xN-,, uN.,)

Let uk-i designate the optimal uN-i which minimizes AN and q" the corresponding minimum,

i.e.,

qk 4- v.ip AN (22)

From 0AN/0uN-i =O, we get the optimal control as :

uk-,=-A.-,xN.,+K.-, (23)

where

xk-i= E(xNffil o"-i, u"-2) 2) (24) A.-,= (BT V.B+ VV.-, )-iBT V.A (2s)

K.-, = (BT V.B+ W.-, )-i(rv.-,a.-,+BT V.`i}.-BT V. Cz..,) (26)

Then substitution of eq.(23) into e, q.(22) yields qN' as follows :

q" == f [xN'-iA' VNAxN-k+2(-zlN-ixk-i+KN-i)TBT VN(AxN-i+(CzN-i- iN))

-2(-A.-,,xX-,+KN-,)TW.-,uA.-,+(-A.-,,vknv,+K.-,)T(BTV.B+rv.-,) t

• ( -A .- ,x"-, + K.- , )+ 2x.T- ,ATV.( Cz.- , - i. )+ ( Cz.-, - thN )T V.( Cz.-, -i. ) + tiiTv-iWN-iuAN-i)P(xN-i, uN-i1 o"-i, u"-2)d(xN-i, aN.i)+ trace ( VNQ)

== E(xN'-iri,scN-i+ÅëiIo"-i, u"-2) (27)

where

r, =AT VNA-n,AN-i (2s)

Åë, = E((,x.nt , - xre-, )Tn, zl .- ,(x.- , - xk- , )+ 2(,T.- , - x"., )Tn, K.", + 2,xkZ ,

'Afi-i(rvN.i tiN.i -BTWi )+ 2xN'- iA 'Wi + Si l o"-i, u"-2] +trace( VNQ) (3o) S, = 2K.T-,BTW,-2K.T-,rv.",ti.-,+K.T-,(BT V.B+VV..,)K.-,-zS-,CT V.i.

+zff",CTV,+i.T V.i.+tifi.,rv.-,uA.-, (31)

Åë, == V.CzN-,- VNiN (32)

b. Last two stages

In this stage observed variable o"-2and control variable u"-3 are assumed to be known.

The control variable uN-2 affects both rN-i and rN. Then it is seen that E( rN-1(XN-1, UN-2)+ rN(XN, UN-1)]

= f{E(( rN-i+ rN)l o"-2, u"-3)}P(o""2, u"-3)d(o"-2, u"-3) (33)

Because the information o"-2, u"-3 is contained in o"-', u"-2, we can rewrite

E(rNloN-2, uN"3) == fANp(oN-i, uN-21o"-2, u"-3)d(oN-i, uN-2) (34) Now we define

qN-, ,A= E(rN-il o"-2, uM3)+ fq" p(oN-ilo"-2, u"-3)doN-i (3s)

and, analogous to eq.(16), define

AN-i 4. E(rN-ilo"'2, u"-3) (36)

2) This is equal to the estimates of the Kalman filter which was shown in the preceding section

or

AN-i -Apt frN-i(aciv-!, uiv-2)p(ac?v-Jo"-2, uNnd3)dxN-i

=: frN-i(XN-i, UN-2)P(XN-iIXIv.2, Ulv-2, eN-2)P(xiv-2, uN-2loN"2, uN-3)

•l}(6N-2)d(xN-i, xN-2, uiv-2, eN-2) Åq37)

Eq.(35) can then be rewritten as

, qN-i = AN-i+ fqXP(oN-ilo"ww2, uiV-3)do.-,

MakiRg use ef eq. (27) we obtain

qN- i : AN- i + E( xNT- iri xN-i + Åëi l oN'2, aN'3)

: E ( x.Tww ,( V.-, +A )x.", - 2x.T- , V..,X.rw , + i.T- , V.-,,i}.me , + u.Tm,Wil.-,u.rm,

-2u;l]-2JVN-2aN.2+ ti2Tym2Wiv-2u-Ar-2+ÅëJ oNww2, u"wh3) Åq3s)

Substituting eq.(2År into eq.(38År we get

q.nv, = E((Ax..,+Bu.fi,+ Cz.-,+ e.ww,)T( V7r.r,+ r, )(Ax..,+Bu.-,+ Cz.r, + 6N.2) -- 2(AxN-2+BuN-2+ CzN-2+ eN-2)T VN-i iN-i + i}ff- i VN-i,i}N-i + esg-2WAr-2UNps2 - 2UXIny2rvN-2 es-'Af-2 + aNT-2WhY-2 tiNum2 + Åë, l eN-2, uAr-3]

: E(x.T-,A( V.-, + r, )Ax.rv, LF 2,xff-,A T(( V.-, + r, ÅrCz.-,- VN-,diN-,}

+ ug-,(B T( V.ww , -l- r, )B+ W.-,) u.-, + 2 uk-,(B T( V.", + r, )Ax.-, + B '( VN-i + L )CzN-2 - WNrm 2 aN-2 - B T IZN-i,iifNrm !] + zrc-2C T( IVN- 2 + ri ) CzN-2 - 2zfiww 2C ' VN nv iiN- i + iN'- i YN" i iN- i + ffN'pt 2rvN-2

-u-,v-,+ Åë,+e.T-,( V.-,+r, )6.-,i oN-2, uN-3) (3g)

Thu$ the optimal coRtrol at this stage can be derived a$:

uk-2 =-.`ÅqI N-2xk-2+KN-2 (4e)

where

x"-2 =E(xN-21orvm2, uNmu3) (41)

A.-,=(BT( V.-,+r, )B+W.-,)-iBT( V.-,+r, )A Åq42)

K.-, = (BT( V.nv,+r,)B+rv.-,)-i(rv.-,ti.ew,+BTV.i.--BT( V.ma,+r,)Cz.-,) (43År

r,=AT(V.-,+r,)A-n,A.-, • (44)

n, =AT( Y.rm,+r, )B (4 5) Substitution eq.(40) into eq.(39) aiso yields : q"-i = E( ar?S-2A'( VN-i-l--L)A,,xN-2+2.xge2A'(( VN-!+L)CzN-2- VN-iiN-i)

-l- (-ANnv 2x".2+ KN-2 )'(B T( VN-i + ri )B+ rvN-2](-AN-2x"-2+ KN-2) +2( -A.-,x"mu,+ K.m, )T(B T( V.ew , + r, )Ax.-, + BT( V.-L , + r, )Cx.., - VV.ww, tiN., - BT V.-,jE}. ww ,) + xfi-, C T( V. nv , + L ) Cz.ww, - 2xN'm, C T V." ,i.-, + jii?e- i VN-iiN-,+ u-N'-2WN-2ti.nd,+ Åë, + es-, Åq VN-, + r, )&-, l oN-2, uN-3]

=E(xNT-2 r2xNm2+Åë21eN-2, ctN'3) Åq46)

where

Åë2 =: E((xN-2-x"-2)Tll2AN-2(xN"2-x"-2)-l-2(xN-2-x""2)'n2KN-2 + 2xk.T,A.T-,(rv.-, ti.-, - B Ter, )+ 2x.Twa ,A Tur, + s, I oN-2, uN-3)

+trace(( VN-i+ ri )Q) (47)

S, = 2K.T.,BTV,-2K.Trm,W.-,tt-.L,+K.T",(BT( Y.-,+r, )B+rv.-,)K.rm,

- xfi rm,CT V.r,i.-,+xgn,C TÅë, +i.T-, Y. nv, ab.",+ u-k ww ,W. nv,ti..,+ Åë, (4 8)

W2 =(VN-i+r,)Cx.m,- IVN-,iNnv, (49)

e. Gefteral ease

Let us consider the general case t=: k. Sequences of control variabie$, uh, uk+i, "', uN-i,

N

are determined so as to minimize the sum E(pu.,.r,t). At this stage the policy is based on the observation sequence ok and past control action uk"i which are assumed known. Then we can also write

E( ?i.i,k) = f{E(,th. ,( rtloh, uk-i)) }p(ok, uk-')d(ok, uh-i) (so)

Next, we adopt the notation

N

qh"+i =A min{E(rk+iloh, uh-')+2 (E(rtlok,ule-')]} (sl) ,u-, •",UN-1 t=k+2

By applying the principle of optimality from the theory of dynamic programming, we have qk'.i = rr}iln (E(rk+iIoh, uh-')+ fqh"., p(oh+iloh, uh-')doh.i] (s2)

Letting

Ak+i =A. E(rh+ilok, uh-') (s3)

Eq.(52) becomes

qk"+i = m.i,n (Ak+i+ fqk"+2 P(ok+iloh, uh-')doh.i] (s4)

Eq./(54) is valid for all le = O, 1, 2, •••, N-1 if it is assumed that

q"., =O (55)

because for le = N-1 the prQper expectation for•optimization is

q" = minAN UN-1

Thus optimal control u: at the general case is now found to be, analogous to the previous two cases, as follows :

ul == -Atx:+Kt (s6)

where

xt'=E(xtio`, u`=i) (57)

A, = (B TÅq V,.,+ r.r, -, )B+W, )n' iB T( V,.,+ r.-,-, )A (58)

K, = (B T( V,.,+ I-'.r,-, )B+W, )- i(W,ti,+BT V,.,i,.i-BT( Vt.i+ rrv-tui)Czt) (s9)

r.., =AT( V,.,+ r.-,r, )A -n.- ,A, (6 0)

n.-, =AV'( V,.,+ r.-,T, )B (6 1)

The optimal performance is

qt'.i == E(xi'TN.t,xt+ÅëN-tlo`, ut-i) (63)

where

ÅëN-t == E((,rrt-,e:)'nN-tAt(,tt-xX)+2(xt-,x:)nrv-tK,+2x:'AtÅqVVtuAt-BiiWN-t)

+2,xi'ATWN-t+SNrtlot, utmi]+trace((Vt+i+rN-t-i)Q) (64)

S.-, = 2KiiB i'ur.-,-2Kii'JV, u"`, +Kii'(Bi'( V,.,+ I-'.-,-, )B+W, )K,

-zii'CTV,.,th,.,+z}i'CTW.-,+ iiih V,.,,i},.,+ u-I' W, u'-i+ Åë.r,-, (65)

Wt,r, =(V,.,+rN-,-,)Cz,- V,.,iiii,., (66)

4. CHARACTER OF THE OPTIMAL SOLUTION

As was seen in the former consideration the optimal control policy of stochastic economic

system led the linear feedback control policy. In this section we consider the character

of this feedback control policy. Let e-t(1) donote the one period ahead prediction error at

tlme t, Le.,

eAt(1) == ot-Dhi't (68)

The eq.(10) can be written as :

xf+i= Ax:+But+ Czt+ Mt+i e'-t+i(1) ' (69,)

Substituting eq.(56) into eq.•(69) yields

x2+i = Mt+i e-t+i(1)+ BKt+ Czt+(A-BAt)x: (7o)

Recursion and substitution lead to

x:+i = Mt+i eAt+i(1)+BKt+ Czt+(4-BAt)

• {Mt eAt (1)+ BKt-i+ Czt-i+ (A-BA t-i )x7- i} (7 1)

Successive recursion and substitution, and after setting Xo = O for simplicity, finally yield

tt

xr+i == Mt+ie-t+i(1)+BKt+ Czt+2 {[ll (A-BAj))(Mie-'i(1)+BKi-i+ Czi-i)} (72) i--o j=i

Then the optimal control policy at time (t+1) is

tt

ut+i = -At+i (Mt+ieAt+i(1 )+ BKt+ Czt)-At+i Z {(n (A-B4 )](Mi eAi(1)+BKi-i t=O J'-Ti t

+ Czi-i)}+Kt+i = 'At+iMt+ie""t+i(1)-At+i2 {( N. (A-BAj))Mie-i(1)}+Kt+i

i=O J=t tt

-At+i(BKt+ Czt)-At+i{Z (n (A-BAj))(BKi-i+ Czi.i)} (73)

i=O J'--i

The first term of the eq.(73) represents an action which is proportional to the current one period ahead prediction error, the scond term reflects aniaction proportional to a weighted sum of past errors in prediction and the residual terms denote the independent term on prediction error.

In a like manner we can derive the similar relationships between control variables ut and observed variables ot. Rearranging eq.(10), we have

xt'., == M,.,o,.,+(I-M,.,D )i,., (74)

Substitution of eqs.(9) and (56) into eq.(74) yields

x2+i = Mt+iot+i+(I- Mt+iD)(A-BAtÅrix2+ (I-Mt+iD )(BKt+ Czt) (75) Successive recursion and substitution, and again after setting io= o , finally give

t t+1

x2+i = Mt+i ot+i+ (I- MtD )(BKt+ Czt)+ 2 { [ H (I- MjD)(A - BAj-i)](Mioi i--O j =i+1

+ (I- M,D )(BK,-,+ Czi-i)) }

Therefore the optimal control policy at time t+1 is

t t+1

uf., = -A,.,(M,., o,.,+ (I- M,D)(BK, + Cz,) )-A,., 2 {( fi (I- M,D )(A - BA,-,)]

i--O .J'--i+1

t t+1

•(Mi oi+(I-MiD)(BKi-i+ Czi-i))}+Kt+i = -At+iMt+iot+i`-At+i:.=,{(,fi...,.SI -M,D)(A-BA,-,)]M,O,}+K,.,-A,.,(I-M,D)(BK,+ c.,)-A,., ik {Qli'i-(I i--O j'--i+1

-MjD)(A-BA j-i))(I- MiD )(BKi-i+ Cz i-i)} (76)

We can again understand that the optimal control policy is composed of three parts, which are parts proportional to the present observation and to a weighted sum of past obser- vations and independent of the observations.

Thus we can extract that the character of optimal control policy becomes well-known proportional-plus-integral control. Furthermore it is known that the proportional-plus-integral feature resembles the feature of exponential smoothing method and results in smoother control actions and adds to overall stability.3) Therefore these approaches are fully accepta- ble from the point of view of actual economic stabilization policy.

3) See K. P. Vishwakarma(4) p3.

5. CONCLUDING REMARKS

Despite of our approach, there ,is another method which concentrates upon the reduced form equation of linear econometric model instead of system and observation equations and introduce the quadratic welfare functionf) Furthermore K.P. Vishwakarma( 4 ) formulates the problem into discrete-time regulator problem under system and observation equations.

The principal characteristics of this paper was leading the recursive estimation formula by using of the Kalman filter since system state could not be directly measured and defining the problem as a discrete-time tracking problem instead of regulator problem. The advan- tage of our methodology will be shown by applying it into real economy.

REFERENCES

( 1 ) Aoki, M. : Optimization of Stochastic Systems. Academic Press, 1967.

( 2 ) Chow, G.C.:`"Optimal Stochastic Control of Linear Economic Systems,'" Journal of Money Credit and Banking,II (August,1970), 291-302

(3 ) :C'Optimal Control of Linear Econometric Systems with Finite Time Horizon,t" International Economic Review, XIII(February,1972), 16-25.

( 4 ) Vishwakarma,K.P. : Macro-economic Regulation. Rotterdam University Press, 1974.

4) For such a method, we can point out the method of G. C. Chow (2,3).