Conformal geometry and branching laws for unitary representations attached to minimal nilpotent orbits

C. R. Acad. Sci. Paris, t. 326, SCrie I, p. 925-930, 1998 Theorie des groupedGroup Theory

Conformal geometry and branching laws for unitary representations attached

to minimal nilpotent orbits

To&y&i KOBAYASHI a, Bent BRSTED b

a Graduate School of Mathematical Sciences, University of Tokyo, Meguro, Komaba, 153-0041, Tokyo, Japan

E-mail: toshiQms.u-tokyo.ac.jp

’ Department of Mathematics and Computer Science, Odensr University, Campusvej 55. 5230, Odense M, Denmark

E-mail: [email protected]

(Requ le 4 novembre 1997, accept6 apt& r&vision le 23 f&rier 1998)

Abstract.

RCsumC.

For the unitary representation of O(p,g) attached to the minimal nilpotent coadjoint orbit, we explicitly calculate its restriction to certain natural dual pairs in O(p, q). We furthermore show how the results are compatible with the orbit method. in particular when viewing the minimal nilpotent orbit as belonging to the limit set of semisimple orbits. 0 AcadCmie des SciencesElsevier, Paris

Gbomh-ie conforme et restrictions des reprkentations unitaires associkes aux orbites nilpotentes minimales

Nous donnons une description explicite de la restriction d certaines paires duales naturelles de la reprksentation unitaire de O(p, q) associie ci l’orbite co-adjointe nilpotente minimale. De plus, nous montrons que les rPsultats obtenus sont compatibles avec la me’thode des orbites, particulidrement lorsqu’on considkre I’orbite comme limite d’une famille bien dt?terminPe d’orbites semi-simples. 0 Acadkmie des Sciences/Elsevier, Paris

Version franqaise abrt!gke

Si G est un groupe de Lie, on dksigne par 2 son dual unitaire, c’est-h-dire l’ensemble des classes d’kquivalence de ses reprksentations un&taires irrkductibles. Soient G un groupe rCductif et G’ l’un de ses sous-groupes rkductifs. Si TT E G, la restriction 7~1~’ 21 G’ n’est en gCn&al pas irkductible.

Note prCsentCe par Michel DUFLO.

Nous nous proposons done d’obtenir, dans certains cas, une formule explicite pour la decomposition en irreductibles de cette restriction :

7rIQ N J G? @ mn (TWdT)

ou m,(r) E N U {co} et dp est une mesure borelienne sur 0.

Dans la suite on suppose que G = O(p: q) et G’ := G’, x GL = O(p’, q’) x O(p”; q”), avec p’ + p” = p, q’ + q” = q. Remarquons que G\ et Gk forment une paire duale reductive de G, en d’autres termes, ce sont deux sous-groupes reductifs, cornmutant l’un de l’autre, de G. Nous construisons les representations irreductibles du groupe O(p. q) (p > 2, q 2 2), wp,q et T:‘$,, ~Tip_‘~~,

assocites respectivement a l’orbite nilpotente minimale et aux orbites elliptiques minimales.

En fait, si go designe l’algebre de Lie de O(p, q). les representations T”+‘,: et ~5:~ sont associees respectivement aux orbites elliptiques Ad* (G)Xfi (i = 1, 2), avec X E T! + i (p + q) et X 2 0, fi, f2 E flgT, Ctant les formes lineaires definies par :

(fl!X) := -J-l&z9 (f2.X) := -v-Xp+q--l,p+q; x = (xij)l<i,j<p+q E go.

De plus. il est commode de definir TI-:‘-~ = 1 (resp. ~$4 = sgn) comme &ant la representation

’ 2

triviale (resp. << signe >>) de O(1,O) = O(1). Alors, si on p&e :

i

{AU+?: X>O} (P > 1, 4 # 0):

{A E z + 9 : x > f - 1)

A(p,q) := 8 (p > 1; q = O),

(P = 1, 4 # 0) ou (P = 0):

i-i, i, (P = 1, 4 = O),

les TT:~,, avec X E A(p, q), constituent l’ensemble des elements de la serie discrete pour l’hyperboloi’de reel Glhomogene X(p, q) := {(XT:, y) E FW : lx\2 - jyj2 = l} (11oir [3], [S]).

Maintenant, nous nous interessons au spectre discret de la restriction de la representation unipotente minimale wP%q de G (definie par exemple dans [l]) a la paire reductive duale G’ = G: x Gb, lorsque p + q E 2N p > 2, q > 2 et (p, q) # (2,2). On pose A’(p: q) := A(p, q) n {X E R : X > l}.

TH~OR~ZME A. - La restriction wJ’,~(G~ contient

c

tp TPh’ jg TP”‘P”

+,x -.A @ c

@ -lrP’%;f q ^+$l”

-1

XEA’(p’,q’)llA’(q”,p”) X;l’(q’,p’)rbi’(p”.q”) dam son spectre discret.

Si l’un des nombres p’, q’, p” ou q” est nul, alors la restriction wp,q\~ se decompose discretement en representations unitaires irreducibles de G’ comme il resulte d’un critere general se trouvant dans [4], Part II. Par suite on a :

THI?OR&ME B. - Supposons que p + q E 2N et que q” > 2. Alors on a :

Supposons p + q E 2N, p, q > 2 et p + q 2 8. Dans ces conditions, l’annulateur infinitesimal de la representation wp,q est l’ideal de Joseph (voir [l], [6]), si bien que la representation est associee a l’orbite nilpotente minimale C’mlin de G dans &i g;l, laquelle est de dimension 2(p + q - 3).

Branching laws

DEFINITION. - Soit M, une famille, parametree par v E W+, de sous-ensembles d’un espace topologique. Si M est une partie de cet espace topologique, on designe par M son adherence. Alors, on definit l’ensemble limite de la famille M,, v E W+, en posant :

liiM” := n u M,.

E>O E>V>O

Le resultat suivant donne la description de l’ensemble limite des orbites elliptiques 0, = Ad* (G)vfl, lorsqu’on s’approche du cone nilpotent :

TH-~OR~ME C. - Si p > 2 et q > 2, on a la d&composition en Ad*(G)-orbites :

oti 00 est une orbite nilpotente de dimension 2(p + q - 2) de G dans -8;.

1. Introduction

1.1. Let G be a reductive Lie group, and G’ be a reductive subgroup of G. We denote by 2 the unitary dual of G, the equivalence classes of irreducible unitary representations of G. Likewise ??

for G’. If 7r E G, then the restriction X/G/ is not necessarily irreducible. By a branching law, we mean an explicit irreducible decomposition formula:

s kI!

7rjp cv

G 4~bddr) (direct integral), (1)

where m,(r) E N ^U {m} and db is a Bore1 measure on 5;.

1.2. We denote by ~0 the Lie algebra of G. The orbit methbd due to Kirillov-Kostant in the unitary representation theory of Lie groups indicates that the coadjoint representation Ad* : G - GL(g;) often has a surprising intimate relation with the unitary dual G. It works perfectly for simply connected nilpotent Lie groups. For real reductive Lie groups G, known examples suggest that the set of coadjoint orbits &i&/G (with certain integral conditions) still gives a fairly good approximation of the unitary dual 6.

1.3. There have been a number of attempts to construct representations attached to nilpotent orbits.

Among all, the Segal-Shale-Weil representation (or the oscillator representation) of sI>(n, W), denoted

by m’, has been best studied, which is supposed to be attached to the minimal nilpotent orbit of sp(n, W). The restriction of w’ to a reductive dual pair G’ = G:G’, gives Howe’s correspondence (see VI).

The group S&n, W) is a split group of type C,, and analogous to w’, Kostant constructed a minimal representation of SO(n, n), a split group of type D,, and then Binegar-Zierau generalized it for SO(p, q) with p + q E 2N. This representation (precisely, of O(p, q), see Section 2) will be denoted by wplq.

1.4. Let G’ := GiG’, = O(p’,q’) x O(p”, q”) (p’ + p” = p, q’ + q” = q), be a subgroup of G = O(p, q). Our object of study is the branching law w*,*l~. We note that G’, and Gh form a mutually centralizing pair of subgroups in G.

A detailed proof will appear elsewhere. The first author expresses his sincere gratitude to Odense University for the warm hospitality.

2. Construction of representations of O(p, q)

2.1. In this section, we construct irreducible representations of the indefinite orthogonal group O(p; 4) (p, 4 2 2), denoted by ~+q and nFpx, x5$, which are supposed to be attached to the minimal nilpotent orbit, and minimal elliptic orbits, respectively.

2.2. Let IP.4 be a manifold IP’+q equipped with pseudo-Riemannian metric ds2 = dxy + . . . + d5; - dy; - . . ’ - dy,‘.

We define submanifolds of k!p,q by:

X(p, q) := {(x: y) E F-P,” : 1x12 - Iyy12 = l}, z := { (2: y) E wp.* : 1x1 = Iyl} \ {O},

M := {(X>Y) E wp,q : 1x1 = IyJ = l} N s-l x sq-I.

The indefinite orthogonal group G := O(p,.q) acts naturally on FP4, X(p, q), and E. The action is denoted by z H g . z (g E G, z E IVq). G also acts on 1Lf. In fact, the dilation action of

‘w; := {7- E W : r > 0} on Z commutes with that of G. The induced action of G on M E Z / W$

will be denoted by x H Lhx (z E M, h E G).

2.3. We start with the conformal construction of the “minimal unipotent” representation wP)q for p + q E 2N by applying ideas from conformal geometry to A4 = Sp-’ x Sq-1 and G = O(p, q) (see [9] and [lo]). We equip M with the pseudo-Riemannian metric induced from IP4. Then G acts conformally on M, and let A,, = ASP-l - As-1 - (9)’ + (9) 2 be the so-called Yamabe operator. We set v : B - W, (~>y) ^H 1x1, and define:

(wp,q(h-l)f)(z) := v(h. z)-vf(Lhz),

for h E G = O(p,q), z E M = S-l x P-l, f E V := {f E C”(SP-l x P-l) : inrf = O}.

Then (wP,q, V) is a representation of G = O(p: q) by conformal geometry. Moreover, (wP,q, V) is a non-zero irreducible representation of G if p + q E 2N. By comparing the construction of [l], the underlying (8, K)-module VK is unitarizable with the inner product

where D :=

J -A,,-1 + v, and dw is the standard measure on IV. We use the same notation wP,q to denote the irreducible unitary representation of G.

2.4. Next, we consider minimal elliptic coadjoint orbits of G = O(p, q). According to the normalization adopted in [5] and [7], we write $‘px for 7r~f,, and ~5:~ for ZAP?, these being the representations attached to the elliptic orbits Ad*(G)Xf, (a = 1, 2) for X E Z + i(p + 4) with X 2 0, where fi, f2 E &i& are defined by:

(fl- x, := -GX12: (f2; X) := -GAY~+~-~,~+~, for X = (Xij)l<i,j<p+q E go:

Branching laws

where go = o(p, 4). It is convenient to put 7ri”_, =

’ 2 1, the trivial representation of O(l,O) = O(1);

and VT’.’ + L

12 = sgn, the signature representation of 0( 1). Then, {xypA : X E A(p, 9)) (c Ocq)) is the totality of discrete series representations for X(p, 4) (see [3], [S]), where we put:

{XEZfF: X>O} (P > 1, Q # 01,

:X25-1} (p>l,q=O),

(p = 1, 4 # 0) or (p = 0):

(p = 1, q = 0).

3. Branching laws

In this section, we co?sider the discrete spectrum in the branching law of the minimal unipotent representation mp*q E G with respect to the reductive dual pair (G, G’) = (O(p, q), O(p’, q’) x O(p”,q”)), where p’ + p” = p (2 2), 4’ + q” = q (2 2), p + q E 2N, and (p:q) # (2,2). We set A’(p,q) I= A@. q)

n {A E R : x > 1).

THEOREM 3.1. - The restriction wiJp,q 1~’ contains

c

9 *y_‘f q *P”y $

c 6 ^TPf;ql q ^TP’f.q”

-1 -,A +,A

XE.-l’(p’,q’)nA’(q”,p”) AEA’(q’,p’)llA’(p”,q”)

in its discrete spectrum.

The idea of the proof is based on the conformal embedding X(P’> q’) x X(q”,p”) 4 lvl = s-l x sq-l together with the conformal construction of wJp.q in Section 2.

3.2. If one of p’, q’, y” or q” is zero, then the restriction c+‘,~(Q is decomposed discretely into irreducible representations of G’ = O(p’: q’) x O(p”! q”) by a general criterion in [4], Part II (see also [3]). Then, by a more delicate analysis, we can determine the full branching laws of z+Q]Q as follows:

THEOREM 3.2. - Let p + q E 2fW. If q” > 2 and q’ + q” = q, then

Remark 3.3. - 1) ~y:;~-, is isomorphic to the irreducible representation of O(q”) on the 2

spherical harmonics of degree e, that is, the representation space is 3-1’ = {f E Cm(SQ”-l) : Ass”-lf = -qe + q” - 2)f).

2) The formula in Theorem 3.2 is nothing but a K-type formula when q’ = 0.

4. Orbit methods

4.1. Suppose p f q E 2N, p, y > 2. The annihilator of the representation &xq is the Joseph ideal for p -t q > 8 (see [1], [6]). In this sense, a P,q is supposed to be attached to the unique minimal nilpotent coadjoint orbit, denoted by (?min (C fl&), whose dimension is 2(p + q - 3).

DEFINITION 4.2. - Let M, be a family of ^subsets, parametrized by u E iR+, of a topological space.

We denote by % the closure of a subset M. Then, the “limit set” is defined by:

iii; M” := n u M,.

E>O E>V>O

4.3. Here is the limit behavior as the elliptic orbits tend to the nilpotent orbits, setting U, = Ad*(G)vfl:

THEOREM 4.3. - Zf p 2 2 and q > 2, then we have the Ad*(G)-orbit decomposition:

liii (3” = UO U O,i, U (0).

where UO is a nilpotent orbit of dimension 2(p + q - 2).

4.4. The construction of the representations .rrP;py, attached to elliptic orbits Ad* (G)vfr (see Section 2.4) is built on the polarization (or, equivalently, the &stable parabolic subalgebra if we employ Zuckerman-Vogan’s construction) depending on the signature of v.

Let us consider the Dolbeault cohomology group with v = -1 where the polarization is kept the same as that of a positive v. It turns out that the Dolbeault cohomology group in the same degree (i.e. p - 2) is still non:zero in this special setting. We write l&,--l 7r”;$ for the underlying (8, K)-module. Then, the following theorem may be regarded as a quantization of Theorem 4.3.

THEOREM 4.4. - Suppose p + q E 2N, and p, q 2 2. Then we have:

in the Grothendieck group of (g, K)-modules.

4.5. Let G’ = O(p,q’) x O(q”) (q’ + q” = q, p + q E 2N).

THEOREM 4.5. - Let ^{pr,,,, : g* - g’*} be the projection dual to g’ q g. Then we have:

prg+gt(Umin) = Ad*(G’){Xfr + Xf2 : X > 0).

The branching law in Theorem 3.2 may be regarded as a quantization of Theorem 4.5.

References

[I] Binegar B., Zierau R., Unitarization of a singular representation of SO(p. (1). Commun. Math. Phys. 138 (1991) 245-258.

[2] Howe R.. Transcending classical invariant theory, J. Amer. Math. Sot. 2 (1989) 535-552.

[3] Kobayashi T., The restriction of .$(A) to reductive subgroups, Proc. Japan Acad. 69 (1993) 262-267; Part II. 71 (1995) 24-26.

[4] Kobayashi T., Discrete decomposability of the restriction of A,(X) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994) 181-205; Part II (to appear in Ann. Math.); Part III, Invent. Math. 131 (1998) 229-256.

[5] Kobayashi T., Harmonic analysis on homogeneous manifolds of reductive type and unitary representation theory, Translations. Series II. Selected Papers on Harmonic Analysis, Groups. and Invariants, Vol. 183, A.M.S., 1998. pp. 1-31.

(61 Kostant B.. The vaanishing scalar curvature and the minimal unitary representation of SO(4.4), In: Operator Algebras, Unitary Representations. Enveloping Algebras. and Invariant Theory, Connes A. et al. (Eds.), Progress in Math. 92.

Birkhauser. 1990, Boston, 85-124.

[7] Vogan D.. Jr., Unitary Representations of Reductive Lie Groups, Ann. Math. Stud. 118, Princeton University Press, 1987.

[8] Vogan D., Jr., Irreducibility of discrete series representations for semisimple symmetric spaces, Adv. Stud. Pure Math.

14 (1988) 191-221.

[9] Brsted B.. A note on the conformal quasi-invariance of the Laplacian on a pseudo-Riemannian manifold. Lett. Math.

Phys. 1 (1977) 183.

[lo] Brsted B., Conformally invariant differential equations and projective geometry, J. Funct. Anal. 44 (1981) l-23.