ToshiyukiKobayashi andBentØrsted II.Branchinglaws AnalysisontheminimalrepresentationofO ð p ; q Þ

http://www.elsevier.com/locate/aim

Analysis on the minimal representation of Oðp; qÞ II. Branching laws

Toshiyuki Kobayashi

and Bent Ørsted

aRIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan

bDepartment of Mathematics and Computer Science, SDU - Odense University, Campusvej 55, Odense M, DK-5230, Denmark

Received 5 November 2001; accepted 3 September 2002 Communicated by Bertram Kostant

Abstract

This is a second paper in a series devoted to the minimal unitary representation of Oðp;qÞ:

By explicit methods from conformal geometry of pseudo Riemannian manifolds, we find the branching law corresponding to restricting the minimal unitary representation to natural symmetric subgroups. In the case of purely discrete spectrum we obtain the full spectrum and give an explicit Parseval–Plancherel formula, and in the general case we construct an infinite discrete spectrum.

r2003 Published by Elsevier Science (USA).

MSC:22E45; 22E46; 53A30; 43A85

Keywords:Minimal unitary representation; Branching law; Analysis on hyperboloid

Contents

Introduction . . . 514

4. Criterion for discrete decomposable branching laws . . . 517

5. Minimal elliptic representations of Oðp;qÞ . . . 521

6. Conformal embedding of the hyperboloid . . . 526

Corresponding author.

E-mail addresses:toshi@kurims.kyoto-u.ac.jp (T. Kobayashi), orsted@imada.sdu.dk (B. Ørsted).

0001-8708/$ - see front matterr2003 Published by Elsevier Science (USA).

doi:10.1016/S0001-8708(03)00013-6

Introduction

This is the second in a series of papers devoted to the analysis of the minimal representation$^p;q of Oðp;qÞ: We refer to[15] for a general introduction; also the numbering of the sections is continued from that paper, and we shall refer back to sections there. However, the present paper may be read independently from[15], and its main object is to study the branching law for the minimal unitary representation

$^p;q from analytic and geometric point of view. Namely, we shall find by explicit means, coming from conformal geometry, the restriction of$^p;q with respect to the symmetric pair

ðG;G⁰Þ ¼ ðOðp;qÞ;Oðp⁰;q⁰Þ Oðp⁰⁰;q⁰⁰ÞÞ:

If one of the factors inG⁰is compact, then the spectrum is discrete (see Theorem 4.2 also for an opposite implication), and we find the explicit branching law; when both factors are non-compact, there will still (generically) be an infinite discrete spectrum, which we also construct (conjecturally almost all of it; see Section 9.8). We shall see that the (algebraic) situation is similar to the theta-correspondence, where the metaplectic representation is restricted to analogous subgroups.

Let us here state the main results in a little more precise form, referring to Sections 8 and 9 for further notation and details.

Theorem A (The branching law for Oðp;qÞkOðp;q⁰Þ Oðq⁰⁰Þ; see Theorem 7.1). If q⁰⁰X1 and q⁰þq⁰⁰¼q; then the twisted pull-back fFF₁₁ of the local conformal map F1

between spheres and hyperboloids gives an explicit irreducible decomposition of the unitary representation$^p;q when restricted toOðp;q⁰Þ Oðq⁰⁰Þ:

ðFg1Þ

ðF1Þ:$^p;qj_Oðp;q0ÞOðq⁰⁰Þ!^BX^N

l¼0

" p^p;q0

þ;lþq⁰⁰

212H^lðR^q00Þ:

The representations appearing in the decompositions are in addition to usual spherical harmonics H^lðR^qÞ for compact orthogonal groups OðqÞ; also the representations p^p;q_þ;l for non-compact orthogonal groups Oðp;qÞ: The latter ones

7. Explicit branching formula (discrete decomposable case) . . . 530

8. Inner product on$^p;qand the Parseval–Plancherel formula . . . 534

9. Construction of discrete spectra in the branching laws . . . 542

Acknowledgments . . . 549

References . . . 549

Further Reading . . . 550

may be thought of as discrete series representations on hyperboloids Xðp;qÞ:¼ fx¼ ðx⁰;x⁰⁰ÞAR^pþq: jx⁰j² jx⁰⁰j²¼1g

for l40 or their analytic continuation for lp0; they may be also thought of as cohomologically induced representations from characters of certain y-stable parabolic subalgebras. The fact that they occur in this branching law gives a different proof of the unitarizability of these modulesp^p;q_þ;lforl41;once we know

$^p;qis unitarizable (cf. Part I, Theorem 3.6.1). It might be interesting to remark that the unitarizability forlo0 (especially,l¼ ¹₂in our setting) does not follow from a general unitarizability theorem on Zuckerman–Vogan’s derived functor modules [20], neither from a general theory of harmonic analysis on semisimple symmetric spaces. For another viewpoint from the orbit method, we refer to[11].

Our intertwining operator ðFðFg11ÞÞ in Theorem A is derived from a conformal change of coordinate (see Section 6 for its explanation) and is explicitly written.

Therefore, it makes sense to ask also about the relation of unitary inner products between the left- and the right-hand side in the branching formula. Here is an answer (see Theorem 8.6): We normalize the inner productjj jj_p^p;q

þ;l (see (8.4.2)) such that forl40;

jjfjj²_p^p;q

þ;l¼ljjfjj²_L2ðXðp;qÞÞ for any fAðp^p;q_þ;lÞ_K:

Theorem B (The Parseval–Plancherel formula for Oðp;qÞkOðp;q⁰Þ Oðq⁰⁰Þ).

(1) If we develop FAKerD*_M as F ¼PN

l F_l^ð1ÞF_l^ð2Þ according to the irreducible decomposition in TheoremA,then we have

jjFjj²_$p;q ¼X^N

l¼0

jjF_l^ð1Þjj²_pp;q0 þ;lþq00

21

jj F_l^ð2Þjj²_L2ðS^{q00 1}Þ:

(2)In particular,if q⁰⁰X3;then all ofp^p;q0

þ;lþ^q₂⁰⁰1are discrete series for the hyperboloid Xðp;q⁰Þand the above formula amounts to

jjFjj²_$p;q ¼X^N

l¼0

lþq⁰⁰ 2 1

jjF_l^ð1Þjj²_L2ðXðp;q⁰ÞÞjj F_l^ð2Þjj²_L2ðS^{q00 1}Þ:

The formula may be also regarded as an explicit unitarization of the minimal representation$^p;q on the ‘‘hyperbolic space model’’ by means of the right side (for an abstract unitarization of$^p;q;it suffices to choose a single pairðq⁰;q⁰⁰Þ). We note that the formula was previously known in the case whereðq⁰;q⁰⁰Þ ¼ ð0;qÞ(namely, when each summand in the right side is finite-dimensional) by Kostant, Binegar–

Zierau by a different approach [1,17]. The formula is new and seems to be particularly interesting even in the special case q⁰⁰¼1; where the minimal

representation $^p;q splits into two irreducible summands when restricted to Oðp;q1Þ Oð1Þ:

In Theorem 9.1, we consider a more general setting and prove:

Theorem C (Discrete spectrum in the restriction Oðp;qÞkOðp⁰;q⁰Þ Oðp⁰⁰;q⁰⁰Þ). The twisted pull-back of the locally conformal diffeomorphism also constructs

X_"

lAA⁰ðp⁰;q⁰Þ-A⁰ðq⁰⁰;p⁰⁰Þ

p^p_þ;l^0;q02p^p_;l^00;q00" X_"

lAA⁰ðq⁰;p⁰Þ-A⁰ðp⁰⁰;q⁰⁰Þ

p^p_;l^0;q02p^p_þ;l^00;q00

as a discrete spectrum in the branching law for the non-compact case.

Even in the special caseðp⁰⁰;q⁰⁰Þ ¼ ð0;1Þ; our branching formula includes a new and mysterious construction of the minimal representation on the hyperboloid as below (see Corollary 7.2.1): LetW^p;rbe the set ofK-finite vectorsðK¼OðpÞ OðrÞÞ of the kernel of the Yamabe operator

KerD*_Xðp;rÞ¼ ffAC^NðXðp;rÞÞ: D_Xðp;rÞf ¼¹₄ðpþr1Þðpþr3Þfg;

on which the isometry group Oðp;rÞ and the Lie algebra of the conformal group Oðp;rþ1Þact. The following proposition is a consequence of Theorem 7.2.2 by an elementary linear algebra.

Proposition D. Let m43be odd. There is a long exact sequence 0-W^1;m1!^j1 W^2;m2!^j2W^3;m3!^j3?^jm2! W^m1;1jm1!0

such thatKerj_p is isomorphic toð$^p;qÞ_K for anyðp;qÞsuch that pþq¼mþ1:

We note that each representation space W^p;q1 is realized on a different space Xðp;q1Þwhose isometry group Oðp;q1Þvaries according top ð1pppmÞ:So, one may expect that only the intersections of adjacent groups can act (infinitesimally) on Kerj_p:Nevertheless, a larger group Oðp;qÞcan act on a suitable completion of Kerj_p; giving rise to another construction of the minimal representation on the hyperboloid Xðp;q1Þ ¼Oðp;q1Þ=Oðp1;q1Þ! We note that Kerj_p is roughly half the kernel of the Yamabe operator on the hyperboloid (see Section 7.2 for details).

We briefly indicate the contents of the paper: In Section 4 we recall the relevant facts about discretely decomposable restrictions from[9,10], and apply the criteria to our present situation. In particular, we calculate the associated variety of$^p;qas well as its asymptotic K-support introduced by Kashiwara–Vergne. Theorem 4.2 and Corollary 4.3 clarify the reason why we start with the subgroup G⁰¼Oðp;q⁰Þ Oðq⁰⁰Þ(i.e.p⁰⁰¼0). Section 5 contains the identification of the representationsp^p;q_þ;l andp^p;q_;l of Oðp;qÞin several ways, namely as: derived functor modules, Dolbeault

cohomologies, eigenspaces on hyperboloids, and quotients or subrepresentations of parabolically induced modules. In Section 6 we give the main construction of embedding conformally a direct product of hyperboloids into a product of spheres;

this gives rise to a canonical intertwining operator between solutions to the so-called Yamabe equation, studied in connection with conformal differential geometry, on conformally related spaces. Applying this principle in Section 7 we obtain the branching law in the case where one factor in G⁰ is compact, and in particular when one factor is just Oð1Þ: In this case we have Corollary 7.2.1, stating that$^p;qrestricted to Oðp;q1Þis the direct sum of two representations, realized in even, respectively, odd functions on the hyperboloid for Oðp;q1Þ:Note here the analogy with the metaplectic representation. Also note here Theorem 7.2.2, which gives a mysterious extention of $^p;q by $^pþ1;q1—both inside the space of solutions to the Yamabe equation on the hyperboloid Xðp;q1Þ ¼Oðp;q 1Þ=Oðp1;q1Þ:We also point out that the representationsp^p;q_þ;l forl¼0;¹₂are rather exceptional; they are unitary, but outside the usual ‘‘fair range’’ for derived functor modules, see the remarks in Section 8.4. Section 8 contains a proof of Theorem 3.9.1 of[15]on the spectra of the Knapp–Stein intertwining operators and gives the explicit Parseval–Plancherel formulas for the branching laws. Finally, in Section 9 we use certain Sobolev estimates to construct infinitely many discrete spectra when both factors inG⁰are non-compact. We also conjecture the form of the full discrete spectrum (true in the case of a compact factor). It should be interesting to calculate the full Parseval–Plancherel formula in the case of both discrete and continuous spectrum.

4. Criterion for discrete decomposable branching laws

4.1. Our object of study is the discrete spectra of the branching law of the restriction

$^p;q with respect to a symmetric pairðG;G⁰Þ ¼ ðOðp;qÞ;Oðp⁰;q⁰Þ Oðp⁰⁰;q⁰⁰ÞÞ: The aim of this section is to give a necessary and sufficient condition onp⁰;q⁰;p⁰⁰andq⁰⁰ for the branching law to be discretely decomposable.

We start with general notation. LetGbe a linear reductive Lie group, andG⁰ its subgroup which is reductive inG:We take a maximal compact subgroupKofGsuch thatK⁰:¼K-G⁰is also a maximal compact subgroup. Letg₀¼k0þp₀be a Cartan decomposition, and g¼kþp its complexification. Accordingly, we have a direct decompositiong¼kþp of the dual spaces.

LetpAG:ˆ We say that the restrictionpj_G0 isG⁰-admissibleifpj_G0 splits into a direct Hilbert sum of irreducible unitary representations ofG⁰with each multiplicity finite (see [7]). As an algebraic analogue of this notion, we say the underlying ðg;KÞ- modulepKisdiscretely decomposable as aðg⁰;K⁰Þ-module, ifpK is decomposed into an algebraic direct sum of irreducibleðg⁰;K⁰Þ-modules (see[10]). We note that if the restriction pj_K0 is K⁰-admissible, then the restriction pj_G0 is also G⁰-admissible

[7, Theorem 1.2]and the underlying ðg;KÞ-modulepK is discretely decomposable (see [10, Proposition 1.6]). Here are criteria for K⁰-admissibility and discrete decomposability:

Fact 4.1 (see Kobayashi[9, Theorem 2.9] for (1);[10, Corollary 3.4]for (2)).

(1)IfASKðpÞ-AdðKÞðk⁰Þ^>¼0;then pis K⁰-admissible and also G⁰-admissible.

(2) If pK is discretely decomposable as a ðg⁰;K⁰Þ-module, then pr_g-g0ðVgðpKÞÞCN_g0:

Here, AS_KðpÞis the asymptotic cone of

Supp_KðpÞ:¼ fhighest weight of tAKKc00: ½pj_K0: ta0g;

whereK₀ is the identity component of K;andðk⁰Þ^>Ck is the annihilator ofk⁰:Let N_p ðCpÞbe the nilpotent cone forp:VgðpKÞdenotes the associated variety ofpK; which is an AdðKCÞ-invariant closed subset ofN_p:We write the projection pr_p-p⁰ : p-p⁰ dual to the inclusionp⁰+p:

4.2. Let us consider our setting where p¼$^p;q and ðG;G⁰Þ ¼ ðOðp;qÞ;Oðp⁰;q⁰Þ Oðp⁰⁰;q⁰⁰ÞÞ:

Theorem 4.2. Suppose p⁰þp⁰⁰¼p ðX2Þ;q⁰þq⁰⁰¼q ðX2Þand pþqA2N:Then the following three conditions on p⁰;q⁰;p⁰⁰;q⁰⁰are equivalent:

(i) $^p;q is K⁰-admissible;

(ii) $^p;q_K is discretely decomposable as aðg⁰;K⁰Þ-module;

(iii) minðp⁰;q⁰;p⁰⁰;q⁰⁰Þ ¼0:

Implication (i))(ii) holds by a general theory as we explained[10, Proposition 1.6]; (ii)) (iii) will be proved in Section 4.4, and (iii) )(i) in Section 4.5, by an explicit computation of the asymptotic cone AS_Kð$^p;qÞand the associated variety Vgð$^p;q_K Þwhich are used in Fact 4.1.

Remark. Analogous results to the equivalence (i)3(ii) in Theorem 4.2 were first proved in[10], Theorem 4.2 in the setting whereðG;G⁰Þis any reductive symmetric pair and the representation is anyAqðlÞmodule in the sense of Zuckerman–Vogan, which may be regarded as ‘‘representations attached to elliptic orbits’’. We note that our representations$^p;q are supposed to be attached to nilpotent orbits. We refer [13], Conjecture A to relevant topics.

4.3. The following corollary is a direct consequence of Theorem 4.2, which will be an algebraic background for the proof of the explicit branching law (Theorem 7.1).

Corollary 4.3. Suppose thatminðp⁰;q⁰;p⁰⁰;q⁰⁰Þ ¼0:

(1)The restriction of the unitary representation$^p;qj_G0 is also G⁰-admissible.

(2)The space of K⁰-finite vectors$^p;q_K0 coincides with that of K-finite vectors$^p;q_K : Proof. See[7, Theorem 1.2] for (1) and[10, Proposition 1.6]for (2). &

A geometric counterpart of Corollary 4.3(2) is reflected as the removal of singularities of matrix coefficients for the discrete spectra in the analysis that we study in Section 6; namely, any analytic function defined on an open subsetMþ(see Section 6 for notation) of M which is a K⁰-finite vector of a discrete spectrum, extends analytically on M if p⁰⁰¼0: The reason for this is not only the decay of matrix coefficients but a matching condition of the leading terms fort-7N:This is not the case for minðp⁰;q⁰;p⁰⁰;q⁰⁰Þ40 (see Section 9).

4.4. Proof of (ii))(iii) in Theorem 4.2.

We identify p with p via the Killing form, which is in turn identified with Mðp;q;CÞby

Mðp;q;CÞ !^Bp; X/ O X

tX O

! :

Then the nilpotent coneN_p corresponds to the following variety:

fXAMðp;q;CÞ: both X^tX and ^tXX are nilpotent matricesg: ð4:4:1Þ We put

M_0;0ðp;q;CÞ:¼ fXAMðp;q;CÞ: X^tX¼O; ^tXX ¼Og:

Then M_0;0ðp;q;CÞ\fOg is the unique K_CCOðp;CÞ Oðq;CÞ-orbit of dimension pþq3: The associated variety Vgð$^p;q_K Þ of $^p;q is of dimension pþq3;

which follows easily from the K-type formula of $^p;q (see [15, Theorem 3.6.1]).

Thus, we have proved:

Lemma 4.4. The associated variety Vgð$^p;q_K Þequals M0;0ðp;q;CÞ:

The projection pr_p-p⁰: p-p⁰ is identified with the map pr_p-p0:Mðp;q;CÞ-Mðp⁰;q⁰;CÞ"Mðp⁰⁰;q⁰⁰;CÞ; X1 X2

X₃ X₄

!

/ðX1;X₄Þ:

Supposep⁰p⁰⁰q⁰q⁰⁰a0: If we take X:¼E1;1Ep⁰þ1;q⁰þ1þ ffiffiffiffiffiffiffi

p1

Ep⁰þ1;1þ ffiffiffiffiffiffiffi p1

E1;q⁰þ1AM0;0ðp;q;CÞ;

then pr_p-p⁰ðXÞ ¼ ðE1;1;Ep⁰þ1;q⁰þ1Þ: But E1;1eN_oðp0;q⁰Þ and Ep⁰þ1;q⁰þ1eN_oðp00;q⁰⁰Þ: Thus, pr_g-g⁰ðXÞeN_g0: It follows from Fact 4.1(2) that $^p;q_K is not discrete decomposable as aðg⁰;K⁰Þ-module. Hence (ii))(iii) in Theorem 4.2 is proved. &

4.5. Proof of (iii))(i) in Theorem 4.2.

We take an orthogonal complementary subspacek⁰⁰₀ ofk⁰₀ink₀CoðpÞ þoðqÞ:Lett^c₀ be a Cartan subalgebra ofk0 such thatt⁰⁰₀:¼t^c₀-k⁰⁰₀ is a maximal abelian subspace in k⁰⁰₀:We choose a positive systemD^þðk;t^cÞwhich is compatible with a positive system of the restricted root systemSðk;t⁰⁰Þ:Then we can find a basisffi: 1pip½^p₂ þ ½^q₂gon

ffiffiffiffiffiffiffi p1

t₀ such that a positive root system ofkis given by W^þðk;t^cÞ ¼ f_i7f_j: 1piojp p

2 h i

n o

, f_i7f_j: p 2

h iþ1piojp p 2 h iþ q

2 h i

n o

, f_l: 1plp p 2 h i

n o

ðp: oddÞ

, f_l: p 2

h iþ1plp p 2 h iþ q

2 h i

n o

ðq: oddÞ

;

and such that

ffiffiffiffiffiffiffi p1

ðt⁰⁰₀Þ¼^minðpX^0;p00Þ

j¼1

Rfiþ^minðqX^0;q00Þ

j¼1

Rf_½p

2þj ð4:5:1Þ

if we regardðt⁰⁰₀Þ as a subspace ofðt^c₀Þ by the Killing form.

Supposep⁰q⁰p⁰⁰q⁰⁰¼0:Without loss of generality we may and do assumep⁰⁰¼0;

namely,G⁰¼Oðp;q⁰Þ Oðq⁰⁰Þwithq⁰þq⁰⁰¼q:

Let us first consider the case pa2: Then the irreducible OðpÞ-representation H^aðR^pÞ remains irreducible when restricted to SOðpÞ: The corresponding highest weight is given byaf1:It follows from the K-type formula of$^p;q (Theorem 3.6.1) that

Supp_Kð$^p;qÞ ¼ af1þbf_½^p

2þ1: a;bAN;aþp

2¼bþq 2

:

Therefore, we have proved

ASKð$^p;qÞ ¼Rþðf1þf_½p=2þ1Þ: ð4:5:2Þ Then AS_Kð$^p;q_Þ-p^{ffiffiffiffiffiffiffi}1

ðt⁰⁰₀Þ¼ f0gfrom (4.5.1) and (4.5.2), which implies ASKð$^p;qÞ- ffiffiffiffiffiffiffi

p1

AdðKÞðk⁰₀Þ^>¼ f0g

because ðt⁰⁰₀Þ meets any AdðKÞ-orbit through ðk⁰₀Þ^>: Therefore, the restriction

$^p;qj_K⁰ isK⁰-admissibleby Fact 4.1(1).

Ifp¼2;then$^p;qsplits into two representations (see Remark 3.7.3), say$^2;q_þ and

$^2;q ;when restricted to the connected component SO0ð2;qÞ:Likewise,H^aðR^pÞis a direct sum of two one-dimensional representations when restricted to SOð2ÞifaX1:

Then we have

ASKð$^2;q₇Þ ¼Rþð7f1þfpþ1Þ:

Applying Fact 4.1(1) to the identity components ðG0;G₀⁰Þ of groups ðG;G⁰Þ; we conclude that the restriction$^p;q₇j_K⁰

0isK₀⁰-admissible. Hence the restriction$^p;qj_K0 is alsoK⁰-admissible. Thus, (iii) )(i) in Theorem 4.2 is proved.

Now the proof of Theorem 4.2 is completed. &

5. Minimal elliptic representations of Oðp;qÞ

5.1. In this section, we introduce a family of irreducible representations of G¼ Oðp;qÞ; denoted byp^p;q_þ;l p^p;q_;l; for lAA0ðp;qÞ; in three different realizations. These representations are supposed to be attached to minimal elliptic orbits, forl40 in the sense of the Kirillov–Kostant orbit method. Here, we set

A0ðp;qÞ:¼

flAZþ^pþq₂ : l41g ðp41;qa0Þ;

flAZþ^pþq₂ : lX^p

21g ðp41;q¼0Þ;

| ðp¼1;qa0Þ or ðp¼0Þ;

f¹₂;¹₂g ðp¼1;q¼0Þ:

8>

>>

><

>>

>>

:

ð5:1:1Þ

It seems natural to include the parameterl¼0;¹₂ in the definition ofA0ðp;qÞas above, althoughl¼ ¹₂ is outside the weakly fair range parameter in the sense of Vogan [22]. Cohomologically induced representations for l¼ ¹₂ and 1 will be discussed in details in a subsequent paper. In particular, the case l¼ 1 is of importance in another geometric construction of the minimal representation via Dolbeault cohomology groups (see Part I, Introduction, Theorem B(4)).

5.2. LetR^p;qbe the Euclidean spaceR^pþqequipped with the flat pseudo-Riemannian metric:

g_R^p;q ¼dv²₀þ?þdv²_p1dv²_p?dv²_pþq1:

We define a hyperboloid by

Xðp;qÞ:¼ fðx;yÞAR^p;q: jxj² jyj²¼1g:

We note Xðp;0ÞCS^p1 and Xð0;qÞ ¼|: If p¼1; then Xðp;qÞ has two connected components. The group G acts transitively on Xðp;qÞ with isotropy subgroup Oðp1;qÞat

x^o:¼^tð1;0;y;0Þ: ð5:2:1Þ ThusXðp;qÞis realized as a homogeneous manifold:

Xðp;qÞCOðp;qÞ=Oðp1;qÞ:

We induce a pseudo-Riemannian metricg_Xðp;qÞonXðp;qÞfromR^p;q(see[15, Section 3.2]), and write DXðp;qÞ for the Laplace–Beltrami operator on Xðp;qÞ: As in [15, Example 2.2], the Yamabe operator is given by

D*_Xðp;qÞ¼D_Xðp;qÞ¹₄ðpþq1Þðpþq3Þ: ð5:2:2Þ ForlAC;we set

C_l^NðXðp;qÞÞ:¼ fAC^NðXðp;qÞÞ: D_Xðp;qÞf ¼ l ²þ¹₄ðpþq2Þ²

n fo

¼ fAC^NðXðp;qÞÞ: D*_Xðp;qÞf ¼ l ²þ¹₄

f

: ð5:2:3Þ

Furthermore, fore¼7; we write

C_l;e^NðXðp;qÞÞ:¼ ffAC_l^NðXðp;qÞÞ: fðzÞ ¼efðzÞ; zAXðp;qÞg:

Then we have a direct sum decomposition

C_l^NðXðp;qÞÞ ¼C_l;þ^NðXðp;qÞÞ þC_l;^NðXðp;qÞÞ ð5:2:4Þ and each space is invariant under left translations of the isometry groupGbecauseG commutes with DXðp;qÞ: With the notation in Section 3.5, we note if q¼0; then C^N

l;sgnð1Þ^kðXðp;0ÞÞ is finite-dimensional and isomorphic to the space of spherical harmonics:

H^kðR^pÞCC_l^NðXðp;0ÞÞ ¼C_l;sgnð1Þ^N kðXðp;0ÞÞ k:¼lþp2 2

:

5.3. LetG¼Oðp;qÞwherep;qX1 and letybe the Cartan involution corresponding toK¼OðpÞ OðqÞ:We extend a Cartan subalgebrat^c₀ofk0(given in Section 4.5) to that of g₀; denoted by h^c₀: If both p and q are odd, then dimh^c₀¼dimt^c₀þ1;

otherwiseh^c₀¼t^c₀:The complexification ofh^c₀ is denoted byh^c:

We can take a basis ffi: 1pip½^pþq₂ g of ðh^cÞ (see Section 4.5; by a little abuse of notation if both p and q are odd) such that the root system of g is

given by

Wðg;h^cÞ ¼ 7ðf_i7f_jÞ: 1piojp pþq 2

h i

n o

, 7f_l: 1plp pþq 2

h i

n o

ðpþq: oddÞ

:

Let fHigCh^c be the dual basis for ffigCðh^cÞ: Set t:¼CH₁ ðCt^cCh^cÞ: Then the centralizer L of t in G is isomorphic to SOð2Þ Oðp2;qÞ: Let q¼lþu be a y-stable parabolic subalgebra ofg with nilpotent radicalugiven by

Wðu;h^cÞ:¼ f17fj: 2pjp pþq 2

h i

n o

,ðff1g ðpþq: oddÞÞ;

and with a Levi partl¼l₀#Cgiven by

l0LieðLÞCoð2Þ þoðp2;qÞ:

Any character of the Lie algebral₀(or any complex character ofl) is determined by its restriction toh^c₀:So, we shall writeC_nfor the character of the Lie algebral₀whose restriction toh^cisnAðh^cÞ:With this notation, the character ofLacting on4^dimuu is written asC_2rðuÞwhere

rðuÞ:¼ pþq 2 1

f₁: ð5:3:1Þ

The homogeneous manifold G=L carries a G-invariant complex structure with canonical bundle4^topTG=LCGLC_2rðuÞ:As an algebraic analogue of a Dolbeault cohomology of aG-equivariant holomorphic vector bundle over a complex manifold G=L;Zuckerman introduced the cohomological parabolic inductionR^j_q ðR^g_qÞ^jðjANÞ;

which is a covariant functor from the category of metaplecticðl;ðL-KÞ^BÞ-modules to that ofðg;KÞ-modules. Here,L˜ is a metaplectic covering ofLdefined by the character of L acting on 4^dimuuCC_2rðuÞ: In this paper, we follow the normalization in [21, Definition 6.20]which is different from the one in[19]by a ‘r-shift’.

The character Clf1 ofl0 lifts to a metaplecticðl;ðL-KÞ^BÞ-module if and only if lAZþ^pþq₂ :In particular, we can defineðg;KÞ-modulesR^j_qðClf1ÞforlAA0ðp;qÞ:The ZðgÞ-infinitesimal character ofR^j_qðClf1Þis given by

l;pþq

2 2;pþq

2 3;y;pþq

2 pþq 2

h i

Aðh^cÞ

in the Harish-Chandra parametrization if it is non-zero. In the sense of Vogan[22], we have

C_lf1 is in the good range 3 l4pþq 2 2;

Clf1 is in the weakly fair range 3 lX0:

We note thatR^j_qðClf1Þ ¼0 ifjap2 and iflAA0ðp;qÞ:This follows from a general result in[20]forlX0;and[6]forl¼ ¹₂:

5.4. ForbAZ;we define an algebraic direct sum ofK¼OðpÞ OðqÞ-modules by XðK:bÞ XðOðpÞ OðqÞ:bÞ:¼

"

m;nAN mnXb mnbmod 2

H^mðR^pÞ2HⁿðR^qÞ: ð5:4:1Þ

ForlAA0ðp;qÞ;we put

bbðl;p;qÞ:¼lp 2þq

2þ1AZ; ð5:4:2Þ

eeðl;p;qÞ:¼ ð1Þ^b: ð5:4:3Þ We define the line bundleLn overG=Lby the characternf1 ofL(see Section 5.3).

Here is a summary for different realizations of the representationp^p;q_þ;l: Fact 5.4. Let p;qAN ðp41Þ:

(1)For anylAA₀ðp;qÞ;each of the following5conditions defines uniquely aðg;KÞ- module, which are mutually isomorphic. We shall denote it by ðp^p;q_þ;lÞ_K: The ðg;KÞ- moduleðp^p;q_þ;lÞ_K is non-zero and irreducible.

(i) A subrepresentation of the degenerate principal representation Ind^G_Pmaxðe#C_lÞ (see Section3.7)with K-typeXðK:bÞ:

ðiÞ⁰ A quotient ofInd^G_Pmaxðe#ClÞwith K-typeXðK:bÞ:

(ii) A subrepresentation of C_l^NðXðp;qÞÞ_K with K-type XðK:bÞ:

(iii) The underlying ðg;KÞ-module of the Dolbeault cohomology group H_@^p2_% ðG=L;L

ðlþpþq2 2 ÞÞ_K:

ðiiiÞ⁰ The Zuckerman–Vogan derived functor module R^p2_q ðClf1Þ:

(2) In the realization of (ii), if fAðp^p;q_þ;lÞ_K; then there exists an analytic function aAC^NðS^p1S^q1Þsuch that

fðocosht;ZsinhtÞ ¼aðo;ZÞe^ðlþrÞtð1þte^2tOð1ÞÞ as t-N:

Here,we putr¼^pþq2₂ :

For details, we refer, for example, to[5]for (i) andðiÞ⁰;to[18]for (ii) and also for a relation with (i) (under some parity assumption on eigenspaces); to[7, Section 6](see

also[6]) forðiiiÞ⁰3(ii); and to[23]for (iii)3ðiiiÞ⁰:The second statement follows from a general theory of the boundary value problem with regular singularities;

or also follows from a classical asymptotic formula of hypergeometric functions (see (8.3.1)) in our specific setting.

Remark. (1) By definition, (i) andðiÞ⁰make sense forp41 andq40;and others for p41 and qX0:

(2) Each of realization (i), ðiÞ⁰; (ii), and (iii) also gives a globalization of p^p;q_þ;l; namely, a continuous representation ofGon a topological vector space. Because all ofðp^p;q_þ;lÞ_K ðlAA₀ðp;qÞÞare unitarizable we may and do take the globalizationp^p;q_þ;lto be the unitary representation ofG:

(3) Ifl40 andlAA0ðp;qÞ;then the realization (ii) of p^p;q_þ;l gives a discrete series representation forXðp;qÞ:Conversely,

fp^p;q_þ;l: lAA₀ðp;qÞ;l40g exhausts the set of discrete series representations forXðp;qÞ:

Ifðp;qÞ ¼ ð1;0Þ;then Oðp;qÞCOð1Þand it is convenient to define representations of Oð1Þby

p^1;0_þ;l¼

1 ðl¼ ¹₂Þ;

sgn ðl¼¹₂Þ;

0 ðotherwiseÞ:

8>

<

>:

As we defined p^p;q_þ;l in Fact 5.4, we can also define an irreducible unitary representation, denoted by p^p;q_;l; for lAA₀ðq;pÞ such that the underlying ðg;KÞ- module has the followingK-type:

"

m;nAN mnplþq 2p

21

mnlþq

2p 21 mod 2

H^mðR^pÞ2HⁿðR^qÞ:

Similarly top^p;q_þ;l;the representationsp^p;q_;lare realized in function spaces on another hyperboloid Oðp;qÞ=Oðp;q1Þ:

In order to understand the notation here, we remark:

(i) p^p;q_;lA^Oðp;

d

d

(ii) p^p;0_þ;lCH^kðR^pÞ;wherek¼l^p2₂ andpX1;kAN:

5.5. The casel¼7¹₂ is delicate, which happens whenpþqA2Nþ1:

First, we assume pþqA2Nþ1: By using the equivalent realizations of p^p;q_þ;l in Fact 5.4 and by the classification of the composition series of the most degenerate principal series representation Ind^G_Pmaxðe#C_lÞ (see[5]), we have non-splitting short exact sequences ofðg;KÞ-modules:

0-ðp^p;q

;1 2

Þ_K-Ind^G_Pmaxðð1Þ

pqþ1

2 #C

1

2Þ-ðp^p;q

þ;1 2

Þ_K-0; ð5:5:1Þ

0-ðp^p;q

þ;1 2

Þ_K-Ind^G_P^maxðð1Þ

pq1

2 #C1

2Þ-ðp^p;q

;1 2

Þ_K-0: ð5:5:2Þ Becausep^p;q_þ;l ðlAA0ðp;qÞÞis self-dual, the dualðg;KÞ-modules of (5.5.1) and (5.5.2) give the following non-splitting short exact sequences ofðg;KÞ-modules:

0-ðp^p;q

þ;1 2

Þ_K-Ind^G_Pmaxðð1Þ

pqþ1 2 #C₁

2Þ-ðp^p;q

;1 2

Þ_K-0; ð5:5:3Þ

0-ðp^p;q

;1 2

Þ_K-Ind^G_P^maxðð1Þ

pq1

2 #C1

2Þ-ðp^p;q

þ;1 2

Þ_K-0: ð5:5:4Þ Next, we assumepþqA2N:Then, $^p;q is realized as a subrepresentation of some degenerate principal series (see [15, Lemma 3.7.2]). More precisely, we have non- splitting short exact sequences ofðg;KÞ-modules

0-$^p;q_{K -}Ind^G_Pmaxðð1Þ

pq

2 #C1Þ-ððp^p;q;1Þ_K"ðp^p;qþ;1Þ_KÞ-0; ð5:5:5Þ

0-ððp^p;q_;1Þ_K"ðp^p;q_þ;1Þ_KÞ-Ind^G_Pmaxðð1Þ

pq

2 #C₁Þ-$^p;q_{K -}0; ð5:5:6Þ and an isomorphism ofðg;KÞ-modules:

Ind^G_Pmaxðð1Þ^pqþ22 #C0ÞCðp^p;q_;0Þ_K"ðp^p;q_þ;0Þ_K: ð5:5:7Þ These results will be used in another realization of the unipotent representation$^p;q; namely, as a submodule of the Dolbeault cohomology group in a subsequent paper (cf. Part I, Introduction, Theorem B(4)).

6. Conformal embedding of the hyperboloid

This section prepares the geometric setup which will be used in Sections 7 and 9 for the branching problem of$^p;qj_G0:Throughout this section, we shall use the following

notation:

jxj² :¼ jx⁰j²þ jx⁰⁰j²¼X^p0

i¼1

ðx⁰_iÞ²þX^p00

j¼1

ðx⁰⁰_jÞ²; for x:¼ ðx⁰;x⁰⁰ÞAR^p0þp00 ¼R^p;

jyj²:¼ jy⁰j²þ jy⁰⁰j²¼X^q0

i¼1

ðy⁰_iÞ²þX^q00

j¼1

ðy⁰⁰_jÞ²; for y:¼ ðy⁰;y⁰⁰ÞAR^q0þq00 ¼R^q:

6.1. We define two open subsets ofR^pþq by

R^p_þ^{0þp00;q0þq00}:¼ fðx;yÞ ¼ ððx⁰;x⁰⁰Þ;ðy⁰;y⁰⁰ÞÞAR^{p0þp00;q0þq00}: jx⁰j4jy⁰jg;

R^{p0þp00;q0þq00} :¼ fðx;yÞ ¼ ððx⁰;x⁰⁰Þ;ðy⁰;y⁰⁰ÞÞAR^{p0þp00;q0þq00}: jx⁰jojy⁰jg:

Then the disjoint union R^p_þ^{0þp00;q0þq00},R^{p0þp00;q0þq00} is open dense in R^pþq: Let us consider the intersection of R^p₇^{0þp00;q0þq00} with the submanifolds M and X given in Section 3.2:

MCXCR^p;q:

Then, we define two open subsets ofMCS^p1S^q1 by

M₇:¼M-R^p₇^{0þp00;q0þq00}: ð6:1:1Þ Likewise, we define two open subsets of the coneXby

X₇:¼X-R^p₇^{0þp00;q0þq00}: ð6:1:2Þ We notice that ifðx;yÞ ¼ ððx⁰;x⁰⁰Þ;ðy⁰;y⁰⁰ÞÞAXthen

jx⁰j4jy⁰j3jx⁰⁰jojy⁰⁰j

because jx⁰j²þ jx⁰⁰j² ¼ jy⁰j²þ jy⁰⁰j²: The following statement is immediate from definition:

Xþ¼| 3 Mþ¼| 3 p⁰q⁰⁰¼0; ð6:1:3Þ X¼| 3 M¼| 3 p⁰⁰q⁰¼0: ð6:1:4Þ

6.2. We embed the direct product of hyperboloids

Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ ¼ fððx⁰;y⁰Þ;ðy⁰⁰;x⁰⁰ÞÞ: jx⁰j² jy⁰j²¼ jy⁰⁰j² jx⁰⁰j²¼1g

intoXþ ðCR^p;qÞby the map

Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ+Xþ;ððx⁰;y⁰Þ;ðy⁰⁰;x⁰⁰ÞÞ/ðx⁰;x⁰⁰;y⁰;y⁰⁰Þ: ð6:2:1Þ The image is transversal to rays (see[15, Section 3.3]for definition) and the induced pseudo-Riemannian metric gXðp⁰;q⁰ÞXðq⁰⁰;p⁰⁰Þ on Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ has signature ðp⁰1;q⁰Þ þ ðp⁰⁰;q⁰⁰1Þ ¼ ðp1;q1Þ:With the notation in Section 5.2, we have

g_Xðp⁰_;q⁰_ÞXðq⁰⁰_;p⁰⁰_Þ¼g_Xðp⁰_;q⁰_Þ"ðgXðq⁰⁰;p⁰⁰ÞÞ:

We note that if p⁰⁰¼q⁰¼0; then Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ is diffeomorphic to S^p1

S^q1;and gXðp⁰;0ÞXðq⁰⁰;0Þ is nothing but the pseudo-Riemannian metricg_S^p1_S^q1 of

signatureðp1;q1Þ(see[15, Section 3.3]).

By the same computation as in (3.4.1), we have the relationship among the Yamabe operators on hyperboloids (see also (5.2.2)) by

D*_Xðp⁰_;q⁰_ÞXðq⁰⁰_;p⁰⁰_Þ¼D*_Xðp⁰_;q⁰_ÞD*_Xðq⁰⁰_;p⁰⁰_Þ: ð6:2:2Þ We denote byF₁ the composition of (6.2.1) and the projection F: X-M (see[15, (3.2.4)]), namely,

F1:Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ+M; ððx⁰;y⁰Þ;ðy⁰⁰;x⁰⁰ÞÞ/ ðx⁰;x⁰⁰Þ jxj ;ðy⁰;y⁰⁰Þ

jyj

: ð6:2:3Þ

Lemma 6.2. (1)The mapF1:Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ-M is a diffeomorphism onto M_þ: The inverse mapF¹₁ :Mþ-Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þis given by the formula:

ððu⁰;u⁰⁰Þ;ðv⁰;v⁰⁰ÞÞ/ ðu⁰;v⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ju⁰j² jv⁰j²

q ; ðv⁰⁰;u⁰⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jv⁰⁰j² ju⁰⁰j² q

0 B@

1

CA: ð6:2:4Þ

(2) F1 is a conformal map with conformal factor jxj¹¼ jyj¹; where x¼ ðx⁰;x⁰⁰ÞAR^p0þp00 and y¼ ðy⁰;y⁰⁰ÞAR^q0þq00:Namely,we have

F₁ðgS^p1S^q1Þ ¼ 1

jxj²g_Xðp⁰_;q⁰_ÞXðq⁰⁰_;p⁰⁰_Þ:

Proof. The first statement is straightforward in light of the formula ju⁰j² jv⁰j²¼ jv⁰⁰j² ju⁰⁰j²40

forðu;vÞ ¼ ððu⁰;u⁰⁰Þ;ðv⁰;v⁰⁰ÞÞAMþCS^p1S^q1:

The second statement is a special case of Lemma 3.3. &

6.3. Now, the conformal diffeomorphismF1:Xðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰Þ !^BMþestablishes a bijection of the kernels of the Yamabe operators owing to Proposition 2.6:

Lemma 6.3. FFf₁₁ gives a bijection fromKerD*Mþ ontoKerD*Xðp⁰;q⁰ÞXðq⁰⁰;p⁰⁰Þ:

Here, the twisted pull-backsFFf₁₁ and^ðFðF

f

F₁:C^NðMþÞ-C^NðXðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰ÞÞ ð6:3:1Þ

f

ðF¹₁ Þ

ðF¹₁ Þ:C^NðXðp⁰;q⁰Þ Xðq⁰⁰;p⁰⁰ÞÞ-C^NðMþÞ ð6:3:2Þ are given by the formulae

ðfFF₁₁FÞðx⁰;y⁰;y⁰⁰;x⁰⁰Þ:¼ ð jx⁰j²þ jx⁰⁰j²Þ

pþq4

4 F ðx⁰;x⁰⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jx⁰j²þ jx⁰⁰j²

q ; ðy⁰;y⁰⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jy⁰j²þ jy⁰⁰j² q

0 B@

1 CA;

ð^ðFðF

f

q ; ðu⁰⁰;v⁰⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jv⁰⁰j² ju⁰⁰j² q

0 B@

1 CA;

respectively. We remark that^ðFðF

f

6.4. Similarly to Section 6.2, we consider another embedding

Xðq⁰;p⁰Þ Xðp⁰⁰;q⁰⁰Þ+X; ððy⁰;x⁰Þ;ðx⁰⁰;y⁰⁰ÞÞ/ðx⁰;x⁰⁰;y⁰;y⁰⁰Þ: ð6:4:1Þ

The composition of (6.4.1) and the projectionF: X-M is denoted by F₂:Xðq⁰;p⁰Þ Xðp⁰⁰;q⁰⁰Þ+M; ððy⁰;x⁰Þ;ðx⁰⁰;y⁰⁰ÞÞ/ ðx⁰;x⁰⁰Þ

jxj ;ðy⁰;y⁰⁰Þ jyj

: ð6:4:2Þ Obviously, results analogous to Lemmas 6.2 and 6.3 hold forF2:For example, here is a lemma parallel to Lemma 6.2:

Lemma 6.4. The mapF2:Xðq⁰;p⁰Þ Xðp⁰⁰;q⁰⁰Þ-M is a conformal diffeomorphism onto M:The inverse mapF¹₂ :M-Xðq⁰;p⁰Þ Xðp⁰⁰;q⁰⁰Þis given by

ððu⁰;u⁰⁰Þ;ðv⁰;v⁰⁰ÞÞ/ ðv⁰;u⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi jv⁰j² ju⁰j²

q ; ðu⁰⁰;v⁰⁰Þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ju⁰⁰j² jv⁰⁰j² q

0 B@

1 CA: