## Analysis on the minimal representation of O(p, q)

## – II. Branching laws

### Toshiyuki KOBAYASHI

RIMS, Kyoto University, Sakyo-ku, Kyoto, 606-8502, Japan

### Bent ØRSTED

Department of Mathematics and Computer Science, SDU - Odense University, Campusvej 55, DK-5230, Odense M, Denmark

Abstract

This is a second paper in a series devoted to the minimal unitary representation of O(p, q). By explicit methods from conformal geometry of pseudo Riemannian man- ifolds, we find the branching law corresponding to restricting the minimal unitary representation to natural symmetric subgroups. In the case of purely discrete spec- trum we obtain the full spectrum and give an explicit Parseval-Plancherel formula, and in the general case we construct an infinite discrete spectrum.

Contents

Introduction

§4. Criterion for discrete decomposable branching laws

§5. Minimal elliptic representations of O(p, q)

§6. Conformal embedding of the hyperboloid

§7. Explicit branching formula (discretely decomposable case)

§8. Inner product on $^{p,q} and the Parseval-Plancherel formula

§9. Construction of discrete spectra in the branching laws

Email addresses: toshi@kurims.kyoto-u.ac.jp (Toshiyuki KOBAYASHI), orsted@imada.sdu.dk (Bent ØRSTED).

Introduction

This is the second in a series of papers devoted to the analysis of the mini-
mal representation$^{p,q} of O(p, q). We refer to [24] for a general introduction;

also the numbering of the sections is continued from that paper, and we shall
refer back to sections there. However, the present paper may be read inde-
pendently from [24], and its main object is to study the branching law for the
minimal unitary representation $^{p,q} from analytic and geometric point of
view. Namely, we shall find by explicit means, coming from conformal geom-
etry, the restriction of $^{p,q} with respect to the symmetric pair

(G, G^{0}) = (O(p, q), O(p^{0}, q^{0})×O(p^{00}, q^{00})).

If one of the factors inG^{0} is compact, then the spectrum is discrete (see The-
orem 4.2 also for an opposite implication), and we find the explicit branching
law; when both factors are non-compact, there will still (generically) be an
infinite discrete spectrum, which we also construct (conjecturally almost all
of it; see §9.8). We shall see that the (algebraic) situation is similar to the
theta-correspondence, where the metaplectic representation is restricted to
analogous subgroups.

Let us here state the main results in a little more precise form, referring to sections 8 and 9 for further notation and details.

Theorem A (the branching law for O(p, q) ↓ O(p, q^{0})× O(q^{00}); see Theo-
rem 7.1) If q^{00} ≥ 1 and q^{0} +q^{00} = q, then the twisted pull-back Φ^{f}^{∗}_{1} of the
local conformal map Φ_{1} between spheres and hyperboloids gives an explicit ir-
reducible decomposition of the unitary representation $^{p,q} when restricted to
O(p, q^{0})×O(q^{00}):

(Φ]_{1})^{∗} :$^{p,q}|O(p,q^{0})×O(q^{00}) ∼

→

∞

X⊕ l=0

π^{p,q}^{0}

+,l+^{q00}_{2} −1H^{l}(R^{q}^{00}).

The representations appearing in the decompositions are in addition to usual
spherical harmonicsH^{l}(R^{q}) for compact orthogonal groupsO(q), also the rep-
resentations π_{+,λ}^{p,q} for non-compact orthogonal groups O(p, q). The latter ones
may be thought of as discrete series representations on hyperboloids

X(p, q) :={x= (x^{0}, x^{00})∈R^{p+q} :|x^{0}|^{2}− |x^{00}|^{2} = 1}

for λ >0 or their analytic continuation for λ ≤0; they may be also thought
of as cohomologically induced representations from characters of certain θ-
stable parabolic subalgebras. The fact that they occur in this branching law
gives a different proof of the unitarizability of these modules π_{+,λ}^{p,q} for λ >

−1, once we know $^{p,q} is unitarizable (cf. Part I, Theorem 3.6.1). It might
be interesting to remark that the unitarizability for λ < 0 (especially, λ =

−^{1}_{2} in our setting) does not follow from a general unitarizability theorem
on Zuckerman-Vogan’s derived functor modules [35], neither from a general
theory of harmonic analysis on semisimple symmetric spaces.

Our intertwining operator (Φ]_{1})^{∗} in Theorem A is derived from a conformal
change of coordinate (see §6 for its explanation) and is explicitly written.

Therefore, it makes sense to ask also about the relation of unitary inner prod- ucts between the left-hand and the right-hand side in the branching formula.

Here is an answer (see Theorem 8.6): We normalize the inner product k kπ^{p,q}_{+,λ}

(see (8.4.2)) such that for λ >0,

kfk^{2}π_{+,λ}^{p,q} =λkfk^{2}L^{2}(X(p,q)), for any f ∈(π^{p,q}_{+,λ})_{K}.

Theorem B (the Parseval-Plancherel formula forO(p, q)↓O(p, q^{0})×O(q^{00}))
1) If we develop F ∈Ker∆^{e}M asF =^{P}^{∞}_{l} F_{l}^{(1)}F_{l}^{(2)} according to the irreducible
decomposition in Theorem A, then we have

kFk^{2}$^{p,q} =

∞

X

l=0

kF_{l}^{(1)}k^{2}_{π}^{p,q0}

+,l+q00 2 −1

k F_{l}^{(2)}k^{2}_{L}^{2}_{(S}q00−1).

2) In particular, if q^{00} ≥ 3, then all of π^{p,q}^{0}

+,l+^{q00}_{2} −1 are discrete series for the
hyperboloid X(p, q^{0}) and the above formula amounts to

kFk^{2}$^{p,q} =

∞

X

l=0

(l+q^{00}

2 −1) kF_{l}^{(1)}k^{2}L^{2}(X(p,q^{0}))k F_{l}^{(2)}k^{2}_{L}^{2}_{(S}q00−1).

The formula may be also regarded as an explicit unitarization of the minimal
representation$^{p,q}on the “hyperbolic space model” by means of the right side
(for an abstract unitarization of$^{p,q}, it suffices to choose a single pair (q^{0}, q^{00})).

We note that the formula was previously known in the case where (q^{0}, q^{00}) =
(0, q) (namely, when each summand in the right side is finite dimensional)
by Kostant, Binegar-Zierau by a different approach. The formula is new and
seems to be particularly interesting even in the special case q^{00} = 1, where
the minimal representation $^{p,q} splits into two irreducible summands when
restricted toO(p, q−1)×O(1).

In Theorem 9.1, we consider a more general setting and prove:

Theorem C (discrete spectrum in the restrictionO(p, q)↓O(p^{0}, q^{0})×O(p^{00}, q^{00}))
The twisted pull-back of the locally conformal diffeomorphism also constructs

X⊕
λ∈A^{0}(p^{0},q^{0})∩A^{0}(q^{00},p^{00})

π_{+,λ}^{p}^{0}^{,q}^{0} π^{p}_{−,λ}^{00}^{,q}^{00}⊕ ^{X}^{⊕}

λ∈A^{0}(q^{0},p^{0})∩A^{0}(p^{00},q^{00})

π^{p}_{−,λ}^{0}^{,q}^{0} π_{+,λ}^{p}^{00}^{,q}^{00}

as a discrete spectrum in the branching law for the non-compact case.

Even in the special case (p^{00}, q^{00}) = (0,1), our branching formula includes a new
and mysterious construction of the minimal representation on the hyperboloid
as below (see Corollary 7.2.1): Let W^{p,r} be the set of K-finite vectors (K =
O(p)×O(r)) of the kernel of the Yamabe operator

Ker∆^{e}_{X(p,r)} ={f ∈C^{∞}(X(p, r)) : ∆_{X}_{(p,r)}f = 1

4(p+r−1)(p+r−3)f}, on which the isometry groupO(p, r) and the Lie algebra of the conformal group O(p, r+ 1) act. The following Proposition is a consequence of Theorem 7.2.2 by an elementary linear algebra.

Proposition D Let m >3 be odd. There is a long exact sequence
0→W^{1,m−1} −→^{ϕ}^{1} W^{2,m−2} −→^{ϕ}^{2} W^{3,m−3} −→ · · ·^{ϕ}^{3} ^{ϕ}−→^{m}^{−}^{2} W^{m−1,1} ^{ϕ}−→^{m}^{−}^{1} 0
such thatKerϕ_{p} is isomorphic to($^{p,q})_{K} for any(p, q)such thatp+q=m+1.

We note that each representation spaceW^{p,q−1} is realized on a different space
X(p, q−1) whose isometry groupO(p, q−1) varies according top(1≤p≤m).

So, one may expect that only the intersections of adjacent groups can act
(infinitesimally) on Kerϕ_{p}. Nevertheless, a larger group O(p, q) can act on
a suitable completion of Kerϕ_{p}, giving rise to another construction of the
minimal representation on the hyperboloid X(p, q−1) = O(p, q −1)/O(p−
1, q−1) ! We note that Kerϕ_{p} is roughly half the kernel of the Yamabe operator
on the hyperboloid (see§7.2 for details).

We briefly indicate the contents of the paper: In section 4 we recall the relevant
facts about discretely decomposable restrictions from [17] and [18], and apply
the criteria to our present situation. In particular, we calculate the associated
variety of$^{p,q} as well as its asymptoticK-support introduced by Kashiwara-
Vergne. Theorem 4.2 and Corollary 4.3 clarify the reason why we start with
the subgroupG^{0} =O(p, q^{0})×O(q^{00}) (i.e.p^{00}= 0). Section 5 contains the identi-
fication of the representationsπ^{p,q}_{+,λ}andπ_{−}^{p,q}_{,λ} ofO(p, q) in several ways, namely
as: derived functor modules, Dolbeault cohomologies, eigenspaces on hyper-
boloids, and quotients or subrepresentations of parabolically induced modules.

In section 6 we give the main construction of embedding conformally a direct
product of hyperboloids into a product of spheres; this gives rise to a canoni-
cal intertwining operator between solutions to the so-called Yamabe equation,
studied in connection with conformal differential geometry, on conformally re-
lated spaces. Applying this principle in section 7 we obtain the branching law
in the case where one factor inG^{0}is compact, and in particular when one factor
is justO(1). In this case we have Corollary 7.2.1, stating that$^{p,q}restricted to
O(p, q−1) is the direct sum of two representations, realized in even respectively
odd functions on the hyperboloid forO(p, q−1). Note here the analogy with the

metaplectic representation. Also note here Theorem 7.2.2, which gives a mys-
terious extention of$^{p,q}by$^{p+1,q−1} - both inside the space of solutions to the
Yamabe equation on the hyperboloidX(p, q−1) = O(p, q−1)/O(p−1, q−1).

We also point out that the representationsπ_{+,λ}^{p,q} forλ= 0,−^{1}_{2} are rather excep-
tional; they are unitary, but outside the usual “fair range” for derived functor
modules, see the remarks in section 8.4. Section 8 contains a proof of Theo-
rem 3.9.1 of [24] on the spectra of the Knapp-Stein intertwining operators and
gives the explicit Parseval-Plancherel formulas for the branching laws. Finally,
in section 9 we use certain Sobolev estimates to construct infinitely many dis-
crete spectra when both factors in G^{0} are non-compact. We also conjecture
the form of the full discrete spectrum (true in the case of a compact factor). It
should be interesting to calculate the full Parseval-Plancherel formula in the
case of both discrete and continuous spectrum.

The first author expresses his sincere gratitude to SDU - Odense University for the warm hospitality.

4 Criterion for discrete decomposable branching laws

4.1 Our object of study is the discrete spectra of the branching law of the
restriction$^{p,q}with respect to a symmetric pair (G, G^{0}) = (O(p, q), O(p^{0}, q^{0})×
O(p^{00}, q^{00})). The aim of this section is to give a necessary and sufficient condition
onp^{0}, q^{0}, p^{00} and q^{00} for the branching law to be discretely decomposable.

We start with general notation. LetGbe a linear reductive Lie group, and G^{0}
its subgroup which is reductive in G. We take a maximal compact subgroup
K of G such that K^{0} := K ∩G^{0} is also a maximal compact subgroup. Let
g_{0} = k_{0}+p_{0} be a Cartan decomposition, and g = k+p its complexification.

Accordingly, we have a direct decompositiong^{∗} =k^{∗}+p^{∗} of the dual spaces.

Let π ∈ G. We say that the restriction^{b} π|G^{0} is G^{0}-admissible if π|G^{0} splits
into a direct Hilbert sum of irreducible unitary representations of G^{0} with
each multiplicity finite (see [16]). As an algebraic analogue of this notion, we
say the underlying (g, K)-module π_{K} isdiscretely decomposable as a (g^{0}, K^{0})-
module, ifπ_{K} is decomposed into an algebraic direct sum of irreducible (g^{0}, K^{0})-
modules (see [18]). We note that if the restriction π|^{K}^{0} isK^{0}-admissible, then
the restrictionπ|G^{0} is alsoG^{0}-admissible ([16], Theorem 1.2) and the underlying
(g, K)-moduleπ_{K} is discretely decomposable (see [18], Proposition 1.6). Here
are criteria forK^{0}-admissibility and discrete decomosability:

Fact 4.1 (see [17], Theorem 2.9 for (1); [18], Corollary 3.4 for (2))

1)IfAS_{K}(π)∩Ad^{∗}(K)(k^{0})^{⊥}= 0, thenπisK^{0}-admissible and alsoG^{0}-admissible.

2)Ifπ_{K}is discretely decomposable as a(g^{0}, K^{0})-module, thenpr_{g→g}0(Vg(π_{K}))⊂

Ng^{∗}^{0}.

Here, ASK(π) is the asymptotic cone of

Supp_{K}(π) := {highest weight of τ ∈K^{d}_{0} : [π|K0 :τ]6= 0}

whereK_{0} is the identity component of K, and (k^{0})^{⊥}⊂k^{∗} is the annihilator of
k^{0}. Let Np^{∗} (⊂p^{∗}) be the nilpotent cone for p. Vg(π_{K}) denotes the associated
variety of π_{K}, which is an Ad^{∗}(KC)-invariant closed subset of Np^{∗}. We write
the projection pr_{p→p}0 p^{∗} →p^{0∗} dual to the inclusion p^{0} ,→p.

4.2 Let us consider our setting whereπ=$^{p,q}and (G, G^{0}) = (O(p, q), O(p^{0}, q^{0})×
O(p^{00}, q^{00})).

Theorem 4.2 Suppose p^{0}+p^{00}=p (≥ 2), q^{0}+q^{00} =q (≥2) and p+q∈ 2N.
Then the following three conditions on p^{0}, q^{0}, p^{00}, q^{00} are equivalent:

i) $^{p,q} is K^{0}-admissible.

ii) $_{K}^{p,q} is discretely decomposable as a (g^{0}, K^{0})-module.

iii) min(p^{0}, q^{0}, p^{00}, q^{00}) = 0.

The implication (i) ⇒ (ii) holds by a general theory as we explained ([18],
Proposition 1.6); (ii)⇒ (iii) will be proved in §4.4, and (iii) ⇒ (i) in§4.5, by
an explicit computation of the asymptotic cone AS_{K}($^{p,q}) and the associated
variety Vg($_{K}^{p,q}) which are used in Fact 4.1.

Remark Analogous results to the equivalence (i)⇔(ii) in Theorem 4.2 were
first proved in [18], Theorem 4.2 in the setting where (G, G^{0}) is any reductive
symmetric pair and the representation is any A_{q}(λ) module in the sense of
Zuckerman-Vogan, which may be regarded as “representations attached to
elliptic orbits”. We note that our representations $^{p,q} are supposed to be
attached to nilpotent orbits. We refer [21], Conjecture A to relevant topics.

4.3 The following corollary is a direct consequence of Theorem 4.2, which will be an algebraic background for the proof of the explicit branching law (Theorem 7.1).

Corollary 4.3 Suppose that min(p^{0}, q^{0}, p^{00}, q^{00}) = 0.

1) The restriction of the unitary representation $^{p,q}|G^{0} is also G^{0}-admissible.

2) The space of K^{0}-finite vectors $^{p,q}_{K}0 coincides with that of K-finite vectors

$_{K}^{p,q}.

PROOF. See [16], Theorem 1.2 for (1); and [18], Proposition 1.6 for (2). 2

A geometric counterpart of Corollary 4.3 (2) is reflected as the removal of
singularities of matrix coefficients for the discrete spectra in the analysis that
we study in §6; namely, any analytic function defined on an open subset M_{+}
(see §6 for notation) of M which is a K^{0}-finite vector of a discrete spectrum,
extends analytically onM ifp^{00} = 0. The reason for this is not only the decay of
matrix coefficients but a matching condition of the leading terms fort→ ±∞.
This is not the case for min(p^{0}, q^{0}, p^{00}, q^{00})>0 (see §9).

4.4 Proof of (ii) ⇒ (iii) in Theorem 4.2.

We identify p^{∗} with p via the Killing form, which is in turn identified with
M(p, q;C) by

M(p, q;C)→^{∼} p, X 7→

O X

tX O

.

Then the nilpotent cone Np^{∗} corresponds to the following variety:

{X ∈M(p, q;C) : both X^{t}X and ^{t}XX are nilpotent matrices}. (4.4.1)
We put

M_{0,0}(p, q;C) :={X ∈M(p, q;C) :X^{t}X=O,^{t}XX =O}.

ThenM_{0,0}(p, q;C)\ {O}is the uniqueKC 'O(p,C)×O(q,C)-orbit of dimen-
sionp+q−3. The associated varietyVg($^{p,q}_{K} ) of$^{p,q} is of dimensionp+q−3,
which follows easily from theK-type formula of$^{p,q}(see [24], Theorem 3.6.1).

Thus, we have proved:

Lemma 4.4 The associated variety Vg($^{p,q}_{K} ) equals M_{0,0}(p, q;C).

The projection pr_{p→p}0 :p^{∗} →p^{0∗} is identified with the map

pr_{p→p}0 :M(p, q;C)→M(p^{0}, q^{0};C)⊕M(p^{00}, q^{00};C),

X_{1} X_{2}
X_{3} X_{4}

7→(X_{1}, X_{4}).

Suppose p^{0}p^{00}q^{0}q^{00} 6= 0. If we take
X :=E_{1,1}−E_{p}0+1,q^{0}+1+√

−1E_{p}0+1,1+√

−1E_{1,q}0+1 ∈M_{0,0}(p, q;C),
then pr_{p}_{→}_{p}0(X) = (E_{1,1},−E_{p}0+1,q^{0}+1). But E_{1,1} 6∈ No(p^{∗} ^{0},q^{0}) and −E_{p}0+1,q^{0}+1 6∈

No(p^{∗} ^{00},q^{00}). Thus, pr_{g→g}0(X)6∈ Ng^{∗}^{0}. It follows from Fact 4.1 (2) that $_{K}^{p,q} is not
discrete decomposable as a (g^{0}, K^{0})-module. Hence (ii)⇒ (iii) in Theorem 4.2
is proved. 2

4.5 Proof of (iii) ⇒ (i) in Theorem 4.2.

We take an orthogonal complementary subspace k^{00}_{0} of k^{0}_{0} in k_{0} 'o(p) +o(q).

Lett^{c}_{0} be a Cartan subalgebra of k_{0} such thatt^{00}_{0} :=t^{c}_{0}∩k^{00}_{0} is a maximal abelian
subspace ink^{00}_{0}. We choose a positive system ∆^{+}(k,t^{c}) which is compatible with
a positive system of the restricted root system Σ(k,t^{00}). Then we can find a
basis {f_{i} : 1≤i≤[^{p}_{2}] + [^{q}_{2}]}on√

−1t^{∗}_{0} such that a positive root system of kis
given by

4^{+}(k,t^{c}) ={f_{i}±f_{j} : 1≤i < j ≤[p
2]}

∪{f_{i}±f_{j} : [p

2] + 1≤i < j ≤[p 2] + [q

2]}

∪

{f_{l} : 1≤l≤[p

2]} (p:odd)

∪

{fl : [p

2] + 1 ≤l ≤[p 2] + [q

2]} (q:odd)

, and such that

√−1(t^{00}_{0})^{∗} =

min(p^{0},p^{00})

X

j=1

Rf_{i}+

min(q^{0},q^{00})

X

j=1

Rf_{[}^{p}

2]+j (4.5.1)

if we regard (t^{00}_{0})^{∗} as a subspace of (t^{c}_{0})^{∗} by the Killing form.

Supposep^{0}q^{0}p^{00}q^{00} = 0. Without loss of generality we may and do assumep^{00}= 0,
namely,G^{0} =O(p, q^{0})×O(q^{00}) withq^{0}+q^{00}=q.

Let us first consider the case p6= 2. Then the irreducible O(p)-representation
H^{a}(R^{p}) remains irreducible when restricted toSO(p). The corresponding high-
est weight is given by af_{1}. It follows from the K-type formula of $^{p,q} (Theo-
rem 3.6.1) that

Supp_{K}($^{p,q}) ={af1+bf_{[}^{p}

2]+1:a, b∈N, a+p

2 =b+q 2}. Therefore we have proved:

AS_{K}($^{p,q}) =R_{+}(f_{1}+f_{[}^{p}

2]+1). (4.5.2)

Then AS_{K}($^{p,q})∩√

−1(t^{00}_{0})^{∗} ={0} from (4.5.1) and (4.5.2), which implies
AS_{K}($^{p,q})∩√

−1 Ad^{∗}(K)(k^{0}_{0})^{⊥}={0}

because (t^{00}_{0})^{∗}meets any Ad^{∗}(K)-orbit through (k^{0}_{0})^{⊥}. Therefore, the restriction

$^{p,q}|^{K}^{0} isK^{0}-admissible by Fact 4.1 (1).

Ifp= 2, then$^{p,q} splits into two representations (see Remark 3.7.3), say$_{+}^{2,q}
and $_{−}^{2,q}, when restricted to the connected component SO_{0}(2, q). Likewise,

H^{a}(R^{p}) is a direct sum of two one-dimensional representations when restricted
toSO(2) ifa ≥1. Then we have

AS_{K}($_{±}^{2,q}) =R_{+}(±f_{1}+f_{p+1}).

Applying Fact 4.1 (1) to the identity components (G_{0}, G^{0}_{0}) of groups (G, G^{0}),
we conclude that the restriction$_{±}^{p,q}|K_{0}^{0} isK_{0}^{0}-admissible. Hence the restriction

$^{p,q}|K^{0} is also K^{0}-admissible. Thus, (iii) ⇒ (i) in Theorem 4.2 is proved.

Now the proof of Theorem 4.2 is completed. 2

5 Minimal elliptic representations of O(p, q)

5.1 In this section, we introduce a family of irreducible representations
of G = O(p, q), denoted by π^{p,q}_{+,λ}, π_{−}^{p,q}_{,λ}, for λ ∈ A0(p, q), in three different
realizations. These representations are supposed to be attached to minimal
elliptic orbits, for λ > 0 in the sense of the Kirillov-Kostant orbit method.

Here, we set

A_{0}(p, q) :=

{λ∈Z+^{p+q}_{2} :λ >−1} (p > 1, q6= 0),
{λ∈Z+^{p+q}_{2} :λ≥ ^{p}_{2} −1} (p > 1, q= 0),

∅ (p= 1, q6= 0) or (p= 0),

{−^{1}_{2},^{1}_{2}} (p= 1, q= 0).

(5.1.1)

It seems natural to include the parameterλ= 0,−^{1}_{2} in the definition ofA_{0}(p, q)
as above, although λ =−^{1}_{2} is outside the weakly fair range parameter in the
sense of Vogan [37]. Cohomologically induced representations forλ=−^{1}_{2} and
λ = −1 will be discussed in details in a subsequent paper. In particular, the
caseλ=−1 is of importance in another geometric construction of the minimal
representation via Dolbeault cohomology groups (see Part I, Introduction,
Theorem B (4)).

5.2 Let R^{p,q} be the Euclidean space R^{p+q} equipped with the flat pseudo-
Riemannian metric:

gR^{p,q} =dv_{0}^{2}+· · ·+dv_{p}_{−}_{1}^{2}−dv_{p}^{2}− · · · −dv_{p+q}_{−}_{1}^{2}.
We define a hyperboloid by

X(p, q) := {(x, y)∈R^{p,q} :|x|^{2}− |y|^{2} = 1}.

We note X(p,0) ' S^{p−1} and X(0, q) = ∅. If p = 1, then X(p, q) has two
connected components. The groupGacts transitively onX(p, q) with isotropy

subgroup O(p−1, q) at

x^{o} :=^{t}(1,0,· · · ,0). (5.2.1)
ThusX(p, q) is realized as a homogeneous manifold:

X(p, q)'O(p, q)/O(p−1, q).

We induce a pseudo-Riemannian metric g_{X}_{(p,q)} onX(p, q) from R^{p,q} (see [24],

§3.2), and write ∆_{X}_{(p,q)} for the Laplace-Beltrami operator on X(p, q). As in
[24], Example 2.2, the Yamabe operator is given by

∆eX(p,q) = ∆X(p,q)− 1

4(p+q−1)(p+q−3). (5.2.2) Forλ ∈C, we set

C_{λ}^{∞}(X(p, q)) := {f ∈C^{∞}(X(p, q)) : ∆_{X(p,q)}f = (−λ^{2}+ 1

4(p+q−2)^{2})f}

={f ∈C^{∞}(X(p, q)) :∆^{e}_{X(p,q)}f = (−λ^{2}+ 1

4)f}. (5.2.3) Furthermore, for=±, we write

C_{λ,}^{∞}(X(p, q)) :={f ∈C_{λ}^{∞}(X(p, q)) :f(−z) = f(z), z∈X(p, q)}.
Then we have a direct sum decomposition

C_{λ}^{∞}(X(p, q)) =C_{λ,+}^{∞} (X(p, q)) +C_{λ,}^{∞}_{−}(X(p, q)) (5.2.4)
and each space is invariant under left translations of the isometry group G
becauseGcommutes with ∆X(p,q). With the notation in§3.5, we note ifq= 0,
then C_{λ,sgn(−1)}^{∞} k(X(p,0)) is finite dimensional and isomorphic to the space of
spherical harmonics:

H^{k}(R^{p})'C_{λ}^{∞}(X(p,0)) =C_{λ,sgn(−1)}^{∞} k(X(p,0)) (k :=λ+ p−2
2 ).

5.3 Let G = O(p, q) where p, q ≥ 1 and let θ be the Cartan involution
corresponding to K = O(p)×O(q). We extend a Cartan subalgebra t^{c}_{0} of k_{0}
(given in §4.5) to that of g_{0}, denoted by h^{c}_{0}. If both p and q are odd, then
dimh^{c}_{0} = dimt^{c}_{0} + 1; otherwise h^{c}_{0} = t^{c}_{0}. The complexification of h^{c}_{0} is denoted
byh^{c}.

We can take a basis {f_{i} : 1≤i≤[^{p+q}_{2} ]} of (h^{c})^{∗} (see §4.5; by a little abuse of
notation if both pand q are odd) such that the root system ofg is given by

4(g,h^{c}) = {±(f_{i}±f_{j}) : 1 ≤i < j ≤[p+q
2 ]}

∪

{±f_{l} : 1≤l ≤[p+q

2 ]} (p+q:odd)

.

Let {Hi} ⊂ h^{c} be the dual basis for {fi} ⊂ (h^{c})^{∗}. Set t := CH1 (⊂ t^{c} ⊂ h^{c}).

Then the centralizer L of t in G is isomorphic to SO(2)×O(p−2, q). Let q=l+ube aθ-stable parabolic subalgebra ofgwith nilpotent radicalu given by

4(u,h^{c}) := {f_{1} ±f_{j} : 2≤j ≤[p+q

2 ]} ∪({f_{1}} (p+q:odd) ),
and with a Levi partl=l_{0}⊗C given by

l_{0} ≡Lie(L)'o(2) +o(p−2, q).

Any character of the Lie algebra l_{0} (or any complex character of l) is deter-
mined by its restriction to h^{c}_{0}. So, we shall write C_{ν} for the character of the
Lie algebra l0 whose restriction to h^{c} is ν ∈ (h^{c})^{∗}. With this notation, the
character ofL acting on ∧^{dim}^{u}u is written as C_{2ρ(u)} where

ρ(u) := (p+q

2 −1)f_{1}. (5.3.1)

The homogeneous manifoldG/Lcarries aG-invariant complex structure with
canonical bundle ∧^{top}T^{∗}G/L ' G ×L C_{2ρ(u)}. As an algebraic analogue of a
Dolbeault cohomology of a G-equivariant holomorphic vector bundle over a
complex manifold G/L, Zuckerman introduced the cohomological parabolic
inductionR^{j}q ≡^{}R^{g}q

j

(j ∈N), which is a covariant functor from the category
of metaplectic (l,(L∩K)^{∼})-modules to that of (g, K)-modules. Here, L^{e} is a
metaplectic covering of L defined by the character of L acting on ∧^{dim}^{u}u '
C_{2ρ(u)}. In this paper, we follow the normalization in [36], Definition 6.20 which
is different from the one in [34] by a ‘ρ-shift’.

The character C_{λf}_{1} of l_{0} lifts to a metaplectic (l,(L∩ K)^{∼})-module if and
only if λ∈Z+^{p+q}_{2} . In particular, we can define (g, K)-modules R^{j}q(C_{λf}_{1}) for
λ∈A_{0}(p, q). The Z(g)-infinitesimal character of R^{j}q(C_{λf}

1) is given by (λ,p+q

2 −2,p+q

2 −3, . . . ,p+q

2 −[p+q

2 ])∈(h^{c})^{∗}

in the Harish-Chandra parametrization if it is non-zero. In the sense of Vogan [37], we have

C_{λf}_{1} is in the good range ⇔ λ > p+q
2 −2,
C_{λf}_{1} is in the weakly fair range ⇔ λ≥0.

We note that R^{j}q(C_{λf}_{1}) = 0 if j 6=p−2 and if λ∈A_{0}(p, q). This follows from
a general result in [35] forλ ≥0; and [15] forλ=−^{1}_{2}.

5.4 For b ∈ Z, we define an algebraic direct sum of K = O(p)×O(q)- modules by

Ξ(K :b)≡Ξ(O(p)×O(q) :b) := ^{M}

m,n∈N m−n≥b m−n≡b mod 2

H^{m}(R^{p})H^{n}(R^{q}). (5.4.1)

Forλ ∈A_{0}(p, q), we put

b≡b(λ, p, q) :=λ− p 2 +q

2 + 1∈Z, (5.4.2)

≡(λ, p, q) := (−1)^{b}. (5.4.3)

We define the line bundle Ln over G/Lby the character nf_{1} of L (see §5.3).

Here is a summary for different realizations of the representation π_{+,λ}^{p,q}:
Fact 5.4 Let p, q ∈N (p > 1).

1) For any λ∈A_{0}(p, q), each of the following 5 conditions defines uniquely a
(g, K)-module, which are mutually isomorphic. We shall denote it by (π^{p,q}_{+,λ})_{K}.
The (g, K)-module(π_{+,λ}^{p,q})_{K} is non-zero and irreducible.

i) A subrepresentation of the degenerate principal representationInd^{G}_{P}max(⊗C_{λ})
(see §3.7) with K-type Ξ(K :b).

i)^{0} A quotient of Ind^{G}_{P}max(⊗C_{−}_{λ}) with K-type Ξ(K :b).

ii) A subrepresentation of C_{λ}^{∞}(X(p, q))_{K} with K-type Ξ(K :b).

iii) The underlying (g, K)-module of the Dolbeault cohomology group
H_{∂}^{p−2}_{¯} (G/L,L_{(λ+}^{p+q}^{−}^{2}

2 ))_{K}.

iii)^{0} The Zuckerman-Vogan derived functor module R^{p−2}q (C_{λf}

1).

2) In the realization of (ii), if f ∈(π^{p,q}_{+,λ})K, then there exists an analytic func-
tion a∈C^{∞}(S^{p−1}×S^{q−1}) such that

f(ωcosht, ηsinht) =a(ω, η)e^{−}^{(λ+ρ)t}(1 +te^{−}^{2t}O(1)) as t→ ∞.
Here, we put ρ= ^{p+q−2}_{2} .

For details, we refer, for example, to [12] for (i) and (i)^{0}; to [31] for (ii) and
also for a relation with (i) (under some parity assumption on eigenspaces); to
[16], §6 (see also [15]) for (iii)^{0} ⇔(ii); and to [38] for (iii) ⇔(iii)^{0}. The second
statement follows from a general theory of the boundary value problem with
regular singularities; or also follows from a classical asymptotic formula of
hypergeometric functions (see (8.3.1)) in our specific setting.

Remark 1) By definition, (i) and (i)^{0} make sense forp > 1 and q > 0; and
others forp > 1 and q≥0.

2) Each of the realization (i), (i)^{0}, (ii), and (iii) also gives a globalization of
π^{p,q}_{+,λ}, namely, a continuous representation of G on a topological vector space.

Because all of (π_{+,λ}^{p,q})_{K} (λ∈A_{0}(p, q)) are unitarizable we may and do take the
globalizationπ_{+,λ}^{p,q} to be the unitary representation of G.

3) If λ > 0 and λ∈ A0(p, q), then the realization (ii) of π_{+,λ}^{p,q} gives a discrete
series representation forX(p, q). Conversely,

{π^{p,q}_{+,λ}:λ∈A_{0}(p, q), λ >0}

exhausts the set of discrete series representations forX(p, q).

If (p, q) = (1,0), then O(p, q)'O(1) and it is convenient to define represen- tations of O(1) by

π^{1,0}_{+,λ}=

1 (λ=−^{1}_{2}),
sgn (λ= ^{1}_{2}),
0 (otherwise).

As we defined π_{+,λ}^{p,q} in Fact 5.4, we can also define an irreducible unitary
representation, denoted by π_{−,λ}^{p,q}, for λ ∈ A_{0}(q, p) such that the underlying
(g, K)-module has the following K-type

M

m,n∈N
m−n≤−λ+^{q}_{2}−^{p}_{2}−1
m−n≡−λ+^{q}_{2}−^{p}_{2}−1 mod 2

H^{m}(R^{p})H^{n}(R^{q}).

Similarly to π^{p,q}_{+,λ}, the representations π_{−}^{p,q}_{,λ} are realized in function spaces on
another hyperboloid O(p, q)/O(p, q−1).

In order to understand the notation here, we remark:

i) π_{−,λ}^{p,q} ∈O(p, q) corresponds to the representation\ π_{+,λ}^{q,p} ∈O(q, p) if we identify\
O(p, q) with O(q, p).

ii) π_{+,λ}^{p,0} ' H^{k}(R^{p}),where k =λ−^{p−2}_{2} and p≥1, k ∈N.

5.5 The case λ=±^{1}_{2} is delicate, which happens when p+q∈2N+ 1.

First, we assume p+q ∈2N+ 1. By using the equivalent realizations of π_{+,λ}^{p,q}
in Fact 5.4 and by the classification of the composition series of the most
degenerate principal series representation Ind^{G}_{P}max(⊗C_{λ}) (see [12]), we have

non-splitting short exact sequences of (g, K)-modules:

0→(π^{p,q}

−,−^{1}_{2})_{K} →Ind^{G}_{P}max((−1)^{p−q+1}^{2} ⊗C_{−}1

2)→(π^{p,q}_{+,}1
2

)_{K} →0, (5.5.1)
0→(π^{p,q}_{+,}

−^{1}_{2})_{K} →Ind^{G}_{P}max((−1)^{p}^{−}^{2}^{q}^{−}^{1} ⊗C_{−}1

2)→(π^{p,q}

−,^{1}_{2})_{K} →0. (5.5.2)
Because π_{+,λ}^{p,q} (λ ∈ A_{0}(p, q)) is self-dual, the dual (g, K)-modules of (5.5.1)
and (5.5.2) give the following non-splitting short exact sequences of (g, K)-
modules:

0→(π^{p,q}_{+,}1
2

)_{K} →Ind^{G}_{P}max((−1)^{p−q+1}^{2} ⊗C1

2)→(π^{p,q}

−,−^{1}_{2})_{K} →0, (5.5.3)
0→(π^{p,q}

−,^{1}_{2})_{K} →Ind^{G}_{P}max((−1)^{p}^{−}^{2}^{q}^{−}^{1} ⊗C1

2)→(π_{+,}^{p,q}

−^{1}_{2})_{K} →0. (5.5.4)
Next, we assume p+q ∈2N. Then, $^{p,q} is realized as a subrepresentation of
some degenerate principal series (see [24], Lemma 3.7.2). More precisely, we
have non-splitting short exact sequences of (g, K)-modules

0→$_{K}^{p,q}→Ind^{G}_{P}max((−1)^{p}^{−}^{2}^{q} ⊗C_{−1})→^{}(π_{−}^{p,q}_{,1})_{K}⊕(π^{p,q}_{+,1})_{K}^{}→0, (5.5.5)
0→^{}(π_{−}^{p,q}_{,1})_{K}⊕(π^{p,q}_{+,1})_{K}^{}→Ind^{G}_{P}max((−1)^{p−q}^{2} ⊗C_{1})→$_{K}^{p,q}→0, (5.5.6)
and an isomorphism of (g, K)-modules:

Ind^{G}_{P}max((−1)^{p−q+2}^{2} ⊗C_{0})'(π_{−}^{p,q}_{,0})_{K}⊕(π^{p,q}_{+,0})_{K}. (5.5.7)
These results will be used in another realization of the unipotent representa-
tion $^{p,q}, namely, as a submodule of the Dolbeault cohomology group in a
subsequent paper (cf. Part 1, Introduction, Theorem B (4)).

6 Conformal embedding of the hyperboloid

This section prepares the geometric setup which will be used in §7 and§9 for
the branching problem of $^{p,q}|G^{0}. Throughout this section, we shall use the
following notation:

|x|^{2} :=|x^{0}|^{2}+|x^{00}|^{2} =

p^{0}

X

i=1

(x^{0}_{i})^{2}+

p^{00}

X

j=1

(x^{00}_{j})^{2}, for x:= (x^{0}, x^{00})∈R^{p}^{0}^{+p}^{00} =R^{p},

|y|^{2} :=|y^{0}|^{2}+|y^{00}|^{2} =

q^{0}

X

i=1

(y_{i}^{0})^{2}+

q^{00}

X

j=1

(y_{j}^{00})^{2}, for y:= (y^{0}, y^{00})∈R^{q}^{0}^{+q}^{00} =R^{q}.

6.1 We define two open subsets of R^{p+q} by
R^{p}

0+p^{00},q^{0}+q^{00}

+ :={(x, y) = ((x^{0}, x^{00}),(y^{0}, y^{00}))∈R^{p}^{0}^{+p}^{00}^{,q}^{0}^{+q}^{00} :|x^{0}|>|y^{0}|},
R^{p}

0+p^{00},q^{0}+q^{00}

− :={(x, y) = ((x^{0}, x^{00}),(y^{0}, y^{00}))∈R^{p}^{0}^{+p}^{00}^{,q}^{0}^{+q}^{00} :|x^{0}|<|y^{0}|}.
Then the disjoint unionR^{p}

0+p^{00},q^{0}+q^{00}

+ ∪R^{p}

0+p^{00},q^{0}+q^{00}

− is open dense inR^{p+q}. Let us
consider the intersection ofR^{p}

0+p^{00},q^{0}+q^{00}

± with the submanifoldsM and Ξ given in§3.2:

M ⊂Ξ⊂R^{p,q}.

Then, we define two open subsets of M 'S^{p−1} ×S^{q−1} by
M_{±}:=M ∩R^{p}

0+p^{00},q^{0}+q^{00}

± . (6.1.1)

Likewise, we define two open subsets of the cone Ξ by
Ξ_{±} := Ξ∩R^{p}

0+p^{00},q^{0}+q^{00}

± . (6.1.2)

We notice that if (x, y) = ((x^{0}, x^{00}),(y^{0}, y^{00}))∈Ξ then

|x^{0}|>|y^{0}| ⇐⇒ |x^{00}|<|y^{00}|

because|x^{0}|^{2}+|x^{00}|^{2} =|y^{0}|^{2}+|y^{00}|^{2}. The following statement is immediate from
definition:

Ξ_{+} =∅ ⇔ M_{+} =∅ ⇔ p^{0}q^{00} = 0. (6.1.3)
Ξ_{−} =∅ ⇔ M_{−} =∅ ⇔ p^{00}q^{0} = 0. (6.1.4)
6.2 We embed the direct product of hyperboloids

X(p^{0}, q^{0})×X(q^{00}, p^{00}) = {((x^{0}, y^{0}),(y^{00}, x^{00})) :|x^{0}|^{2} − |y^{0}|^{2} =|y^{00}|^{2}− |x^{00}|^{2} = 1}.
into Ξ_{+} (⊂R^{p,q}) by the map

X(p^{0}, q^{0})×X(q^{00}, p^{00}),→Ξ_{+}, ((x^{0}, y^{0}),(y^{00}, x^{00}))7→(x^{0}, x^{00}, y^{0}, y^{00}). (6.2.1)
The image is transversal to rays (see [24],§3.3 for definition) and the induced
pseudo-Riemannian metricg_{X(p}0,q^{0})×X(q^{00},p^{00}) onX(p^{0}, q^{0})×X(q^{00}, p^{00}) has signa-
ture (p^{0} −1, q^{0}) + (p^{00}, q^{00}−1) = (p−1, q−1). With the notation in §5.2, we
have

g_{X}_{(p}0,q^{0})×X(q^{00},p^{00}) =g_{X}_{(p}0,q^{0})⊕(−g_{X}_{(q}00,p^{00})).

We note that if p^{00} = q^{0} = 0, then X(p^{0}, q^{0})×X(q^{00}, p^{00}) is diffeomorphic to
S^{p−1}×S^{q−1}, andg_{X}_{(p}0,0)×X(q^{00},0) is nothing but the pseudo-Riemannian metric
g_{S}p−1×S^{q−1} of signature (p−1, q−1) (see [24], §3.3).

By the same computation as in (3.4.1), we have the relationship among the Yamabe operators on hyperboloids (see also (5.2.2)) by

∆e_{X(p}0,q^{0})×X(q^{00},p^{00}) =∆^{e}_{X(p}0,q^{0})−∆^{e}_{X(q}00,p^{00}). (6.2.2)
We denote by Φ_{1} the composition of (6.2.1) and the projection Φ : Ξ → M
(see [24], (3.2.4)), namely,

Φ_{1} :X(p^{0}, q^{0})×X(q^{00}, p^{00}),→M, ((x^{0}, y^{0}),(y^{00}, x^{00}))7→ (x^{0}, x^{00})

|x| ,(y^{0}, y^{00})

|y|

!

.
(6.2.3)
Lemma 6.2 1) The map Φ_{1} :X(p^{0}, q^{0})×X(q^{00}, p^{00})→M is a diffeomorphism
onto M_{+}. The inverse map Φ^{−1}_{1} : M_{+}→X(p^{0}, q^{0})×X(q^{00}, p^{00}) is given by the
formula:

((u^{0}, u^{00}),(v^{0}, v^{00}))7→

(u^{0}, v^{0})

q|u^{0}|^{2}− |v^{0}|^{2}, (v^{00}, u^{00})

q|v^{00}|^{2}− |u^{00}|^{2}

. (6.2.4)

2) Φ_{1} is a conformal map with conformal factor |x|^{−}^{1} = |y|^{−}^{1}, where x =
(x^{0}, x^{00})∈R^{p}^{0}^{+p}^{00} and y= (y^{0}, y^{00})∈R^{q}^{0}^{+q}^{00}. Namely, we have

Φ^{∗}_{1}(g_{S}p−1×S^{q−1}) = 1

|x|^{2}g_{X}_{(p}0,q^{0})×X(q^{00},p^{00}).

PROOF. The first statement is straightforward in light of the formula

|u^{0}|^{2}− |v^{0}|^{2} =|v^{00}|^{2}− |u^{00}|^{2} >0
for (u, v) = ((u^{0}, u^{00}),(v^{0}, v^{00}))∈M+ ⊂S^{p−1}×S^{q−1}.

The second statement is a special case of Lemma 3.3. 2

6.3 Now, the conformal diffeomorphism Φ_{1} : X(p^{0}, q^{0})×X(q^{00}, p^{00})→^{∼} M_{+}
establishes a bijection of the kernels of the Yamabe operators owing to Propo-
sition 2.6:

Lemma 6.3 Φ^{f}^{∗}_{1} gives a bijection from Ker∆^{e}_{M}_{+} onto Ker∆^{e}_{X}_{(p}0,q^{0})×X(q^{00},p^{00}).
Here, the twisted pull-backs Φ^{f}^{∗}_{1} and ^

(Φ^{−1}_{1} )^{∗} (see Definition 2.3), namely,
Φf^{∗}_{1} : C^{∞}(M_{+}) →C^{∞}(X(p^{0}, q^{0})×X(q^{00}, p^{00})), (6.3.1)
(Φ^^{−1}_{1} )^{∗} : C^{∞}(X(p^{0}, q^{0})×X(q^{00}, p^{00})) → C^{∞}(M_{+}), (6.3.2)

are given by the formulae

(Φ^{f}^{∗}_{1}F)(x^{0}, y^{0}, y^{00}, x^{00}) := (|x^{0}|^{2}+|x^{00}|^{2})^{−}^{p+q−4}^{4} F

(x^{0}, x^{00})

q|x^{0}|^{2}+|x^{00}|^{2}, (y^{0}, y^{00})

q|y^{0}|^{2}+|y^{00}|^{2}

,

(^

(Φ^{−}_{1}^{1})^{∗}f)(u^{0}, u^{00}, v^{0}, v^{00}) := (|u^{0}|^{2}− |v^{0}|^{2})^{−}^{p+q−4}^{4} f

(u^{0}, v^{0})

q|u^{0}|^{2}− |v^{0}|^{2}, (u^{00}, v^{00})

q|v^{00}|^{2}− |u^{00}|^{2}

,

respectively. We remark that(Φ^^{−1}_{1} )^{∗} = (Φ^{f}^{∗}_{1})^{−1}.

6.4 Similarly to §6.2, we consider another embedding

X(q^{0}, p^{0})×X(p^{00}, q^{00}),→Ξ_{−}, ((y^{0}, x^{0}),(x^{00}, y^{00}))7→(x^{0}, x^{00}, y^{0}, y^{00}). (6.4.1)
The composition of (6.4.1) and the projection Φ : Ξ→M is denoted by

Φ_{2} :X(q^{0}, p^{0})×X(p^{00}, q^{00}),→M, ((y^{0}, x^{0}),(x^{00}, y^{00}))7→ (x^{0}, x^{00})

|x| ,(y^{0}, y^{00})

|y|

!

. (6.4.2) Obviously, results analogous to Lemma 6.2 and Lemma 6.3 hold for Φ2. For example, here is a lemma parallel to Lemma 6.2:

Lemma 6.4 The map Φ_{2} : X(q^{0}, p^{0})×X(p^{00}, q^{00}) → M_{−} is a conformal dif-
feomorphism onto M_{−}. The inverse map Φ^{−}_{2}^{1} :M_{−} →X(q^{0}, p^{0})×X(p^{00}, q^{00}) is
given by

((u^{0}, u^{00}),(v^{0}, v^{00}))7→

(v^{0}, u^{0})

q|v^{0}|^{2}− |u^{0}|^{2}, (u^{00}, v^{00})

q|u^{00}|^{2}− |v^{00}|^{2}

.

7 Explicit branching formula (discrete decomposable case)

If one of p^{0}, q^{0}, p^{00} or q^{00} is zero, then the restriction $^{p,q}|^{G}^{0} is decomposed
discretely into irreducible representations of G^{0} = O(p^{0}, q^{0})×O(p^{00}, q^{00}) as we
saw in §4. In this case, we can determine the branching laws of $^{p,q}|G^{0} as
follows:

Theorem 7.1 Let p+q ∈ 2N. If q^{00} ≥ 1 and q^{0} +q^{00} = q, then we have an
irreducible decomposition of the unitary representation$^{p,q} when restricted to