MATHEMATICS

Corresponding Member of the Academy of Sciences of the USSR I. M. GELFAND, I. I. PIATETSKII-SHAPIRO

UNITARY REPRESENTATIONS IN HOMOGENEOUS SPACES WITH DISCRETE STATIONARY GROUPS

Let \(G\) be a semisimple Lie group, and \(\Gamma\) a discrete subgroup of the group \(G\). Denote by \(X\) the space of left cosets of \(G\) modulo \(\Gamma\). To each element \(g \in G\) there corresponds, evidently, a motion in \(X\) sending \(x\) to \(xg\). Denote by \(L_2(X)\) the totality of all functions \(f(x)\) on \(X\) with summable square. As is easy to see, the operators \(T_g : T_g f(x) = f(xg)\) are unitary in \(L_2(X)\). It is well known \((^2)\) that if \(X\) is compact, then the representation in \(L_2(X)\) decomposes into a sum of a countable number of irreducible unitary representations. In the case when \(X\) is noncompact, this is no longer true. In this case, as a rule, the “spectrum” of irreducible representations is mixed, i.e. continuous and discrete.

In the present article an effective method is indicated for separating the discrete spectrum from the continuous one. This method is a development of the method of horospheres \((^7)\). In the following paper \((^8)\) we shall use the method of horospheres to study the structure of the continuous “spectrum.”

Denote by \(L_2^0(X)\) the totality of all functions in \(L_2(X)\) whose integrals over all compact horospheres are equal to zero (the definition of horospheres is given below). The following theorem holds:

Theorem. If \(\Gamma\) is a regular subgroup, then \(L_2^0(X)\) decomposes into a sum of a countable number of irreducible unitary representations of the group \(G\).1

In the following paper we shall show that the converse of this theorem holds, namely: every irreducible unitary representation of the group \(G\), lying discretely in \(L_2(X)\), either belongs to \(L_2^0(X)\), or is an “analytic continuation” of representations of the continuous spectrum belonging to \(L_2(X)\).

The present work consists of three parts: 1) the definition of some concepts relating to arbitrary real semisimple Lie groups and homogeneous spaces; 2) the definition of regular discrete groups; 3) an outline of the proof of the theorem on the decomposition of \(L_2^0(X)\) into a sum of a countable number of irreducible unitary representations of the group \(G\).

1. Let \(G\) be a real semisimple Lie group; \(g(t)\) a certain one-parameter subgroup of the group \(G\). The set \(Z \subset G\), consisting of all \(z \in G\) for which

\[ \lim_{t \to +\infty} g(-t)\, z g(t) = 1, \tag{1} \]

is called the horospherical subgroup associated with the group \(g(t)\).

Let \(X\) be a homogeneous space of the group \(G\). Horospheres in \(X\) are the orbits of horospherical groups. A horosphere is called compact if the set of points of which it consists is compact.

Let \(A\) be a commutative subgroup of \(G\), consisting of semisimple transformations with real eigenvalues. \(A\) is called an accompanying subgroup of the group \(Z\), if \(A\) is generated by such one-parameter subgroups \(h_1(t), \ldots, h_\nu(t)\), for

for which (1) is valid, and every \(g \in G\) for which (1) holds for all subgroups \(h_1,\ldots,h_\nu\) belongs to \(Z\).

2. In the present section we give a definition of regular discrete subgroups for arbitrary semisimple groups. First we introduce the following simple notion. A set \(Y \subset X\) shall be called cylindrical if it can be decomposed into mutually disjoint compact horospheres which are transformed into one another by transformations from \(G\).

A discrete subgroup \(\Gamma\) of a semisimple group \(G\) is called regular if the quotient space \(X = G/\Gamma\) has a finite covering by regular bounded cylindrical sets, with the intersection of any two of them compact. The definition of which cylindrical sets are called regular and bounded is given below.

It is easy to verify that for every regular discrete subgroup \(\Gamma\) of a semisimple group \(G\), the quotient space \(G/\Gamma\) has finite volume. Although there are grounds to think that the converse assertion is always true, at present it has been established only for the group of real matrices of second order. This result belongs to C. L. Siegel.

We now pass to the definition of regular bounded cylindrical sets. It is not difficult to see that for any cylindrical set \(Y\) there exist a horospherical subgroup \(Z\) and a set \(S \subset G\) whose image in \(G/\Gamma\) is \(Y\), with the following properties: 1) for any \(g \in S\) and \(z \in Z\) there exists \(\delta \in \Delta = \Gamma \cap Z\) such that \(\delta z g \in S\); 2) if \(g_1, g_2 \in S\) and \(g_1 g_2^{-1} \subset \Gamma\), then \(g_1 g_2^{-1} \subset \Delta = \Gamma \cap Z\). Condition 1) means that the image of \(S\) consists of a horosphere, and condition 2), that these horospheres do not intersect.

If the set \(Y\) is compact, then \(S\) can be chosen so that it also has the following properties: 3) there exists a neighborhood of the identity \(V\) of the group \(G\) such that from \(g_1^{-1}\gamma g_2 \in V\), where \(g_1, g_2 \in S\), \(\gamma \in \Gamma\), it follows that \(\gamma \in \Delta\); 4) for any \(z \in Z\) there exists a compact neighborhood of the identity \(V_z\) such that \(g^{-1} z g \subset V_z\) for every \(g \in S\). Condition 3) is a certain strengthening of condition 2). Conditions 3 and 4) play a central role in the work.

We shall call a cylindrical set \(Y\), for which there exist \(S\) and \(Z\) possessing the properties 1)—4) indicated above, bounded. Bounded cylindrical sets, generally speaking, are not compact, and this is apparently the geometric reason for the existence in semisimple groups of discrete subgroups for which the quotient space \(G/\Gamma\) has finite volume and at the same time is not compact.

Let \(Y\) be a bounded cylindrical set. It can be shown that all elements \(g \in S\) are representable in the form \(g = zat\), where \(a\) belongs to some accompanying subgroup \(A\) of the group \(Z\), and \(t\) belongs to some compact set \(T\) in \(G\). It is well known that \(A\) is contained in the normalizer of the group \(Z\). We shall henceforth call a normal divisor \(\widetilde Z\) of the group \(Z\) \(A\)-admissible, or simply admissible, if the Lie algebra of the group \(\widetilde Z\) is the sum of root spaces\(^*\). A bounded cylindrical set is called regular if: 5) for any admissible subgroup \(\widetilde Z\) of the group \(Z\) (including \(Z\) itself) the quotient space \(\widetilde Z / \widetilde Z \cap \Gamma\) is compact.

Thus, ultimately, regular bounded cylindrical sets are characterized by the existence for them of \(S, Z\), and \(A\), for which 1)—5) hold.

In a certain special case 5) coincides with a condition appearing in the work of Harish-Chandra \((^6)\).

Finally, let us note that all presently known examples of irre—

\(^*\) The totality of all \(\xi\) from the Lie algebra of the group \(Z\) for which \([a,\xi] = \beta(a)\xi\) for any \(a\) from the Lie algebra of the group \(A\) is called the root space corresponding to the root \(\beta(a)\).

reducible* discrete subgroups of semisimple Lie groups with finite volume of the quotient space, with the exception of subgroups of the group of real matrices of second order, are arithmetic groups in the sense of Borel and Harish-Chandra (⁵).

A. Borel and Harish-Chandra (⁵) proved that the volume of the quotient space \(G/\Gamma\) for such groups is finite. Apparently, by their methods one can show that every arithmetic group is a regular discrete subgroup of the corresponding Lie group. In any case, for classical discrete subgroups, such as, for example, the group of integral matrices, it is easily proved that they are regular.

1. Denote by \(L_2^0(X)\) the set of functions \(f(x)\in L_2(X)\) whose integrals over all compact horospheres are equal to zero. In the present section we shall show that the operator

\[ T_\varphi=\int_G \varphi(g)T_g\,dg \]

in the space \(L_2^0(X)\) is completely continuous for any continuous finite function \(\varphi(g)\). From this it follows at once that \(L_2^0(X)\) decomposes into a sum of a countable number of irreducible representations of the group \(G\).

In fact, it will be proved that the operator \(T_\varphi\) is a kernel operator in \(L_2^0(X)\), if \(\varphi\) is a finite infinitely differentiable function of the form

\[ \int_G \psi(gg_1)\overline{\psi(g_1)}\,dg_1, \]

where \(\psi(g)\) is some continuous finite function.

Let \(Y\) be some regular “bounded” cylindrical set. By definition \(Y\) is decomposed canonically into compact horospheres. Each compact horosphere, generally speaking, is also decomposed into compact horospheres. The decomposition obtained in this way will be called a subordinate decomposition.

Denote now by \(L_2^0(Y)\) the set of functions in \(L_2(X)\), equal to zero outside \(Y\), whose integrals over the horospheres of any decomposition subordinate to the canonical decomposition of \(Y\) are equal to zero. It is sufficient to prove that the operator \(PT_\varphi P\) is a kernel operator, where \(P\) is the operator of projection of \(L_2(X)\) onto \(L_2^0(Y)\).

As is known (²), the operator \(T_\varphi\) is an integral operator on \(X\) with kernel

\[ K(x_1,x_2)=\sum\nolimits' \varphi(g_1^{-1}\gamma g_2), \tag{2} \]

where \(g_1,g_2\) are representatives of the cosets \(x_1,x_2\). It is easy to show that, if the function \(\varphi(g)\) is different from zero only in a sufficiently small neighborhood of the identity of the group \(G\), then for \(x_1,x_2\in Y\)

\[ K(x,x_2)=\sum \varphi(g_1^{-1}\delta g_2), \tag{3} \]

where \(g_1,g_2\) are representatives of cosets belonging to \(S\).

The function \(K\) is not bounded on \(X\), and therefore the trace of the operator \(T_\varphi\) in \(L_2(X)\) does not exist. However, the estimate of the trace of the operator \(PT_\varphi P\) can be reduced to an estimate of derivatives of the function \(K\) in directions lying in the horospheres. With the help of (4) the required estimate of the derivatives is obtained quite simply.

1. It is unknown whether the asymptotic formulas for the distribution of the “numbers” of representations entering into \(L_2(G/\Gamma)\), established for the case when \(G/\Gamma\) is compact, remain valid for the distribution of the numbers of representations entering into \(L_2^0(G/\Gamma)\), where \(\Gamma\) is a regular discrete subgroup of the semisimple group \(G\). In particular, for the case when \(G/\Gamma\) is compact, the following is true—

* A discrete subgroup \(\Gamma\) of a group \(G\) is called irreducible if \(G\) cannot be represented as the product of two subgroups \(G_1\) and \(G_2\) such that the group \(\Gamma_1\times \Gamma_2\), \(\Gamma_k\subset G_k\), is a subgroup of finite index in \(\Gamma\).

following asymptotic relation. The irreducible unitary representations into which \(L_2(G/U)\) (\(U\) is a maximal compact subgroup of the group \(G\)) decomposes are naturally specified by a “number” \(\rho\), ranging over the following domain.

Denote by \(\mathfrak A\) the Cartan subalgebra of the symmetric space \(G/U\). Further, let \(\mathfrak A^+\) denote the set of vectors in \(\mathfrak A\) for which \((a,\alpha)\geq 0\) for all positive roots \(\alpha\). \(\mathfrak A^+\) is the natural domain of variation of the “number” \(\rho\). Further, let \(B_n\) be an expanding sequence of subdomains of \(\mathfrak A^+\). Denote by \(N(B_n)\) the number of irreducible representations entering into \(L_2(X)\) with numbers belonging to \(B_n\). Then

\[ N(B_n)\sim C_G C_\Gamma \int_{B_n} \prod_{\alpha>0}(\rho,\alpha)^{\nu_\alpha}\,d\rho, \]

where \(C_\Gamma\) is the volume of \(G/\Gamma\); \(C_G\) is a constant depending only on \(G\); \(\nu_\alpha\) is the multiplicity of the root \(\alpha\).

This formula is proved using certain results of F. I. Karpelevich and S. G. Gindikin \(\left({}^{9}\right)\).

Received
11 VIII 1962

[REFERENCES]

\({}^{1}\) I. M. Gel'fand, S. V. Fomin, UMN, 8, 118 (1952).
\({}^{2}\) I. M. Gel'fand, I. I. Pyatetskii-Shapiro, UMN, 14, 171 (1959).
\({}^{3}\) A. Selberg, Collected Translations. Mathematics, 1, 4 (1957).
\({}^{4}\) A. Selberg, Collected Translations. Mathematics, 6, 3 (1962).
\({}^{5}\) A. Borel, Harich-Chandra, Bull. Am. Math. Soc., 67, 6, 579 (1961).
\({}^{6}\) Harich-Chandra, Proc. Nat. Ac. Sci. USA, 45, 570 (1959).
\({}^{7}\) I. M. Gel'fand, M. I. Graev, Tr. Moskovsk. matem. obshch., 8, 323 (1959).
\({}^{8}\) I. M. Gel'fand, I. I. Pyatetskii-Shapiro, DAN, 147, No. 2 (1962).
\({}^{9}\) F. I. Karpelevich, S. G. Gindikin, DAN, 145, No. 2 (1962).

1. For the definition of a regular subgroup, see below. ↩