MATHEMATICS

Corresponding Member of the Academy of Sciences of the USSR I. M. GELFAND, I. I. PIATETSKI-SHAPIRO

UNITARY REPRESENTATIONS IN THE SPACE \(G/\Gamma\), WHERE \(G\) IS THE GROUP OF REAL MATRICES OF ORDER \(n\), AND \(\Gamma\) IS THE SUBGROUP OF INTEGRAL MATRICES

The present work is devoted to the study of the unitary representation in the space \(X = G/\Gamma\), where \(G\) is the group of real matrices of order \(n\), and \(\Gamma\) is the subgroup of integral matrices. The principal tool is the method of horocycles. Let us recall the definitions of horocycles and horocyclic subgroups.

Let \(G\) be a real semisimple Lie group, and let \(g(t)\) be some one-parameter subgroup of the group \(G\). The set \(Z \subset G\), consisting of all \(z\) for which
\[ \lim_{t \to \infty} g(-t) z g(t) = 1, \]
is called the horocyclic subgroup associated with the subgroup \(g(t)\). Let \(X\) be a homogeneous space of the group \(G\). Horocycles in \(X\) are the orbits of horocyclic groups. A horocycle is called compact if the set of points of which it consists is compact.

If \(G\) is the group of real matrices of order \(n\), then there exist as many nonconjugate horocyclic subgroups as there are representations of \(n\) in the form of a sum of positive summands \(n = k_1 + k_2 + \cdots + k_s\) (representations differing in order are considered distinct). The corresponding horocyclic subgroups have the form
\[ \begin{pmatrix} E_{k_1} & * & \cdots & * \\ 0 & E_{k_2} & \cdots & * \\ \cdot & \cdot & \cdots & \cdot \\ 0 & 0 & \cdots & E_{k_s} \end{pmatrix}, \tag{1} \]
where below the “diagonal” there stand zeros, and above it arbitrary numbers; \(E_{k_i}\) denotes the identity matrix of order \(k_i\).

We denote the subgroup (1) by \(Z_{k_1,\ldots,k_s}\). The following theorem holds, describing the structure of all compact horocycles in the space \(X = G/\Gamma\).

Theorem 1. Every compact horocycle in \(X\) is the image of a set \(Z_{k_1,\ldots,k_s} g_0\) under the natural mapping of \(G\) onto \(X\), where \(g_0\) denotes an arbitrary fixed element of the group \(G\). The horocycles \(Z_{k_1,\ldots,k_s} g_0\) and \(Z_{k'_1,\ldots,k'_{s'}} g'_0\) are transformed into one another by motions from \(G\) if and only if \(k_1 = k'_1, \ldots, k_s = k'_s, s = s'\).

Let us briefly outline the proof of this theorem. From the compactness of the quotient space \(Z_{k_1,\ldots,k_s}/\Gamma \cap Z_{k_1,\ldots,k_s}\) it follows that the image of the set \(Z_{k_1,\ldots,k_s} g_0\) is a compact horocycle in \(X\). It is also evident that the images of the sets \(Z_{k_1,\ldots,k_s} g_0\) and \(Z_{k'_1,\ldots,k'_{s'}} g'_0\) in \(X\) are transformed into one another by motions from \(G\) if and only if the groups \(Z_{k_1,\ldots,k_s}\) and \(Z_{k'_1,\ldots,k'_{s'}}\) are conjugate in \(G\),

and, consequently, if and only if \(s=s'\), \(k_1=k'_1,\ldots,k_s=k'_s\). More difficult is the proof of the fact that every compact horosphere in \(X\) is the image of the set \(Z_{k_1,\ldots,k_s} g_0\). For this one must first prove that if \(Z\) is a horospherical subgroup in \(G\) such that the quotient space \(Z/\Gamma\cap Z\) is compact, then \(Z\) has the form \(h^{-1}Z_{k_1,\ldots,k_s}h\), where \(h\) is some matrix with rational entries. Then, using uniqueness of factorization in the ring of integers, it is shown that \(Z\) has the form \(\gamma^{-1}Z_{k_1,\ldots,k_s}\gamma\), where \(\gamma\in\Gamma\).

We now pass to the consideration of functions on \(X\). Denote by \(H^0_{k_1,\ldots,k_s}\) the collection of all functions in \(L_2(X)\) whose integrals over all compact horospheres of the form \(Z_{k_1,\ldots,k_s}g_0\) are equal to zero. It is easy to see that the space \(H^0_{k_1,\ldots,k_s}\) is invariant with respect to the translation operators \(T_g\). Next denote by \(H'_{k_1,\ldots,k_s}\) the intersection of all spaces \(H^0_{k'_1,\ldots,k'_s}\) corresponding to the groups \(Z_{k'_1,\ldots,k'_s}\) that contain \(Z_{k_1,\ldots,k_s}\) as a proper subgroup. It is readily checked that \(H^0_{k_1,\ldots,k_s}\subset H'_{k_1,\ldots,k_s}\). We shall further denote by \(H_{k_1,\ldots,k_s}\) the quotient space \(H'_{k_1,\ldots,k_s}/H^0_{k_1,\ldots,k_s}\). In \(H_{k_1,\ldots,k_s}\) the action of the operators \(T_g\) is defined in a natural way. Finally, denote by \(H^0\) the collection of all functions \(f(x)\) in \(L_2(X)\) whose integrals over all compact horospheres are equal to zero.

Theorem 2. The space \(L_2(X)\) is isomorphic to the sum of the spaces \(H_{k_1,\ldots,k_s}\) and \(H^0\)

\[ L_2(X)\cong \sum_{k_1+\cdots+k_s=n} H_{k_1,\ldots,k_s}+H^0 . \tag{2} \]

In our paper (1) it was shown that \(H^0\) decomposes into a sum of a countable number of irreducible unitary representations of the group \(G\). In the present paper we shall deal with the decomposition into irreducible representations of the spaces \(H_{k_1,\ldots,k_s}\).

Apart from its immediate interest, the significance of this problem also lies in the fact that it leads to a number of remarkable analytic functions closely connected with such functions as the classical Riemann zeta-function and its generalizations. It is not excluded that a deep development of the theory of these and analogous functions will shed light also on the unresolved questions in the theory of the classical Riemann zeta-function. We shall now give the definition of these functions for \(H_{k_1,\ldots,k_s}\). The general definition for arbitrary regular groups will be given later.

Consider the set \(\Omega_1\) of all compact horospheres of maximal dimension. Since a motion carries a compact horosphere again into a compact one, it is natural to define an action of the group \(G\) on \(\Omega_1\).

It follows from Theorem 1 that the group \(G\) acts transitively on \(\Omega_1\). To each function \(f(x)\in L_2(X)\) we assign its integral over the compact horospheres from \(\Omega_1\), and denote it by \(\check f(\omega)\). Let \(\mathcal L\) denote the collection of functions \(\check f(\omega)\) on \(\Omega_1\) that are integrals of functions \(f(x)\in L_2(X)\). \(\mathcal L\), evidently, is isomorphic to \(H_{1,1,\ldots,1}\).

Denote by \(L_2(\Omega)\) the collection of all square-summable functions on \(\Omega_1\). The decomposition of \(L_2(\Omega)\) into irreducible representations is carried out, as is known, in the following way. Denote by \(A\) the connected component of the group of left translations of the space \(\Omega\), i.e. transformations \(\omega\mapsto a\omega\) commuting with motions.

The group \(A\) is naturally identified with the group of diagonal matrices with positive entries on the diagonal. Denote by

by \(a^\chi=\exp(\chi\ln a)\), where \(\ln a\) denotes the canonical mapping of \(A\) into its Lie algebra. Put

\[ \Phi(\chi,\omega)=\int_A \varphi(a\omega)\,a^{-\chi} j^{1/2}(a)\,da, \tag{3} \]

where \(j(a)\) is the Jacobian of the transformation \(\omega\to a\omega\), which depends only on \(a\).

Formula (3), where \(\chi\) is purely imaginary, gives a decomposition of \(L_2(\Omega)\) into irreducible representations; moreover, two representations with “numbers” \(\chi\) and \(\tilde\chi\) are equivalent if and only if \(\tilde\chi(a)\equiv \chi(a^\sigma)\) for all \(a\), where \(a^\sigma\) denotes \(a\) with permuted eigenvalues; \(\sigma\) is a permutation.

It can be shown that \(\mathcal L\) decomposes into the sum of two spaces \(\mathcal L'\) and \(\mathcal L_1\), where \(\mathcal L'\) consists only of constants, and \(\mathcal L_1\subset L_2(\Omega)\). It turns out that in \(\mathcal L_1\), unlike in \(L_2(\Omega)\), each irreducible representation occurs once. Therefore the \(\Phi(\chi,\omega)\) for functions from \(\mathcal L_1\), at points corresponding to equivalent irreducible representations, turn out to be related. Put

\[ \Phi(\sigma\chi,\omega)=\xi_\sigma(\chi)\Phi(\chi,\omega). \tag{4} \]

The functions \(\xi_\sigma(\chi)\) are of principal interest. They are meromorphic in the whole complex space and satisfy the functional equation

\[ \xi_{\sigma_1\sigma_2}(\chi)=\xi_{\sigma_1}(\sigma_2\chi)\xi_{\sigma_2}(\chi). \tag{5} \]

The latter is equivalent to the fact that each representation in \(\mathcal L_1\) occurs once.

Using the functional equation (4), one can show that

\[ \xi_\sigma(\chi)=\prod_\alpha \theta((\chi,\alpha)), \]

where \(\theta(s)=B\left(\dfrac12,\dfrac{s}{2}\right)\zeta(s)\zeta^{-1}(s+1)\), \(\zeta(s)\) is the classical Riemann zeta-function, and \(\alpha\) runs through a certain set of roots, which for each \(\sigma\) consists of all negative roots \(\alpha\) for which \(\sigma\alpha\) is a positive root; the brackets denote the Cartan scalar product; \(\sigma\alpha\) is the root obtained from \(\alpha\) by the substitution \(\sigma\).

The spaces \(H_{k_1,\ldots,k_s}\) are studied analogously. It turns out that all of them are the sum of two spaces \(\hat{\mathcal L}_{k_1,\ldots,k_s}\) and \(\mathcal L'_{k_1,\ldots,k_s}\), the first of which is embedded in \(L_2(\Omega_{k_1,\ldots,k_s})\)* and decomposes into the same irreducible representations as \(L_2(\Omega_{k_1,\ldots,k_s})\), but contains them with smaller multiplicity. The latter circumstance leads to functional equations for the arising analogues of the functions \(\xi_\sigma(\chi)\). \(\mathcal L'_{k_1,\ldots,k_s}\) consists of representations corresponding to special points of the functions \(\xi_\sigma(\chi)\) and their analogues.

Let us briefly outline the proof for \(H_{1,\ldots,1}\). (We shall denote the scalar product of two functions \(\check f_1,\check f_2\in\mathcal L\) by \([\check f_1,\check f_2]\).) We shall also agree, for any two functions \(\varphi_1(\omega)\) and \(\varphi_2(\omega)\) on \(\Omega\), for which the integral \(\int \varphi_1\overline{\varphi}_2\,d\omega\) converges absolutely, to denote its value by \((\varphi_1,\varphi_2)\).

It is easy to show that for every finite continuous function \(\varphi(\omega)\) on \(\Omega\) the expression \((\check f,\varphi)\) has meaning for all \(\check f\in\mathcal L\) and is a linear functional on \(\mathcal L\). According to Riesz’s theorem, in \(\mathcal L\) there exists such

* \(\Omega_{k_1,\ldots,k_s}\) denotes the space of horospheres associated with the group \(z_{k_1,\ldots,k_s}\).

a vector \(M_\varphi\) such that \((f,\varphi)=[\check f,M_\varphi]\) for all \(\check f\in \mathcal L\). The operator \(M\) plays a fundamental role in further investigations. For \(M\) there exists a comparatively simple explicit expression, which we shall now give. First let us agree on the following notation: \(\Delta=\Gamma\cap Z\); \(\Delta'\) is the normalizer of \(\Delta\) in \(\Gamma\); \(Z_g=Z\cap g^{-1}Zg\), where \(g\in G\); \(D_g\) is the set of left cosets of the adjacent group with respect to the subgroup \(Z_g\). Now put

\[ M_{g_0}(\varphi(\omega))=\int_{Z_{g_0}}\varphi^*(g_0zg)\,dz, \tag{6} \]

where \(\varphi^*(g)\) denotes the function on \(G\) corresponding to the function \(\varphi(\omega)\) on \(\Omega\).

The following formula holds

\[ M_\varphi=\sum M_\gamma, \tag{7} \]

where \(\gamma\) ranges over the set of all representatives of the double cosets \(\Delta'\gamma\Delta\) of the group \(\Gamma\) with respect to the subgroups \(\Delta'\) and \(\Delta\).

The subsequent arguments are based on consideration of the form \((M\varphi_1,\varphi_2)\) and on the study of the Mellin transform of functions of the form \(M\varphi\), where \(\varphi\) denotes a continuous finite function.

Received
11 VIII 1962

References

1. I. M. Gel'fand, I. I. Pyatetskii-Shapiro, DAN, 147, No. 1 (1962).