MATHEMATICS

Corresponding Member of the Academy of Sciences of the USSR I. M. GELFAND, M. I. GRAEV

CATEGORIES OF REPRESENTATIONS OF GROUPS AND THE PROBLEM OF CLASSIFYING IRREDUCIBLE REPRESENTATIONS

1. Let \(G\) be an arbitrary (discrete or continuous) group, and \(N\) a subgroup of it. Consider the set \(X\) of representations \(T_x\) \((x \in X)\) of the group \(G\), induced by irreducible unitary representations of the subgroup \(N\). To each representation \(T_x\) we assign the ring \(R_x\) of all linear operators commuting with the operators of the representation. To each pair of representations \(T_{x_1}, T_{x_2}\), acting respectively in the spaces \(H_1, H_2\), we assign the linear family \(R_{x_2 x_1}\) of all linear mappings from \(H_1\) into \(H_2\) that commute with the operators of the representations. The family of representations \(T_x\), with the rings \(R_x\) and the linear spaces \(R_{x_2 x_1}\), will be called the category of representations of the group \(G\), associated with the subgroup \(N\)*. The problem arises of describing this category. This problem is interesting for a number of reasons: its solution leads to a convenient classification of irreducible unitary representations of the group \(G\) (those entering the regular representation), gives a key to constructing special functions connected with the group \(G\), etc.

In the present note we study the group \(G\) of unimodular matrices of the second order

\[ g=\begin{pmatrix}\alpha&\beta\\ \gamma&\delta\end{pmatrix},\quad \alpha\delta-\beta\gamma=1, \]

with entries from a finite field \(K\) of order \(k\) (as is known, \(k\) is a power of a prime number \(p\); we assume that \(p\ne 2\)). A description is given of the category of representations associated with the subgroup \(Z\) of matrices of the form

\[ \xi_t=\begin{pmatrix}1&t\\0&1\end{pmatrix}. \]

On the basis of this description, a convenient classification is given of all irreducible representations of the group \(G\), as well as their effective construction (the latter problem is connected with the definition of Bessel functions over a finite field). At the end of the note similar results are formulated for the group of matrices over the field of real numbers.

2. Each irreducible representation of the subgroup \(Z\) is given by an additive character \(\chi(t)\), \(t\in K\) \((\chi(t_1+t_2)=\chi(t_1)\chi(t_2))\). The representation \(T_\chi\) of the group \(G\) induced by it is constructed in the space \(H_\chi\) of functions \(f(g)\) on \(G\) (with values in the field of complex numbers) satisfying the condition

\[ f(\xi_t g)=\chi(t)f(g) \]

for any \(g\in G\) and \(t\in K\). The operator of the represen-

* Let us recall the definition of an induced representation for the case of a unimodular subgroup \(N\) (for the general definition see, for example, (1)). Let \(c(n)\) be a unitary representation of the subgroup \(N\), acting in a space \(H\) with norm \(\|h\|\). The representation \(T(g)\) of the group \(G\), induced by the representation \(c(n)\), is constructed in the Hilbert space of functions \(f(g)\) on \(G\) with values in \(H\) such that \(f(ng)=c(n)f(g)\) for any \(n\in N\) and

\[ \int_{\widetilde G}\|f(\widetilde g)\|^2\,d\widetilde g<\infty, \]

where \(\widetilde G=G/N\), \(d\widetilde g\) is an invariant measure on \(\widetilde G\). The operator of the representation \(T(g)\) is the shift operator:

\[ T(g_0)f(g)=f(gg_0). \]

We note that in the case of Lie groups it is useful, besides unitary representations, also to consider nonunitary representations acting in certain naturally defined nuclear spaces.

has the form \(T_\chi(g_0)f(g)=f(gg_0)\). Thus, in all there are \(k\) (\(k\) is the order of the field \(K\)) induced representations \(T_\chi\).

The representation \(T_\chi\) can also be realized in the space of functions \(f(x)\equiv f(x_1,x_2)\) on the “affine plane” \((x_1,x_2\in K,\ (x_1,x_2)\ne(0,0))\); the set of such pairs \((x_1,x_2)\) is identical with the set of cosets \(G/Z\). In this realization the representation operator has the form
\[ T_\chi(g)f(x)=f(xg)a(x,g), \]
where \(xg=(\alpha x_1+\gamma x_2,\ \beta x_1+\delta x_2)\), and \(a(x,g)\) is a fixed function satisfying the relations \(a((0,1),\xi_t)=\chi(t)\), \(a(x,g_1g_2)=a(x,g_1)a(xg_1,g_2)\). Thus, in essence we are studying the category of representations of the group \(G\) connected with the affine plane.

3. Description of the rings \(R_\chi\) and the spaces \(R_{\chi_2\chi_1}\). The operators \(A\in R_{\chi_2\chi_1}\) have the form
\[ Af(g)=\frac1k\sum \varphi(gg_1^{-1})f(g_1) \]
(the sum is taken over all \(g_1\in G\)), where \(\varphi(g)\) is an arbitrary function on \(G\) satisfying, for any \(g\in G,\ t_1,t_2\in K\), the relation
\[ \varphi(\xi_{t_2}g\xi_{t_1})=\chi_2(t_2)\varphi(g)\chi_1(t_1). \tag{1} \]

Hence, for \(\chi_1=\chi_2=\chi\), we obtain that \(R_\chi\) is isomorphic to the ring of functions \(\varphi(g)\) satisfying relation (1), with multiplication law
\[ \varphi_1(g)*\varphi_2(g)=\frac1k\sum_{g_1}\varphi_1(gg_1^{-1})\varphi_2(g_1) * . \]
In particular, the ring \(R_1\), corresponding to the character \(\chi\equiv 1\), is isomorphic to the ring of functions constant on the double cosets with respect to the subgroup \(Z\). We first describe this ring \(R_1\).

Let \(A_\lambda\) be the characteristic function of the double coset with respect to \(Z\) with representative
\[ \begin{pmatrix} \lambda^{-1} & 0\\ 0 & \lambda \end{pmatrix}, \]
and \(B_\lambda\) the characteristic function of the coset with representative
\[ \begin{pmatrix} 0 & -\lambda^{-1}\\ \lambda & 0 \end{pmatrix}. \]
Then \(A_\lambda, B_\lambda\) form a basis in the linear space \(R_1\) (and hence the dimension of \(R_1\) is \(2k-2\)). The multiplication operation in \(R_1\) is given by the formulas
\[ A_{\lambda_1}A_{\lambda_2}=A_{\lambda_1\lambda_2},\quad A_{\lambda_1}B_{\lambda_2}=B_{\lambda_1\lambda_2},\quad B_{\lambda_1}A_{\lambda_2}=B_{\lambda_1\lambda_2^{-1}},\quad B_{\lambda_1}B_{\lambda_2}=\sum_\lambda B_\lambda+kA_{-\lambda_1\lambda_2^{-1}}. \]
From these formulas it is easy to find that the center of the ring \(R_1\) consists of elements of the form
\[ A=\alpha_1A_1+\alpha_{-1}A_{-1}+\sum_{\lambda'}\beta_\lambda(A_\lambda+A_{\lambda^{-1}}) +\gamma'\sum_{\lambda'}B_\lambda+\gamma''\sum_{\lambda''}B_\lambda, \]
where \(\lambda'\) runs through the set of all squares, and \(\lambda''\) through the complementary set (and hence the dimension of the center is \(\tfrac12(k+5)\)). Further, the elements \(A\) of the ring satisfying the condition \(A'A=c_{A'}A\) for every element \(A'\) of the ring have, up to a scalar factor, the form
\[ A=\pm\sqrt{k\pi(-1)}\sum \pi(\lambda)A_\lambda+\sum\pi(\lambda)B_\lambda, \]
where \(\pi(\lambda')=1,\ \pi(\lambda'')=-1\) (the notation \(\lambda',\lambda''\) is the same as above), or the form
\[ A=\sum_\lambda(A_\lambda+B_\lambda),\qquad A=\sum_\lambda(-kA_\lambda+B_\lambda). \]
From the formulated results it follows immediately: the ring \(R_1\) is a direct sum of \(\tfrac12(k+5)\) full matrix rings, of which 4 summands are matrices of order 1, and the remaining \(\tfrac12(k-3)\) summands are matrices of order 2.

Now let \(\chi\not\equiv1\). Then the elements of the ring \(R_\chi\)—the functions \(\varphi(g)\)—are equal to zero on double cosets with respect to \(Z\) having representatives
\[ \begin{pmatrix} \lambda^{-1} & 0\\ 0 & \lambda \end{pmatrix}, \quad \lambda\ne\pm1. \]
We define in the space \(R_\chi\) a basis \(A_1,A_{-1},B_\lambda\). Each of the functions \(A_1,A_{-1},B_\lambda\) is determined by the following conditions: it is equal to 1 respectively on the matrix
\[ \begin{pmatrix} 1 & 0\\ 0 & 1 \end{pmatrix}, \quad \begin{pmatrix} -1 & 0\\ 0 & -1 \end{pmatrix}, \quad\text{or}\quad \begin{pmatrix} 0 & -\lambda^{-1}\\ \lambda & 0 \end{pmatrix} \]
and is nonzero only on the double coset with respect to \(Z\) containing this matrix. The multiplication operation in \(R_\chi,\ \chi\not\equiv1\), is given by formu-

* We note that the rings \(R_\chi\) are semisimple, i.e., decompose into a direct sum of full matrix rings (this follows from the complete reducibility of representations of a finite group).

by
\[ A_{-1}A_{-1}=A_1,\quad B_\lambda A_{-1}=A_{-1}B_\lambda=B_{-\lambda},\quad B_{\lambda_1}B_{\lambda_2}=\sum_\lambda \chi\left(\frac{\lambda}{\lambda_1\lambda_2}+\frac{\lambda_1}{\lambda\lambda_2}+ \frac{\lambda_2}{\lambda\lambda_1}\right)B_\lambda+k\delta_{\lambda_1+\lambda_2}A_1+k\delta_{\lambda_1-\lambda_2}A_{-1}, \]
\(A_1\) is the identity of the ring. (Here \(\delta_t=1\) for \(t=0\), \(\delta_t=0\) for \(t\ne0\).) Hence we conclude that for \(\chi\ne1\) the ring \(R_\chi\) is commutative and has dimension \(k+1\).

Taking into account the relations (1), it is easy to determine the dimensions of the spaces \(R_{\chi_2\chi_1}\), \(\chi_1\ne\chi_2\). If the characters \(\chi_1(t)\) and \(\chi_2(t)\) are related by
\[ \chi_2(t)=\chi_1(\lambda^2 t) \]
for some \(\lambda\ne0\), then the dimension of \(R_{\chi_2\chi_1}\) is equal to \(k+1\); otherwise the dimension of \(R_{\chi_2\chi_1}\) is equal to \(k-1\).

1. Classification of irreducible representations of the group \(G\). Let us divide the representations \(T_\chi\), \(\chi\ne1\), into two classes; we assign \(T_{\chi_1}\) and \(T_{\chi_2}\) to the same class if
   \[ \chi_2(t)=\chi_1(\lambda^2t) \]
   for some \(\lambda\ne0\). On the basis of §3 we obtain: the representation \(T_\chi\), \(\chi\ne1\), is the direct sum of \(k+1\) pairwise inequivalent representations. If the representations \(T_{\chi_1}\) and \(T_{\chi_2}\) belong to the same class, then they are equivalent; but if \(T_{\chi_1}\) and \(T_{\chi_2}\) belong to different classes, then they have \(k-1\) common irreducible summands. The representation \(T_1\), corresponding to the character \(\chi=1\), contains 4 irreducible representations with multiplicity 1 and \(\frac12(k-3)\) irreducible representations with multiplicity 2. The representations \(T_\chi\), \(\chi\ne1\), and \(T_1\) have \(\frac12(k+1)\) common irreducible components; of these, two occur in \(T_1\) with multiplicity 1, and the remaining ones with multiplicity 2. We can now classify all irreducible representations of the group \(G\) according to the multiplicities with which they occur in the representations \(T_\chi\). (We note that the regular representation of the group \(G\) is the direct sum of the representations \(T_\chi\); hence every irreducible representation of the group \(G\) is contained in at least one of the representations \(T_\chi\).)

We obtain the following types of irreducible representations: 1) representations of the “principal” series; they occur in all \(T_\chi\), and in \(T_1\) they occur with multiplicity 2; the number of such representations is \(\frac12(k-3)\), the dimension of each is \(k+1\). 2) Representations of the “analytic” series; they occur in all \(T_\chi\) except \(T_1\); the number of such representations is \(\frac12(k-1)\), the dimension of each is \(k-1\). 3) One representation of dimension \(k\); it occurs in all \(T_\chi\), and in \(T_1\) it occurs with multiplicity 1. 4) Two representations of dimension \(\frac12(k+1)\); each of them occurs in the representations \(T_\chi\), \(\chi\ne1\), belonging to only one of the two classes; in \(T_1\) they occur with multiplicity 1. 5) Two representations of dimension \(\frac12(k-1)\). Each of them occurs in the representation \(T_\chi\), \(\chi\ne1\), belonging to only one of the two classes, and does not occur in \(T_1\). 6) The identity representation; it is contained only in \(T_1\).

1. All irreducible representations of the group \(G\) can be obtained by decomposing the spaces \(H_\chi\) into irreducible subspaces. In the case \(\chi\equiv1\), the irreducible subspaces are the subspaces of homogeneous functions of a given degree of homogeneity \(\pi\) (\(\pi(t)\) is a multiplicative character):
   \[ f(tx_1,tx_2)=\pi(t)f(x_1,x_2) \]
   for any \(t\ne0\)*. We note that in these subspaces only one half of all irreducible representations is realized. Here a description will be given of all irreducible subspaces of the space \(H_\chi\), \(\chi\ne1\). It is interesting that the analogues of homogeneous functions for \(\chi\ne1\) are Bessel functions.

Consider functions \(\varphi(g)\in R_\chi\) satisfying the condition
\[ \psi(g)*\varphi(g)=c_\psi\varphi(g) \tag{2} \]
for every \(\psi\in R_\chi\); \(c_\psi\) is a complex number (the mapping \(\psi\mapsto c_\psi\) is a homomorphism of the ring \(R_\chi\)). The functions \(\varphi(g)\) will be normalized by the condition \(\varphi(e)=1\), where \(e\) is the identity of the group. Between such functions \(\varphi(g)\) and irreducible subspaces of the space \(H_\chi\), \(\chi\ne1\), there exist—

* Not counting the special cases when \(\pi(t)\equiv1\) or \(\pi(t)=\pm1\). In each of these cases the space of homogeneous functions is the sum of two irreducible subspaces.

there is a one-to-one correspondence. The irreducible subspace corresponding to the function \(\varphi(g)\) consists of all functions representable in the form \(\varphi(g) * f(g)\), where \(f(g)\) ranges over \(H_\chi\). The problem is to describe all functions \(\varphi(g)\) satisfying condition (2). This description is given below.

In view of (1), it is enough for us to know the values of the functions \(\varphi(g)\) only on the matrices \(\pm e\) and
\[ \begin{pmatrix} 0 & -\lambda^{-1}\\ \lambda & 0 \end{pmatrix}. \]

Denote by \(\varphi(\pm e)\) and \(\varphi(\lambda)\) the values of the function \(\varphi(g)\) on these matrices. Then for \(\varphi(\lambda)\) we obtain the following functional equation:
\[ k\varphi(\lambda_1)\varphi(\lambda_2) = \varphi(-e)\sum_{\lambda}\chi\left( \frac{\lambda}{\lambda_1\lambda_2} + \frac{\lambda_1}{\lambda\lambda_2} + \frac{\lambda_2}{\lambda\lambda_1} \right)\varphi(\lambda) + \]
\[ + \delta_{\lambda_1-\lambda_2}\varphi(-e) + \delta_{\lambda_1+\lambda_2}\varphi(e) \tag{3} \]
for any \(\lambda_1,\lambda_2\ne 0\); \(\varphi(-e)\varphi(\lambda)=\varphi(-\lambda)\); \(\varphi(-e)=\pm\varphi(e)\), where \(\varphi(e)=1\). There are two classes of solutions of equation (3). The functions of the first class are given by the formula
\[ \varphi(\lambda)=J_\pi(\lambda;\chi)\equiv \frac{1}{k}\sum_t \chi\left(-\lambda^{-1}(t+t^{-1})\right)\pi(t); \qquad \varphi(-e)=\pi(-1), \]
where \(\pi(t)\) is a multiplicative character on \(K\); the summation is over all \(t\ne 0\) from \(K\). To describe the second class of solutions of equation (3), extend the field \(K\) by adjoining to it one square root. In the resulting extension consider the set \(K_1\) of elements “equal in modulus to one” (i.e., such that the product of an element by its conjugate is equal to 1). This set is a multiplicative group of order \(k+1\). The functions of the second class are given by the formula
\[ \varphi(\lambda)=K_\pi(\lambda;\chi)= -\frac{1}{k}\sum_t \chi\left(-\lambda^{-1}(t+t^{-1})\right)\pi(t); \qquad \varphi(-e)=\pi(-1), \]
where \(\pi(t)\) is a multiplicative character on \(K_1\), \(\pi\ne 1\); the summation is over all \(t\ne 0\) from \(K_1\). The functions \(J_\pi(\lambda;\chi)\) and \(K_\pi(\lambda;\chi)\) are naturally called Bessel functions over the finite field \(K\)*. (The formulas for \(J_\pi(\lambda;\chi)\) and \(K_\pi(\lambda;\chi)\) are analogous to the formulas of the integral representation of ordinary Bessel functions.)

1. Let us formulate analogous results for the group \(G_B\) of unimodular matrices of the second order over the field of real numbers. It can be shown that also for this group the ring \(R_\chi\), \(\chi(t)=e^{i\sigma t}\), is commutative for \(\chi\ne 1\). In this case the decomposition of the representation \(T_\chi\) into irreducible representations contains all irreducible representations of the principal continuous series and half of the irreducible representations of the discrete series. (The other half of the representations of the discrete series enters into the decomposition of the representation \(T_{\bar\chi}\), \(\bar\chi(t)=e^{-i\sigma t}\).) The realization of irreducible unitary representations (of the principal series) can be carried out in the same way as in § 5 for the case of a finite field. The determination of functions \(\varphi(g)\) satisfying condition (2) is easily reduced to solving a differential equation of the form
   \[ \lambda^2\varphi''(\lambda)+3\lambda\varphi'(\lambda)+\left(4\sigma^2/\lambda^2-\nu\right)\varphi(\lambda)=0; \]
   the solutions of this equation are the functions \(\lambda^{-1}J_\alpha(2\sigma/\lambda)\), \(\alpha=\pm\sqrt{1+\nu}\), where \(J_\alpha(x)\) is the Bessel function.

Received
3 VII 1962

CITED LITERATURE

\(^{1}\) Mackey, Ann. Math., 55, 101 (1952).
\(^{2}\) Frobenius, Theory of Characters and Representations of Groups, Kharkov, 1937.
\(^{3}\) Hecke, Abh. Math. Seminar Hamburg, 6, 235 (1928).

* Note that the functions of the first class correspond to irreducible representations contained in \(\tilde T_1\), while the functions of the second class correspond to those not contained in \(T_1\).