MATHEMATICS

M. I. GRAEV

IRREDUCIBLE UNITARY REPRESENTATIONS OF THE GROUP OF THIRD-ORDER MATRICES PRESERVING AN INDEFINITE HERMITIAN FORM

(Presented by Academician A. N. Kolmogorov on 15 XI 1956)

1. In our note \((^1)\) certain series of irreducible unitary representations of the group of unimodular matrices of order \(n\) preserving an indefinite Hermitian form were described. In the present note, for the case of the group of matrices of order 3, a complete description is given of all irreducible unitary representations occurring in the decomposition of the regular representation.*

We consider the group \(G\) of unimodular matrices of order 3 preserving the Hermitian form
\[ H(z,\bar z)=-z_1\bar z_1-z_2\bar z_2+z_3\bar z_3. \]
The most general approach to the description of all irreducible unitary representations of the group \(G\) is as follows.

Denote by \(M\) the manifold of points \(\zeta=(\zeta_1,\zeta_2)\) of the two-dimensional complex space for which \(|\zeta_1|^2+|\zeta_2|^2=1\). Let \(\mathcal H\) be the space of continuous functions on \(M\). To each matrix \(g=\|g_{ij}\|\) from \(G\) we associate a transformation of the manifold \(M\):
\(\zeta\to \zeta_g=(\zeta'_1,\zeta'_2)\), where
\[ \zeta'_1=\frac{\zeta_1 g_{11}+\zeta_2 g_{21}+g_{31}} {\zeta_1 g_{13}+\zeta_2 g_{23}+g_{33}}; \qquad \zeta'_2=\frac{\zeta_1 g_{12}+\zeta_2 g_{22}+g_{32}} {\zeta_1 g_{13}+\zeta_2 g_{23}+g_{33}}. \]

The operators of the representation \(T_g\) may initially be defined in the space \(\mathcal H\) by the formula
\[ T_g f(\zeta)= f(\zeta_{\bar g}) (\zeta_1 g_{13}+\zeta_2 g_{23}+g_{33})^{-\sigma} \overline{(\zeta_1 g_{13}+\zeta_2 g_{23}+g_{33})}^{-\tau}. \tag{1} \]
Here \(\sigma\) and \(\tau\) are complex numbers characterizing the representation. From considerations of single-valuedness of expression (1), the difference \(\sigma-\tau\) is assumed to be an integer.

If \(\sigma\) and \(\tau\) are not real integers, then the space \(\mathcal H\) has no proper invariant subspaces. In this case, in \(\mathcal H\) one can define, uniquely up to a constant factor, a bilinear functional \(A(f_1,\bar f_2)\) that commutes with the operators \(T_g\):
\[ A(T_g f_1,T_g f_2)=A(f_1,f_2). \]
For certain values of \(\sigma,\tau\) this functional turns out to be positive definite. Then, defining the scalar product in \(\mathcal H\) by means of the bilinear functional \(A\) and completing \(\mathcal H\), we obtain a Hilbert space on which an irreducible unitary representation of the group \(G\) is realized. In this way all nondiscrete (principal and supplementary) series of irreducible unitary representations of the group \(G\) are constructed.

If \(\sigma\) and \(\tau\) are real integers, then the space \(\mathcal H\) always contains proper subspaces invariant with respect to the operators \(T_g\). In this case, in \(\mathcal H\) one can single out such irreducible

* The irreducible unitary representations of the group of matrices of order 2 were studied in \((^2)\).

invariant subspaces on which the bilinear functional, permutable with the operators \(T_g\), is sign-definite. In this way one can obtain all the discrete series of irreducible unitary representations of the group \(G\). However, the direct determination of such subspaces of the space \(\mathcal H\) presents certain difficulties. Below we give another approach to the description of the discrete series of unitary representations of the group \(G\).

2. We shall regard \(G\) as a group of linear transformations of the three-dimensional complex vector space of vectors \(z=(z_1,z_2,z_3)\); the vector \(z'\) into which the vector \(z\) is transformed by the matrix \(g\) will be denoted by \(zg\).

A function \(f(z)\), given in the domain where
\[ -|z_1|^2-|z_2|^2+|z_3|^2 \geq 0, \]
or in the domain where
\[ -|z_1|^2-|z_2|^2+|z_3|^2 \leq 0, \]
will be called harmonic if it satisfies the equation
\[ \left( -\frac{\partial^2}{\partial z_1\partial \bar z_1} -\frac{\partial^2}{\partial z_2\partial \bar z_2} +\frac{\partial^2}{\partial z_3\partial \bar z_3} \right)f=0. \tag{2} \]

We shall say that \(f(z)\) is a homogeneous function of type \((\sigma,\tau)\) if \(f(z)\) satisfies the following homogeneity condition:
\[ f(tz)=t^{-\bar\sigma}t^{-\tau}f(z). \tag{3} \]
For reasons of single-valuedness it is always assumed here that \(\sigma-\tau\) is an integer.

Let \(H^+_{\sigma,\tau}\) be the set of all harmonic functions \(f(z)\) of type \((\sigma,\tau)\), defined in the domain where
\[ -|z_1|^2-|z_2|^2+|z_3|^2 \geq 0, \]
for which
\[ \|f\|^2= \int_{H(z,z)=c} |f(z)|^2\,d\mu(z). \tag{4} \]
Here the integration is carried out over the hyperboloid
\[ -|z_1|^2-|z_2|^2+|z_3|^2=c, \]
where \(c>0\), and \(d\mu(z)\) denotes the measure on the hyperboloid invariant with respect to transformations of the group \(G\).

Completing \(H^+_{\sigma,\tau}\) in the norm (4) and introducing a scalar product in the corresponding way, we obtain a Hilbert space denoted by \(\bar H^+_{\sigma,\tau}\). In an analogous manner, considering functions in the domain where
\[ -|z_1|^2-|z_2|^2+|z_3|^2 \leq 0, \]
we define the spaces \(H^-_{\sigma,\tau}\) and \(\bar H^-_{\sigma,\tau}\).

Putting
\[ w_1=\frac{z_1}{z_3},\qquad w_2=\frac{z_2}{z_3}, \]
we represent a function \(f(z)\) of type \((\sigma,\tau)\) in the form
\[ f(z)=z_3^{-\bar\sigma}\bar z_3^{-\tau}\varphi(w_1,w_2). \tag{5} \]
Therefore, instead of functions of type \((\sigma,\tau)\), one may consider functions \(\varphi(w)=\varphi(w_1,w_2)\), defined in the domain
\[ |w_1|^2+|w_2|^2 \leq 1 \]
or in the domain
\[ |w_1|^2+|w_2|^2 \geq 1 \]
of the two-dimensional complex projective space. The norm in \(H^\pm_{\sigma,\tau}\), in terms of the functions \(\varphi(w)\), takes the form
\[ \|\varphi\|^2= \int |\varphi(w)|^2\left|1-|w_1|^2-|w_2|^2\right|^{\operatorname{Re}(\sigma+\tau)-3}\,dw_1dw_2, \tag{6} \]
where the integration is carried out respectively over the domain
\[ |w_1|^2+|w_2|^2 \leq 1 \]
or over the domain
\[ |w_1|^2+|w_2|^2 \geq 1. \]

Substituting (5) into equation (2), we obtain the equation for the functions \(\varphi(w)\):
\[ \left\{ (1-|w_1|^2)\frac{\partial^2}{\partial w_1\partial \bar w_1} +(1-|w_2|^2)\frac{\partial^2}{\partial w_2\partial \bar w_2} -w_1\bar w_2\frac{\partial^2}{\partial w_1\partial \bar w_2} -\bar w_1w_2\frac{\partial^2}{\partial \bar w_1\partial w_2} \right. \]
\[ \left. -\tau\left(w_1\frac{\partial}{\partial w_1}+w_2\frac{\partial}{\partial w_2}\right) -\sigma\left(\bar w_1\frac{\partial}{\partial \bar w_1}+\bar w_2\frac{\partial}{\partial \bar w_2}\right) +\sigma\tau \right\}\varphi=0^*. \tag{7} \]

\[ \text{* When } \sigma=\tau=0,\text{ this equation defines functions studied in detail for the case of the domain } |w_1|^2+|w_2|^2<1 \text{ in the work of Lo Ken Hua }(3). \]

For reasons of convergence of the integral (6), we shall henceforth assume that \(\operatorname{Re}(\sigma+\tau)>2\).

Theorem 1. If \(\sigma\) and \(\tau\) are not integers, then \(H_{\sigma,\tau}^{+}=0\), \(H_{\sigma,\tau}^{-}=0\). If \(\sigma\) and \(\tau\) are integers, then \(H_{\sigma,\tau}^{-}\ne0\), \(H_{\sigma,\tau}^{+}\ne0\) if and only if \(\sigma\leq0\) or \(\tau\leq0\).

Theorem 2. The functions \(\varphi(w)\in H_{\sigma,\tau}^{+}\) and \(\varphi(w)\in H_{\sigma,\tau}^{-}\) are uniquely determined by their boundary values on the manifold \(M:\ |w_1|^2+|w_2|^2=1\).

In the space \(\bar H_{\sigma,\tau}^{\pm}\ne0\) one can define a unitary representation \(T_g\) of the group \(G\) by the formula

\[ T_g f(z)=f(zg). \tag{8} \]

On passing to the functions \(\varphi(w)\), formula (8) for the operator \(T_g\) takes the form

\[ T_g\varphi(w)=\varphi(w\bar g)\,(w_1g_{13}+w_2g_{23}+g_{33})^{-\sigma} (\bar w_1g_{13}+\bar w_2g_{23}+g_{33})^{-\tau}. \tag{8'} \]

Theorem 3. The representations of the group \(G\) given in the nonzero spaces \(\bar H_{\sigma,\tau}^{+}\) and \(\bar H_{\sigma,\tau}^{-}\) by formula (8) are unitary and irreducible.

By virtue of Theorem 2, these representations may also be given in the space of functions on the manifold \(M\) that are boundary values of functions from \(\bar H_{\sigma,\tau}^{+}\) (or from \(\bar H_{\sigma,\tau}^{-}\)).

3. We now indicate how the representations described above decompose into irreducible representations of the maximal compact subgroup \(\mathfrak U\) of the group \(G\). The subgroup \(\mathfrak U\) of the group \(G\) is isomorphic to the full group of unitary matrices of order 2. Therefore the irreducible representations of the subgroup \(\mathfrak U\) are characterized by a pair of integers \(f_1\geq f_2\). The space \(H_{(f_1,f_2)}\) in which the corresponding representation acts has dimension \(f_1-f_2+1\).

Theorem 4. Each of the nonzero spaces \(\bar H_{\sigma,\tau}^{\pm}\) decomposes, with respect to the representations of the subgroup \(\mathfrak U\), into a direct sum of pairwise inequivalent subspaces \(H_{(f_1,f_2)}\). The indices \(f_1,f_2\) of the subspaces entering into the decomposition of \(\bar H_{\sigma,\tau}^{\pm}\) are given by the expressions

\[ f_1=2p-q+(\sigma-\tau),\qquad f_2=p-2q+(\sigma-\tau). \tag{9} \]

Here, if \(\sigma>0,\ \tau>0\), then \(p=\tau-1,\tau,\tau+1,\ldots;\ q=\sigma-1,\sigma,\sigma+1,\ldots\).* If \(\sigma\leq0\), then for \(H_{\sigma,\tau}^{+}\), \(p=0,1,\ldots,-\sigma\), while \(q\) runs through all nonnegative integers; for \(H_{\sigma,\tau}^{-}\), \(p=\tau-1,\tau,\tau+1,\ldots\), while \(q\) runs through all nonnegative integers. An analogous description is obtained also for \(\tau\leq0\).

The highest-weight vectors \(\varphi_{f_1,f_2}\) from \(H_{(f_1,f_2)}\) have the following form:

For the space \(\bar H_{\sigma,\tau}^{-}\ne0\),

\[ \varphi_{f_1,f_2} = w_1^p \bar w_2^q \left(|w_1|^2+|w_2|^2\right)^{-(p+q+1)} \times \]

\[ \times F\left(\tau-1-p,\sigma-1-q;\ \sigma+\tau-1;\ 1-|w_1|^2-|w_2|^2\right). \]

Here \(F\) is the hypergeometric function, which in the present case is a polynomial in \(1-|w_1|^2-|w_2|^2\) of degree \(\min(p-\tau+1,q-\sigma+1)\). The indices \(p,q\) are related to \(f_1,f_2\) by relation (9).

For the space \(H_{\sigma,\tau}^{+}\), where \(\sigma\leq0\),

\[ \varphi_{f_1,f_2} = w_1^p\bar w_2^q G_{-\sigma-p}\left(p+q+\sigma+\tau,\ p+q+2;\ |w_1|^2+|w_2|^2\right), \]

where \(G_{-\sigma-p}\) is a Jacobi polynomial.

\[ \text{* Recall that in this case the space } \bar H_{\sigma,\tau}^{-} \text{ is meant, since } \bar H_{\sigma,\tau}^{+}=0. \]

For the space \(\bar H_{\sigma,\tau}^{+}\), where \(\tau \leqslant 0\):

\[ \varphi_{f_1,f_2}=w_1^{p}\bar w_2^{q}G_{-\tau-q}\bigl(p+q+\sigma+\tau,\;p+q+2;\; |w_1|^2+|w_2|^2\bigr). \]

The representations realized in the spaces \(\bar H_{\sigma,\tau}^{+}\ne 0\) are pairwise inequivalent. They all contain an irreducible representation of the subgroup \(U\) of dimension \(1\). Another description of these representations was given in \((^1)\). The representations realized in \(\bar H_{\sigma,\tau}^{-}\) for \(\sigma>0\) and \(\tau>0\) are likewise pairwise inequivalent. On the other hand, for \(\sigma \leqslant 0\), the representations realized in the spaces \(\bar H_{\sigma,\tau}^{-}\) and \(\bar H_{2-\sigma,\sigma+\tau-1}^{-}\) are equivalent to one another (an analogous assertion holds for \(\tau \leqslant 0\)). All representations realized in the spaces \(\bar H_{\sigma,\tau}^{-}\) contain a unique representation of the subgroup \(U\) of lowest dimension, and this dimension is always greater than one.

Thus, three discrete series of irreducible unitary representations of the group \(G\) have been obtained: 1) representations realized in the spaces \(\bar H_{\sigma,\tau}^{+}\), where \(\sigma \leqslant 0\); 2) representations realized in the spaces \(\bar H_{\sigma,\tau}^{+}\), where \(\tau \leqslant 0\); 3) representations realized in the spaces \(\bar H_{\sigma,\tau}^{-}\), where \(\sigma>0,\ \tau>0\).

1. In the preceding arguments it was assumed throughout that \(\operatorname{Re}(\sigma+\tau)>2\). However, one may also consider the limiting case when \(\operatorname{Re}(\sigma+\tau)=2\), if \(\|\varphi\|^2\) is defined by the formula

\[ \|\varphi\|^2=\left\{\lim_{\lambda\to 2+0}(\lambda-2)\int |\varphi(w)|^2\left|1-|w_1|^2-|w_2|^2\right|^{\lambda-3}\,dw_1dw_2.\right. \tag{10} \]

The integral (10) then reduces to an integral over the boundary \(|\zeta_1|^2+|\zeta_2|^2=1\)

\[ \|\varphi\|^2=\int |\varphi(\zeta)|^2\,d\mu(\zeta), \]

where \(d\mu(\zeta)=c\,d|\zeta_1|^2\,d(\arg\zeta_1)\,d(\arg\zeta_2)\). In view of this, it is natural to define the representation in the space of functions given on the boundary. As a result we obtain the principal non-discrete series of irreducible unitary representations of the group \(G\), specified by the integer \(n=\sigma-\tau\) and the real number \(\rho=\operatorname{Im}(\sigma+\tau)\), which was described earlier in \((^1)\).

Theorem 5. The regular representation of the group \(G\) decomposes into the representations of the discrete series described above and of the principal non-discrete series.

Moscow Institute of Steel
named after I. V. Stalin

Received
17 IX 1956

CITED LITERATURE

\(^1\) M. I. Graev, DAN, 98, No. 4 (1954). \(^2\) V. Bargmann, Ann. Math., 48, 3 (1947). \(^3\) Hua Lo-keng, Acta Math. Sinica, 2 (1952); 5 (1955); 6 (1956) (Chinese) (the materials of these articles are also presented in a brochure published in English: Harmonic Analysis of the Classical Domain in the Study of Analytic Functions of Several Complex Variables).