UDC 519.46

MATHEMATICS

I. A. ZHIGULIN

ON SOME CLASSES OF ORBITS FOR THE GROUP OF UNITRIANGULAR MATRICES

(Presented by Academician I. G. Petrovskii on 25 II 1966)

The problem of classifying orbits arises in connection with the study of irreducible unitary representations of one-connected nilpotent Lie groups \((^{1})\).

Let \(\mathfrak G_n\) be the group of upper unitriangular matrices of order \(n\), and let \(G_n\) be its Lie algebra. The space \(G_n'\), conjugate to \(G_n\), is identified with the space of lower triangular matrices

\[ x=\|x^{\alpha\beta}\|, \tag{1} \]

where \(x^{\alpha\beta}=0\) for \(\alpha \leq \beta\). In the space \(G_n'\) there is defined the representation \(\rho'(\mathfrak G_n)\) of the group \(\mathfrak G_n\), dual to its adjoint representation \(\rho(\mathfrak G_n)\) in \(G_n\), by the formula

\[ \rho'(g)x=[gxg^{-1}]_{\mathrm n} \tag{2} \]

\((g\in \mathfrak G_n,\ x\in G_n')\). In (2) the symbol \([gxg^{-1}]_{\mathrm n}\) denotes the lower triangular matrix whose entries lying below the main diagonal coincide with the corresponding entries of the matrix \(gxg^{-1}\). The orbits in \(G_n'\) with respect to \(\rho'(\mathfrak G_n)\) are in one-to-one correspondence with the irreducible unitary representations of the group \(\mathfrak G_n\).

We assign two orbits to the same class if, as algebraic varieties, they are constructed in the same way.

In \((^{1})\) a class of orbits was indicated, called orbits of general position and given by the equations

\[ \Delta_\nu(x)=c_\nu\ne 0 \quad (\nu=1,2,\ldots,[n/2]), \tag{3} \]

where \(\Delta_\nu(x)\) is the determinant of the minor of order \(\nu\) in the lower left corner of the matrix \(x\).

In the present note we study those classes of orbits which are obtained when among the constants \(c_\nu\) occurring in (3) there are constants equal to zero.

In the space \(G_n'\) fix a basis \(B_0\) consisting of the coordinate dyads \(e_{\alpha\beta}\) \((\alpha>\beta)\), i.e., of matrices with a unit at the intersection of the \(\alpha\)-th row and the \(\beta\)-th column and zeros in all other positions.

Denote \(g=\|g_{\alpha\beta}\|\) and \(g^{-1}=\|h_{\alpha\beta}\|\). The elements of the basis \(B_0\) are transformed according to the law

\[ e'_{\alpha\beta} = \rho'(g)e_{\alpha\beta} = \sum_{\gamma=\beta+1}^{\alpha} \sum_{\delta=\beta}^{\alpha-1} g_{\gamma\alpha}h_{\beta\delta}e_{\gamma\delta}. \tag{4} \]

The coordinates \(\widetilde{x}^{\alpha\beta}\) of the image \(\widetilde{x}\) of an element \(x\in G_n'\) under transformations from \(\rho'(\mathfrak G_n)\) are found by the formula

\[ \widetilde{x}^{\alpha\beta} = \sum_{\gamma=\alpha}^{n} \sum_{\delta=1}^{\beta} g_{\alpha\gamma}x^{\gamma\delta}h_{\delta\beta}. \tag{5} \]

For a fixed pair of elements \(x=\|x^{\alpha\beta}\|\) and \(\tilde{x}=\|\tilde{x}^{\alpha\beta}\|\), formulas (5) may be regarded as a system of equations in \(g_{\alpha\gamma}\) and \(h_{\delta\beta}\). Then the following holds.

Proposition 1. In order that the elements \(x\) and \(\tilde{x}\) belong to the same orbit with respect to \(\rho'(\mathfrak{G}_n)\), it is necessary and sufficient that the system of equations (5), with unknowns \(g_{\alpha\gamma}\) and \(h_{\delta\beta}\), be consistent.

Thus the problem of classifying the orbits reduces to the study of the consistency of system (5).

We shall begin the study of system (5) for consistency by considering those of its equations for which

\[ \alpha=n,n-1,\ldots,n-[n/2]+1;\qquad \beta=1,2,\ldots,[n/2]. \]

We arrange the selected equations according to the following principle: first we write the equations containing \(\tilde{x}^{\alpha\beta}\) that enter the minor \(\Delta_1(\tilde{x})\); second, those containing the \(\tilde{x}^{\alpha\beta}\) which supplement \(\Delta_1(\tilde{x})\) to \(\Delta_2(\tilde{x})\), and so on. We obtain the system of equations (6), divided into \([n/2]\) subsystems:

\[ \begin{aligned} &1\left\{ \tilde{x}^{n1}=x^{n1}; \right.\\[4pt] &2\left\{ \begin{aligned} (2a)\quad &\tilde{x}^{\,n-1,1}=x^{\,n-1,1}+g_{n-1,n}x^{n1},\\ (2b)\quad &\tilde{x}^{\,n2}=x^{n2}+x^{n1}h_{12},\\ &\tilde{x}^{\,n-1,2}=\sum_{\gamma=n-1}^{n}\sum_{\delta=1}^{2} g_{n-1,\gamma}x^{\gamma\delta}h_{\delta2}, \end{aligned} \right.\\[-2pt] &\hspace{2.5em}\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots \tag{6}\\ &\left. \begin{aligned} &\left([\tfrac n2]\,a\right)\left\{ \begin{aligned} &\tilde{x}^{\,n-[n/2]+1,1} =\sum_{\gamma=n-[n/2]+1}^{n} g_{\,n-[n/2]+1,\gamma}x^{\gamma1},\\ &\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\\ &\tilde{x}^{\,n-[n/2]+1,[n/2]-1} =\sum_{\gamma=n-[n/2]+1}^{n}\sum_{\delta=1}^{[n/2]-1} g_{\,n-[n/2]+1,\gamma}x^{\gamma\delta}h_{\delta,[n/2]-1}, \end{aligned}\right.\\ &\left([\tfrac n2]\,b\right)\left\{ \begin{aligned} &\tilde{x}^{\,n,[n/2]} =\sum_{\delta=1}^{[n/2]}x^{n\delta}h_{\delta,[n/2]},\\ &\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\\ &\tilde{x}^{\,n-[n/2]+2,[n/2]} =\sum_{\gamma=n-[n/2]+2}^{n}\sum_{\delta=1}^{[n/2]} g_{\,n-[n/2]+2,\gamma}x^{\gamma\delta}h_{\delta,[n/2]}, \end{aligned}\right.\\ &\tilde{x}^{\,n-[n/2]+1,[n/2]} =\sum_{\gamma=n-[n/2]+1}^{n}\sum_{\delta=1}^{[n/2]} g_{\,n-[n/2]+1,\gamma}x^{\gamma\delta}h_{\delta,[n/2]}. \end{aligned} \right\} \end{aligned} \]

Let us note that subsystem 1 consists of the single invariant relation

\[ \rho'(g)x^{n1}=x^{n1}. \]

In what follows it is assumed everywhere that \(x^{n1}\ne0\).

First of all we point out the following circumstance: if the subsystems \(2,3,\ldots,(l-1)\) are consistent, then, substituting their solutions into the \(l\)-th one, we obtain a system of linear equations in \(g_{n-l+1,\gamma}\) \((\gamma=n-l+2,\ldots,n)\) and \(h_{\delta l}\) \((\delta=1,2,\ldots,l-1)\).

Suppose that all the first \((l-1)\) subsystems are consistent. Substitute one of the solutions of these \((l-1)\) subsystems into the \(l\)-th one. The system of \(2l-1\) linear equations thus obtained is divided into two subsystems: the subsystem \(l_a\) with respect to \(g_{n-l+1,\gamma}\) \((\gamma=n-l+2,\ldots,n)\) and

subsystem \(l_\delta\) with respect to \(h_{\delta l}\) \((\delta=1,2,\ldots,l-1)\). The subsystems \(l_a\) and \(l_\delta\) have a common matrix of coefficients at the unknowns, which we denote by \(A_l\). It can be shown that

\[ |A_l|=\Delta_{l-1}(x). \tag{7} \]

From (7) it follows

Proposition 2. In order that system (6) have a unique solution, it is necessary and sufficient that

\[ \Delta_\nu(x)\ne 0 \]

for all \(\nu=1,2,\ldots,[n/2]-1\).

For the case when \(\Delta_{l-1}(x)=0\), one can prove the following condition for the compatibility of the systems \(l_a\) and \(l_\delta\):

Proposition 3. Let \(\Delta_{l-1}(x)=0\). In order that the subsystem \(l_a(l_\delta)\) be compatible, it is necessary and sufficient that, in the matrices \(x\) and \(\widetilde{x}\), the elements in the \((n-l+1)\)-st row up to and including the \((l-1)\)-st column (in the \(l\)-th column up to and including the \((n-l+2)\)-nd row) be linear combinations of the rows (columns), respectively, of the minors \(\Delta_{l-1}(x)\) and \(\Delta_{l-1}(\widetilde{x})\).

From Proposition 3 there follows

Corollary. Let \(\Delta_l(x)=0\) for some \(2\le l<[n/2]\). In order that system (6) be compatible, it is necessary that \(\Delta_k(x)=0\) for all \(k>l\).

In the \(l\)-th subsystem the last equation

\[ \widetilde{x}^{\,n-l+1,l} = \sum_{\gamma=n-l+1}^{n} \sum_{\delta=1}^{l} g_{n-l+1,\gamma}x^{\gamma\delta}h_{\delta l} \tag{8} \]

relates the solutions of the subsystems in \(l_a\) and \(l_\delta\). When \(\Delta_{l-1}(x)\ne0\), this condition is equivalent to the condition

\[ \Delta_l(x)=\Delta_l(\widetilde{x}). \tag{9} \]

When \(\Delta_{l-1}(x)=0\), condition (8) is not equivalent to condition (9), since, if the subsystem \(l\) is compatible, then \(\Delta_l(x)=\Delta_l(\widetilde{x})=0\) for arbitrary values of \(x^{n-l+1,l}\) for \(x\) and \(x^{n-l+1,l}\) for \(\widetilde{x}\). In this case the infinite set of solutions of the subsystem \(l\) consists of those pairs of solutions of the subsystems \(l_a\) and \(l_\delta\) which satisfy condition (8).

We have investigated for compatibility only a part of system (5). Let us consider the remaining equations

\[ \widetilde{x}^{\alpha\beta} = \sum_{\gamma=\alpha}^{n} \sum_{\delta=1}^{\beta} g_{\alpha\gamma}x^{\gamma\beta}h_{\delta\beta}, \tag{10} \]

containing those \(\widetilde{x}^{\alpha\beta}\) for which

\[ \alpha\le n-[n/2]\quad \text{when } \beta\le [n/2]; \tag{10a} \]

\[ \alpha\ge n-[n/2]+1\quad \text{when } \beta\ge [n/2]+1. \tag{10b} \]

Substituting one of the solutions of system (6) into system (10), we obtain a system of linear equations with respect to \(g_{\alpha\gamma}\) under conditions (10a) and with respect to \(h_{\delta\beta}\) under conditions (10b). For each fixed value of \(\alpha\) and \(\beta\) we obtain subsystems, and the compatibility of each such

* It is assumed that, in \(\Delta_{l-1}(x)\) and in \(\Delta_{l-1}(\widetilde{x})\), the \((l-1)\)-st column and the \((n-l+2)\)-nd row are nonzero.

** We shall say that the elements \(x^{\alpha\beta}\) of the \(\alpha\)-th row of the matrix \(x\) are a linear combination of the rows following it if
\[ x^{\alpha\beta}=\sum_{a=\alpha+1}^{n}q_a x^{a\beta} \]
(\(q\) are numbers).

subsystems does not depend on the compatibility of other subsystems in this series.

With regard to the compatibility of these subsystems, the following two propositions are valid.

Proposition 4. Every subsystem of system (10) is compatible if:

a) \(\Delta_\nu(x) \ne 0\) for \(\nu = 1, 2, \ldots, [n/2]\) when \(n\) is odd;

b) \(\Delta_\nu(x) \ne 0\) for \(\nu = 1, 2, \ldots, [n/2]-1\) when \(n\) is even.

Proposition 5. Suppose that \(\Delta_l(x)=0\) for some \(l\), and \(\Delta_m(x)\ne 0\) for all \(m<l\). The subsystems of system (10) with respect to \(g_{\alpha\gamma}\) for \(\alpha \le l\) (\(h_{\delta\beta}\) for \(\beta > n-l\)) are always compatible. In order that the subsystem of system (10) with respect to \(g_{\alpha\gamma}\) for \(l+1 \le \alpha \le n-[n/2]\) (\(h_{\delta\beta}\) for \([n/2]+1 \le \beta \le n-l\)) be compatible, it is necessary and sufficient that the elements of the \(\alpha\)-th row (the \(\beta\)-th column) of the matrices \(x\) and \(\tilde x\) be a linear combination of all subsequent rows (preceding columns).

From Propositions 2 and 4 it follows:

Theorem 1. The orbit of general position is given by the equations

\[ \Delta_\nu(x)=c_\nu \qquad (\nu=1,2,\ldots,[n/2]), \]

where \(\Delta_\nu(x)\) is the determinant of the minor of order \(\nu\) in the left lower corner of the matrix \(x\), and

a) \(c_\nu \ne 0\) for all \(\nu\) when \(n\) is odd;

b) \(c_\nu \ne 0\) for \(\nu=1,2,\ldots,[n/2]-1\) when \(n\) is even.

From Propositions 3 and 5 it follows:

Theorem 2. Suppose that \(x\) and \(x_1\) are such elements in \(G_n'\) that \(\Delta_l(x)=\Delta_l(x_1)=0\) for some \(2 \le l \le [n/2]\), and \(\Delta_\nu(x)\ne 0\) and \(\Delta_\nu(x_1)\ne 0\) for all \(\nu<l\). The elements \(x\) and \(x_1\) belong to one orbit if and only if

\[ \Delta_\nu(x)=\Delta_\nu(x_1) \qquad (\nu=1,2,\ldots,l-1) \]

and in the matrices \(x\) and \(x_1\) the elements of the \(\alpha\)-th \((l+1 \le \alpha \le n-l)\) row (of the \(\beta\)-th \((l+1 \le \beta \le n-l)\) column) are expressed as a linear combination of all subsequent rows (preceding columns).

Obviously, the results presented here also hold for orbits in invariant subspaces introduced by me in \({}^{2}\).

Moscow State Pedagogical Institute
named after V. I. Lenin

Received
16 II 1966

CITED LITERATURE

\({}^{1}\) A. A. Kirillov, UMN, 17, no. 4, 57 (1962). \({}^{2}\) I. A. Zhigulin, DAN, 164, No. 1, 33 (1965).