UDC 512.88+519.46

MATHEMATICS

I. B. BRODSKII

ON INVARIANTS OF THE GROUP OF TRIANGULAR MATRICES OF ORDER \(n\) WITH ONES ON THE MAIN DIAGONAL

(Presented by Academician I. N. Vekua on 10 III 1969)

As is known (see \((^1)\)), the problem of describing all irreducible unitary representations of the group \(G\) of all upper triangular matrices of order \(n\) with ones on the main diagonal reduces to the following algebraic problem:

Let \(M\) be the space of lower triangular matrices of order \(n\) with zeros on the main diagonal. Denote by \(g\) and \(\mu\) elements of \(G\) and \(M\), respectively, and consider the transformation in \(M\)

\[ \rho(g):\quad \mu \to [g\mu g^{-1}]_{\text{lower}}=\mu_g, \tag{1} \]

where the matrix \([g\mu g^{-1}]_{\text{lower}}\) coincides in its lower triangular part with the matrix \(g\mu g^{-1}\), and in the remaining part with the zero matrix, i.e. \(\mu_g\in M\). The set \(\{\mu_g\mid g\in G\}\) is called the orbit of \(\mu\). It is required to classify the orbits in \(M\). Thus, as was proved by A. A. Kirillov in \((^1)\), the problem posed above will be solved.

Fig. 1

In the present note we consider the geometry determined in \(M\) by the group \(\rho(G)\). We note that the orbit defined above is analogous to a mixed tensor of rank two. On the other hand, there is an essential difference between an orbit and a tensor. Indeed, any coordinate of a tensor in a new basis is expressed, generally speaking, through all coordinates in the old basis. With an orbit the situation is different.

Lemma 1. Let \(\mu\in M\) and \(g\in G\). Then any lower step-like part of \(g\mu g^{-1}\) is a function of \(g\) and of the same step-like part of \(\mu\).

The proof follows immediately from the equalities \(g_{ij}=\delta_{ij}\ \forall i\ge j\).

We now consider the subspace \(M'\) consisting of matrices of the form represented in Fig. 1. Put \(M'=\{\mu'\}\) and consider the transformations in \(M'\)

\[ \mu' \to [g\mu' g^{-1}]'\quad (g\in G), \tag{2} \]

where \([g\mu' g^{-1}]'\) denotes the matrix whose \((ij)\)-th component is equal to \((g\mu' g^{-1})_{ij}\) if \((ij)\) indexes a component of the shaded part of \(\mu\), and is zero otherwise.

It follows from Lemma 1 that the transformations (2) define a representation of \(G\) in \(M'\). Therefore, alongside the orbits in \(M\) with respect to (1), it is natural to consider orbits in subspaces of type \(M'\) with respect to (2). These orbits are also analogous to a mixed tensor of rank 2. Thus one may suppose that in \(M\) there exist invariant operators analogous to contraction, alternation, and addition. Below we describe these invariant operators constructively.

* The precise meaning of the expression “lower step-like part” is clear from Fig. 1, where such a part consists of zeros.

We introduce the following notation. Let \(S\) be the set of pairs of integers numbering the lower triangular part of a matrix of order \(n\), and let
\[ \tau=\{(p_1q_1),(p_2q_2),\ldots,(p_rq_r)\} \]
be a nonempty subset of \(S\), whose elements satisfy the inequalities
\[ p_1<p_2<\cdots<p_r,\qquad q_1<q_2<\cdots<q_r. \]
Such subsets will be called broken lines.

By the broken line \(\tau(\mu)\) we shall mean the set of those \(\mu_{ij}\) for which either \(i=p_k\) and \(q_{k-1}\leq j\leq q_k\), or \(j=q_k\) and \(p_k\leq i\leq p_{k+1}\) \((1\leq k\leq r)\). Let
\[ \tau^0=\{(p_1^0q_1^0),(p_2^0q_2^0),\ldots,(p_{r_0}^0q_{r_0}^0)\} \]
be a broken line. We shall say that \(\tau^0\leq\tau\) if for any pair \((p_{k_0}^0q_{k_0}^0)\) there is at least one such pair \((p_kq_k)\) that
\[ p_{k_0}^0\geq p_k,\qquad q_{k_0}^0\leq q_k, \]
and for at least one \(k^0\) one of the inequalities is strict.

Let \(\tau^0\leq\tau\) and \(\xi=(\tau\tau^0)\). We shall call the \(\xi\)-section of \(\mu\) the set of components of \(\mu\) lying between the broken lines \(\tau(\mu)\) and \(\tau^0(\mu)\), including the boundaries. In the drawing such a section is shaded. Its upper boundary is \(\tau(\mu)\), its lower boundary \(\tau^0(\mu)\). Finally, by the symbol \([\mu]_\xi\) we shall denote the matrix whose \(\xi\)-section is equal to the \(\xi\)-section of \(\mu\), while the remaining components are equal to zero. Now, in order to emphasize the analogy between orbits and tensors, we give the following

Definition 1. The set \(\{[g\mu g^{-1}]_\xi\mid g\in G\}\) is called the \(\xi\)-tensor.

It is precisely for these objects that we shall seek analogues of the classical tensor operators. First we give an exact definition of an invariant operator.

Definition 2. Let \(M_\xi\) and \(M_{\xi'}\) be subspaces of \(M\), \(\bar M=\{\bar\mu\}\) the space of lower triangular matrices of order \(\bar n\), \(\bar G=\{\bar g\}\) the group of upper triangular matrices with ones on the main diagonal of the same order, and let \(\gamma(\mu,\mu')\) be a function on \(M_\xi\times M_{\xi'}\) with values in \(\bar M\). We shall say that the operator
\[ (\mu,\mu')\mapsto \gamma(\mu,\mu') \]
is invariant if there exists a pair of broken lines \(\bar\xi=(\bar\tau\bar\tau^0)\), defined for matrices in \(\bar M\), such that
\[ [\gamma([g\mu g^{-1}]_\xi,[g\mu' g^{-1}]_{\xi'})]_{\bar\xi} = [\bar g\,\gamma(\mu,\mu')\,\bar g^{-1}]_{\bar\xi}, \]
where \(\bar g\) depends on \(g\) and on \(\xi,\xi'\) (but not on \(\mu,\mu'\)), and \(\bar\xi\) depends only on \(\xi,\xi'\).

It follows easily from this definition that the product of invariant operators is itself an invariant operator.

We now describe the first invariant operator. It has no analogue in tensor algebra, but is a direct consequence of the triangularity of the matrices.

Lemma 2. Let \(\xi=(\tau\tau^0)\) be a pair of broken lines \((\tau^0\leq\tau)\), \(\mu\in M_\xi\), and let \(\bar\xi=(\bar\tau\bar\tau^0)\) be a pair of broken lines satisfying the inequalities
\[ \tau^0\leq\bar\tau^0\leq\bar\tau\leq\tau. \]
Then the operator \(\mu\mapsto[\mu]_{\bar\xi}\) is invariant.

The proof follows easily from Lemma 1. The operator \(\mu\mapsto[\mu]_{\bar\xi}\) will be called a \(\bar\xi\)-section, or sometimes simply a section.

The next invariant operator—the \(G\)-contraction—we introduce with the aid of the mapping
\[ (\mu,\mu')\mapsto \mu\mu'\qquad(\mu\in M_\xi,\ \mu'\in M_{\xi'}). \]
From the equality
\[ g\mu g^{-1}g\mu'g^{-1}=g\mu\mu'g^{-1} \]
there does not yet follow, generally speaking, the existence of such a \(\bar\xi\) that
\[ [[g\mu g^{-1}]_\xi [g\mu'g^{-1}]_{\xi'}]^{\bar\xi} = [g\mu\mu'g^{-1}]_{\bar\xi}; \]
however, if the \(\bar\xi\)-section of the matrix
\[ [g\mu g^{-1}]_\xi [g\mu'g^{-1}]_{\xi'},\qquad \forall g\in G, \]
depends only on the components of the \(\xi\)-section of \(g\mu g^{-1}\) and of the \(\xi'\)-section of \(g\mu'g^{-1}\), then the left-hand side of the last equality is
\[ [g\mu g^{-1}g\mu'g^{-1}]_{\bar\xi}=[g\mu\mu'g^{-1}]_{\bar\xi}, \]
i.e. the operator \((\mu,\mu')\mapsto[\mu\mu']_{\bar\xi}\) is invariant. The conditions that will be imposed below in Theorem 1 on \(\xi\) and \(\xi'\) are precisely those which guarantee dependence of this kind.

We introduce the following notation. Let
\[ \tau=\{(p_1q_1),(p_2q_2),\ldots,(p_rq_r)\} \]
be some broken line; denote by the symbol \(\tau(i)\) the number of the column in which the \(i\)-th row of the matrix \(\mu\) intersects the broken line \(\tau(\mu)\) when moving from left to right (here \(i>p_1\)), i.e.
\[ \tau(i)=\max_{k\in\{k\mid i>p_k\}} q_k. \]
Similarly, by the symbol \(\tau(j)\) we denote the number of the column in which the \(j\)-th column of \(\mu\) intersects the broken line \(\tau(\mu)\) when moving from bottom to top, i.e.
\[ \tau(j)=\min_{k\in\{k\mid j<q_k\}} p_k \]
(here \(j<q_r\)).

Theorem 1. Let \(\xi=(\tau\tau^0)\), \(\xi'=(\tau'\tau^{\prime 0})\). Suppose that there exists a broken line
\[ \bar\tau=\{(\bar p_1\bar q_1),(\bar p_2\bar q_2),\ldots,(\bar p_r\bar q_r)\}, \]
defined for matrices from \(M_\xi\), for each component \((\bar p_i\bar q_i)\) of which the following conditions are satisfied:

1. The numbers \(\tau(\bar p_i)\), \(\tau'(\bar p_i)\), \(\tau^0(\bar q_i)\), \(\tau^{\prime 0}(\bar q_i)\) are defined.
2. \(\tau^0(\bar p_i)\geqslant \tau'(\bar q_i)\), \(\tau(\bar p_i)\geqslant \tau^{\prime 0}(\bar q_i)\).

Put
\[ \bar\tau^0=\{(n1)\} \]
and
\[ \bar\xi=(\bar\tau\bar\tau^0). \]
Then the operator \((\mu,\mu')\to[\mu\mu']_{\bar\xi}\) is invariant (here \(\mu\in M_\xi\), \(\mu'\in M_{\xi'}\)). This operator will be called the \(G\)-convolution.

We shall now construct one more invariant operator—the \(G\)-alternation. For this, to each \(\mu\) there will be assigned a matrix of the same order, composed in a definite way from the minors of the matrix \(\mu\).

Let \(\tilde S\) be the set of \((s-1)\)-dimensional integer vectors \((1\leqslant s\leqslant n)\) whose components are not less than 1, not greater than \(n\), and, moreover, are distinct and arranged in increasing order. Let, further, \(\mu_{\tilde i\tilde j}\) \((\tilde i,\tilde j\in\tilde S)\) be the minor of order \(s-1\) standing at the intersection of the rows and columns of \(\mu\) whose numbers are the components of \(\tilde i\) and \(\tilde j\), respectively. Introduce on \(\tilde S\) the lexicographic order relation and associate with each \(\mu\) the matrix \(\tilde\mu\), composed of the minors \(\mu_{\tilde i\tilde j}\). (Such a matrix is called the \((s-1)\)-st associated matrix for \(\mu\) (see (2).)

Let now \(\mu\in M_\xi\) and \(\xi=(\tau\tau^0)\). Recall that, by the definition of \(M_\xi\), the broken line \(\tau^0(\mu)\) bounds the zero step-shaped part of \(\mu\) (the line \(\tau^0(\mu)\) itself does not belong to this step-shaped part). It is easy to verify that this zero part of \(\mu\) uniquely determines a certain zero step-shaped part of \(\tilde\mu\). Let this step-shaped part be bounded by the broken line \(\tilde\tau^0(\tilde\mu)\),
\[ \tilde\tau^0=\{(\tilde p_1^0\tilde q_1^0),\ldots,(\tilde p_l^0\tilde q_l^0)\}, \quad \tilde p_m^0\tilde q_m^0\in\tilde S\quad (1\leqslant l\leqslant m). \]
(Here, as above, we assume that \(\tilde\tau^0(\tilde\mu)\) itself does not belong to that zero step-shaped part of \(\tilde\mu\) which it bounds.) Finally, let
\[ \tilde p_m^0=(p_{1m}^0,p_{2m}^0,\ldots,p_{s-1\,m}^0),\qquad \tilde q_m^0=(q_{1m}^0,q_{2m}^0,\ldots,q_{s-1\,m}^0). \]

Define the matrix of order \(s\), \(\mu^{(s)}\), as follows: put
\[ (\mu^{(s)})_{ij} = \mu_{(i,p_{1m}^0,p_{2m}^0,\ldots,p_{s-1\,m}^0)\,(q_{1m}^0,q_{2m}^0,\ldots,q_{s-1\,m}^0,j)} \]
if the numbers \((ij)\) satisfy the inequalities
\[ p_{s-1\,m}^0<i<p_{1m}^0,\quad i\geqslant p_k,\quad q_{s-1\,m}^0<j<q_{1\,m+1}^0,\quad j\leqslant q_k \]
for at least one \(m\) and \(k\), and \((\mu^{(s)})_{ij}=0\) otherwise. It is easy to show that the latter inequalities determine such a \(\tilde\xi\) that \(\mu^{(s)}\in M_{\tilde\xi}\).

Theorem 2. The operator \(\mu\to\mu^{(s)}\) is invariant.

The proof follows from the Binet–Cauchy formula (see (2)), expressing a minor of a product of matrices in terms of the minors of the factors: from \(\alpha=\beta\gamma\) it follows that
\[ \alpha_{\tilde i\tilde j}=\sum_{\tilde s\in\tilde S}\alpha_{\tilde i\tilde s}\beta_{\tilde s\tilde j}. \]

Finally, the last invariant operator—addition—we introduce by means of the mapping \((\mu\mu')\to\mu+\mu'\), where \(\mu,\mu'\in M_\xi\). The invariance of this operator is obvious.

A polynomial on \(M_\xi\), \(f(\mu)\), will be called an invariant of a \(\xi\)-tensor if
\[ f(\mu)=f(g\mu g^{-1})\quad \forall g\in G. \]

Lemma 3. Let \(\gamma:M_\xi\to\overline M\) be an invariant operator and let \(f(\bar\mu)\) be an invariant of a \(\bar\xi\)-tensor; then \(f(\gamma(\mu))\), considered as a polynomial on \(M_\xi\), is an invariant of a \(\xi\)-tensor.

The proof is obvious.

Let \(\xi=(\tau\tau^0)\) and
\[ \tau^0=\{(p_1^0q_1^0),(p_2^0q_2^0),\ldots,(p_{r^0}^0q_{r^0}^0)\}. \]
It is easily verified that the polynomials
\[ \mu_{p_k^0}^{\,q_{k-1}^0}\qquad (k=1,2,\ldots,r^0+1) \]
are invariants of a \(\xi\)-tensor (here \(q_0^0=1\) and \(p_{r^0+1}^0=n\)). Such invariants we shall call triangular. The results of the present note can be summarized in the following theorem.

Theorem 3. Let \(\gamma: M_{\xi}\to \overline{M}_{\xi}\) be an operator that is the product of the invariant operators introduced above. Suppose, further, that \(f(\overline{\mu})\) is a triangular invariant of the \(\xi\)-tensor.

Then \(f(\gamma(\mu))\) is an invariant of the \(\xi\)-tensor.

A direct verification shows that the invariants obtained are sufficient to separate any two orbits for \(n\leq 6\) and, for arbitrary \(n\), to find a number of invariant surfaces and orbits distinct from those described in \((1)\). It is not known, however, whether with the help of the last theorem one can separate any two orbits for arbitrary \(n\).

Moscow Institute of Economics and Statistics

Received
3 III 1969

REFERENCES

\({}^{1}\) A. A. Kirillov, Uspekhi Mat. Nauk, 17, no. 4, 57 (1962). \({}^{2}\) F. R. Gantmakher, Matrix Theory, 3rd ed., Nauka, 1967.