UDC 513.88:517.948.35

MATHEMATICS

E. A. LARIONOV

ON NILPOTENT \(J\)-SELF-ADJOINT OPERATORS

(Presented by Academician I. N. Vekua on 11 III 1968)

Let \(\mathfrak H\) be a \(J\)-space. This is the name for a Hilbert space in which, along with the usual scalar product \([x,y]\), an indefinite scalar product \((x,y)=[Jx,y]\) is given, where \(J=P_+-P_-\), \(P_+\) and \(P_-\) are orthoprojectors, \(P_+ + P_- = I\). By \(\mathfrak M_+\) \((\mathfrak M_-)\) we shall denote the totality of all maximal subspaces in \(\mathfrak P_+ = \{x\in\mathfrak H : (x,x)\geq 0\}\) \((\mathfrak P_-=\{x\in\mathfrak H : (x,x)\leq 0\})\). Put \(\mathfrak P_0=\{x\in\mathfrak H : (x,x)=0\}\).

The notions of a \(J\)-self-adjoint and a \(J\)-unitary operator in \(\mathfrak H\) are introduced, with respect to the form \((x,y)\), in the usual way. Linear operators whose spectrum is concentrated only at zero occupy a special place in the spectral theory of operators \((^{1,2})\). Such operators are called nilpotent.

In contrast to the usual scalar product \([x,y]\), with respect to which there exists no nonzero nilpotent self-adjoint, or even normal, operator, there do exist nonzero \(J\)-self-adjoint nilpotent operators.

Let us give the simplest example. Introduce in the complex Euclidean space \(E_n\) of dimension \(n=2k\) the quadratic form

\[ (x,x)=\xi_1\overline{\xi}_{2k}+\xi_2\overline{\xi}_{2k-1}+\cdots+\xi_{2k}\overline{\xi}_1, \tag{1} \]

It is easy to verify that the form (1) contains \(k\) negative squares, so that the space \(E_n\), furnished with the form (1) and the scalar product

\[ [x,y]=\sum_{i=1}^{2k}\xi_i\overline{\eta}_i, \tag{2} \]

where \(x=(\xi_1,\xi_2,\ldots,\xi_{2k})\), \(y=(\eta_1,\eta_2,\ldots,\eta_{2k})\in E_n\), becomes a finite-dimensional space with an indefinite metric. We shall denote such a space by \(\Pi_k^{2k}\).

Now define an operator \(A\) in the space \(\Pi_k^{2k}\) by the following square matrix of order \(2k+1\)

\[ A= \left( \begin{array}{cccccc} 0 & a_{1,k+1} & a_{1,k+2} & \ldots & a_{1,2k} \\ 0 & 0 & a_{2,k+2} & \ldots & a_{2,2k} \\ \vdots & \vdots & \ddots & \ddots & \vdots \\ 0 & 0 & 0 & \ddots & a_{k,2k} \\ 0 & 0 & 0 & \ldots & 0 \end{array} \right) \tag{3} \]

The operator \(A\) is nilpotent.

If now the matrix is self-adjoint with respect to the second main diagonal, then it is verified directly that, with respect to the form (1),

\[ (Ax,y)=(x,Ay). \tag{4} \]

Let now \(A\) be a Volterra operator in \(\mathfrak H\).

Consider in the space \(\mathfrak H_0=\mathfrak H\oplus\mathfrak H\) the operators \(H\) and \(J\), whose matrix representation has the form

\[ H=\begin{pmatrix} 0 & (A^*A)^{1/2}\\ -(A^*A)^{1/2} & (A+A^*) \end{pmatrix}, \qquad J=\begin{pmatrix} I & 0\\ 0 & -I \end{pmatrix}. \]

It is easy to prove that \(\sigma(H)=0\). On the other hand,

\[ (JH)^*=JH. \]

Denote by \(\Omega\) the totality of all nilpotent \(J\)-self-adjoint operators in the space \(\mathfrak H\).

Theorem 1. If an operator \(A\ne 0\) from \(\Omega\) has an invariant subspace \(\mathcal L\) from \(\mathfrak M_+\) or \(\mathfrak M_-\), then the subspace \(\mathcal L\) necessarily contains a \(J\)-neutral subspace \(\mathcal L_0\).

Proof. Suppose that \(\mathcal L\) is a \(J\)-definite invariant subspace of the operator \(A\). For definiteness let \(\mathcal L\in\mathfrak M_+\). Then the \(J\)-orthogonal complement \(\mathcal F\) to the \(J\)-positive subspace \(\mathcal L\) in \(\mathfrak H\) will be a \(J\)-negative maximal invariant subspace of the operator \(A\).

It is known \((^3)\) that the direct \(J\)-orthogonal sum \(\hat{\mathfrak H}=\mathcal L\oplus\mathcal F\) is dense in the norm \(\|x\|=[x,x]^{1/2}\) in \(\mathfrak H\). On the other hand, with respect to the definite scalar product

\[ [x,y]_1=(x_+,y_+)-(x_-,y_-),\qquad x_+,y_+\in\mathcal L,\quad x_-,y_-\in\mathcal F, \]

the restriction \(\hat A\) of the operator \(A\) to the subspace \(\hat{\mathfrak H}_1\) becomes a self-adjoint operator. It is easy to see that the operator \(\hat A\) is equal to zero only together with the operator \(A\). From the self-adjointness of \(\hat A\) and \(A\in\Omega\) it follows that \(\hat A=0\), which contradicts the fact that \(A\ne 0\). The theorem is proved.

Corollary 1. If an operator \(A\in\Omega\) has an invariant subspace \(\mathcal L\) from \(\mathfrak M_+\) or \(\mathfrak M_-\), then it has respectively the form

\[ \mathcal L=\mathcal L_0\oplus\mathcal L_+ \quad\text{or}\quad \mathcal L=\mathcal L_0\oplus\mathcal L_-, \]

where \(\mathcal L_0\ne\{0\}\), and \(\mathcal L_+\) and \(\mathcal L_-\) are \(J\)-definite subspaces obtained by decomposing \(\mathcal L\) with the aid of some skew-related \((^4)\) subspace \(\mathcal F_0\) from \(\mathfrak H\) with \(\mathcal L_0\).

Corollary 2. Invariant subspaces from \(\mathfrak M_+\) and \(\mathfrak M_-\) of a Volterra \(J\)-self-adjoint operator \(A\) have the form indicated in Corollary 1.

Let us note that the subspace \(\mathcal L_0\) in Corollary 1 is invariant with respect to the operator \(A\), since it is the intersection of the subspace \(\mathcal L\) and its \(J\)-orthogonal complement, and both these subspaces are invariant with respect to the operator \(A\).

Let us see how the operator \(A\) acts on the invariant subspace \(\mathcal L=\mathcal L_0\oplus\mathcal L_+\) or \(\mathcal L=\mathcal L_0\oplus\mathcal L_-\) for \(\mathcal L_\pm\ne\{0\}\), when \(\mathcal L_0\) is a maximal \(J\)-neutral invariant subspace of the operator \(A\) in \(\mathcal L\). Let \(\mathcal L_0^\perp\) be the \(J\)-orthogonal complement to \(\mathcal L_0\) in \(\mathfrak H\). Obviously, \(\mathcal L_0\subset\mathcal L_0^\perp\), since \(\mathcal L_0\subset\mathfrak P_0\). Let \(\mathcal F_0=J\mathcal L_0\). Put \(\mathcal E_0=\mathcal L_0+\mathcal F_0\), and let \(\mathcal E_0^\perp\) be the \(J\)-orthogonal complement to \(\mathcal E_0\) in \(\mathfrak H\). It is easy to show that \(\mathcal E_0^\perp\) is a \(J\)-space, and therefore the factor space \(\mathfrak H_1=\mathcal L_0^\perp/\mathcal L_0\) is also a \(J\)-space. Let \(A_1\) be the restriction of the operator \(A\) to \(\mathfrak H_1\). Obviously, \(A_1\) is a nilpotent \(J\)-self-adjoint operator. By assumption (and if \(A\) is Volterra, then by the theorem from \((^5)\)) there exists a \(J\)-nonnegative maximal invariant subspace \(\mathcal L_1\) of the operator \(A_1\) in \(\mathfrak H_1\). Since \(\mathcal L_0\) is a maximal \(J\)-neutral invariant ...

subspace of the operator \(A\) in \(\mathcal L\), then \(\mathcal L_1\) is \(J\)-positive, and therefore, as in Theorem 1, there exists a definite scalar product with respect to which the operator \(A_1\) becomes self-adjoint. Thus, \(A_1=0\), and therefore the operator \(A\) maps \(\mathcal L_0^\perp\) into \(\mathcal L_0\). Let \(\mathcal L_2\) be the \(J\)-orthogonal complement in \(\mathfrak H_1\) to the subspace \(\mathcal L_1\). Putting now \(\mathcal L_+=\mathcal L_1\) and \(\mathcal L_-=\mathcal L_2\), we obtain

\[ A\mathcal L=A(\mathcal L_0\oplus \mathcal L_+)=A\mathcal L_0\oplus A\mathcal L_+\subseteq \mathcal L_0. \tag{5} \]

and an analogous relation is obtained for \(\mathcal L=\mathcal L_0\oplus\mathcal L_-\).

Consider the class \(\Omega\) in the space \(\Pi_k\).

Theorem 2. Every operator \(A\) from \(\Omega\) in the Pontryagin space \(\Pi_k\) is finite-dimensional, and its dimension does not exceed \(2k\), where \(k\) is the finite rank of indefiniteness of the space \(\Pi_k\).

Proof. Let \(\mathcal L_0\) be a maximal \(J\)-neutral invariant subspace of the operator \(A\). If \(A\ne0\), then, by Theorem 1, \(\mathcal L_0\ne\{0\}\). On the other hand (4), \(\dim\mathcal L\le k\). Let \(\mathcal F_0=J\mathcal L_0,\ \mathcal E_0=\mathcal F_0\dotplus\mathcal L_0\). Then (4)

\[ \Pi_k=\mathcal F_0\dotplus\mathcal L_0\oplus\mathcal E_0^\perp, \tag{6} \]

the factor space \(\mathfrak H_1=\mathcal L_0^\perp/\mathcal L_0\) is a \(J\)-space and, as was shown, \(A\mathcal L_0^\perp\subset\mathcal L_0\).

Let \(x\in\Pi_k\). By virtue of (5), \(x\) has the decomposition

\[ x=y+z+t, \tag{7} \]

where \(y\in\mathcal F_0,\ z\in\mathcal L_0,\ t\in\mathcal E_0^\perp\).

\[ Ax=Ay+Az+At. \tag{8} \]

Since \(\mathcal L_0^\perp=\mathcal L_0\oplus\mathcal E_0^\perp\) and \(A\mathcal L_0^\perp\subset\mathcal L_0\), we have \(At\in\mathcal L_0\) and \(Az\in\mathcal L_0\). Put \(\mathcal Q_0=A\mathcal F_0\); and let \(\mathcal N_0\) be the linear span of the subspaces \(\mathcal L_0\) and \(\mathcal Q_0\). Then from (6) and (7), taking into account the inclusions written above, we obtain

\[ A\Pi_k\subset\mathcal N_0. \tag{9} \]

Moreover \(\dim\mathcal F_0=\dim\mathcal L_0\), and therefore \(\dim\mathcal F_0\le k\). Hence, \(\dim\mathcal N_0\le2k\), and Theorem 2 is proved.

Corollary 1. A \(J\)-self-adjoint operator \(A\) from \(\Omega\) in a Pontryagin space is nilpotent and \(A^{2k_1+1}=0\), where \(k_1\) is the dimension of the maximal \(J\)-neutral invariant subspace of the operator \(A\), so that \(k_1\le k\).

M. A. Naimark in the works \((^6,^7)\) obtained a description of symmetric commutative algebras \(R\) in \(\Pi_k\). From Corollary 1 and \((^6,^7)\) we obtain

Corollary 2. If the algebra \(R\) is not semisimple, then the class \(\Omega\cap R\) belongs to the radical of the algebra \(R\).

We now indicate a class of nilpotent operators \(\mathfrak D(0)\) in \(\mathfrak H\) for which there exists no \(J\)-metric with respect to which the operators of this class are \(J\)-self-adjoint.

Theorem 3. Let \(A\ne0\) be a nilpotent operator in \(\mathfrak H\). If the imaginary part of the operator \(A\) is sign-definite, then there exists no \(J\)-metric with respect to which the operator \(A\) is \(J\)-self-adjoint.

Proof. Without loss of generality one may assume that

\[ \operatorname{Im}A=(A-A^*)/2i\ge0, \]

since one may pass from the operator \(A\) to the operator \(-A\). Suppose that there exists a \(J\)-metric such that the operator \(A\) is \(J\)-self-adjoint. The Cayley transform of the operator \(A\) is a \(J\)-unitary operator \(U\) in \(\mathfrak H\). On the other hand, from the condition \(\operatorname{Im}A\ge0\) we obtain that \((^8)\)

\[ \|U\|\le1, \tag{10} \]

and then for the operator \(U\) there exists \((^9)\) an invariant maximal dual pair of subspaces \(\{\mathcal L,\mathcal F\}\) and

\[ \mathfrak H=\mathcal L\oplus\mathcal F. \tag{11} \]

The pair \(\{\mathcal L,\mathcal F\}\) will be invariant also with respect to the operator \(A\), and therefore, with respect to the definite scalar product \([x,y]_1\), introduced as in Theorem 1, the operator \(A\) becomes self-adjoint. Thus, \(A=0\), which contradicts the condition of the theorem.

Corollary. If a nilpotent operator \(A\) is dissipative, then there exists no indefinite scalar product with respect to which the operator is \(J\)-self-adjoint.

Let us give an example. The integral operator \(N\) in \(L_2(0,1)\), defined by the equality

\[ (Nf)(t)=2i\int_0^t f(s)\,ds, \]

is dissipative, and therefore there is no operator \(J=P_+-P_-\), \(P_+\oplus P_-=I\), such that \((JN)^*=JN\).

In conclusion I express my sincere gratitude to M. A. Naimark and I. S. Iokhvidov for their attention to this work.

Central Economics and Mathematics Institute
of the Academy of Sciences of the USSR

Received
4 III 1968

CITED LITERATURE

\(^{1}\) N. Dunford, Pacific J. Math., 4, 321 (1954).
\(^{2}\) N. Dunford, Bull. Am. Math. Soc., 64, 217 (1958).
\(^{3}\) Yu. P. Ginzburg, I. S. Iokhvidov, UMN, 17, 4 (106), 3 (1962).
\(^{4}\) I. S. Iokhvidov, M. G. Krein, Tr. Mosk. matem. obshch., 5, 367, 432 (1956).
\(^{5}\) H. Sanger, Math. Ann., 152, 434 (1963).
\(^{6}\) M. A. Naimark, Rev. Roum. Math. Pure et Appl., 9, 6, 499 (1964).
\(^{7}\) M. A. Naimark, Math. Ann., 162, 147 (1965).
\(^{8}\) I. Ts. Gokhberg, M. G. Krein, Introduction to the theory of linear non-self-adjoint operators in Hilbert space, Moscow, 1965.
\(^{9}\) R. S. Phillips, Proc. Int. Sympos. on Linear Spaces, Jerusalem, 1960, 1961, p. 366.