MATHEMATICS

G. K. LANGER

ON INVARIANT SUBSPACES OF LINEAR OPERATORS ACTING IN A SPACE WITH AN INDEFINITE METRIC

(Presented by Academician L. S. Pontryagin on 5 XI 1965)

1. Let \(\mathfrak H\) be a \(J\)-space \((^1)\), i.e., a Hilbert space in which, in addition to the usual scalar product \((x,y)\), an indefinite scalar product
\[ [x,y]=(Jx,y),\qquad J=P_+-P_-, \]
is introduced, where \(P_+\) and \(P_-\) are two mutually complementary orthogonal projectors. Put
\[ \mathfrak P_+=\{x:[x,x]\geq 0\}; \]
\(\mathfrak P_-\) and \(\mathfrak P_0\) are defined analogously. By \(\mathfrak M_1(\mathfrak M_2)\) we denote the set of all maximal subspaces from \(\mathfrak P_+(\mathfrak P_-)\). Two subspaces \(\mathfrak L_1\) and \(\mathfrak L_2\) are called \(J\)-orthogonal \((\mathfrak L_1\perp \mathfrak L_2)\) if \([\mathfrak L_1,\mathfrak L_2]=\{0\}\). For any set \(\mathfrak G\subset\mathfrak H\) put
\[ \mathfrak G^\perp=\{x:[x,\mathfrak G]=\{0\}\}. \]

Denote by \(\mathfrak R\) the ring of all linear bounded operators \(A\) acting in \(\mathfrak H\), and by \(\mathfrak S_\infty\) its ideal consisting of all completely continuous \(A\in\mathfrak R\).

For arbitrary \(A\in\mathfrak R\), the equality
\[ [Ax,y]=[x,A^+y]\qquad (x,y\in\mathfrak H) \]
defines the \(J\)-adjoint \(A^+\); if \(A=A^+\) \((U^+U=UU^+=I)\), then the operator \(A\) \((U)\) is called \(J\)-selfadjoint (\(J\)-unitary).

We shall say that a commutative (with respect to multiplication) family \(\mathfrak A\subset\mathfrak R\) has property \((P)\) if, for each pair of subspaces \(\mathfrak L_1\) and \(\mathfrak L_2\) satisfying the conditions
\[ \text{1) }\mathfrak L_1\subset\mathfrak P_+,\ \mathfrak L_2\subset\mathfrak P_-;\qquad \text{2) }\mathfrak L_1\perp\mathfrak L_2;\qquad \text{3) }A\mathfrak L_j\subset\mathfrak L_j\ (j=1,2;\ A\in\mathfrak A), \]
there exists such a pair of subspaces \(\mathfrak L_1^{\max}\) and \(\mathfrak L_2^{\max}\) that
\[ \text{1′) }\mathfrak L_j\subset\mathfrak L_j^{\max}\in\mathfrak M_j;\qquad \text{2′) }\mathfrak L_1^{\max}\perp\mathfrak L_2^{\max};\qquad \text{3′) }A\mathfrak L_j^{\max}\subset\mathfrak L_j^{\max}\ (j=1,2;\ A\in\mathfrak A). \]
If the family \(\mathfrak A\) consists of one operator and has property \((P)\), then we shall say that this operator has property \((P)\).

R. S. Phillips \((^2)\) conjectured that an arbitrary commutative algebra \(\mathfrak A\subset\mathfrak R\), closed with respect to the operation of \(J\)-adjunction \((\mathfrak A=\mathfrak A^+)\), has property \((P)\), and proved it for the case when \(\mathfrak A=\mathfrak A^+-\mathfrak A^*\). It is unknown whether an arbitrary \(J\)-selfadjoint (\(J\)-s.a.) or \(J\)-unitary operator has property \((P)\). In this connection, the following generalization of Theorem 3 from \((^3)\) is of interest.

Theorem 1. Every \(J\)-unitary operator \(U\) for which
\[ P_+UP_-\in\mathfrak S_\infty \]
has property \((P)\).

We give some explanations for the proof of the theorem. Let \(\mathfrak L_1\) and \(\mathfrak L_2\) be subspaces with properties 1)—3). Without loss of generality one may assume that
\[ U\mathfrak L_j=\mathfrak L_j\qquad (j=1,2). \]
Represent \(\mathfrak L_j\) in the form
\[ \mathfrak L_j=\{P_j(x)+K_jP_jx:x\in\mathfrak H_j\}, \]
where \(\mathfrak H_1=P_+\mathfrak H\), \(\mathfrak H_2=P_-\mathfrak H\); \(P_j\) is the orthoprojector in \(\mathfrak H_j\); \(K_j\mid \mathfrak H_j\to\mathfrak H_k\), \(\|K_j\|\leq 1\), \(j,k\ne 1,2\), \(j\ne k\)*. Then the following holds.

Lemma 1. A subspace \(\mathfrak L\in\mathfrak M_1\) satisfies the conditions \(\mathfrak L\supset\mathfrak L_1\), \(\mathfrak L\perp\mathfrak L_2\) if and only if for its angular operator \(K_{\mathfrak L}\) the equality
\[ K_{\mathfrak L}=K_1P_1+(K_2P_2)^*(I-P_1)+(I-P_2)K_{\mathfrak L}(I-P_1) \]
holds.

* For \(\mathfrak L_j\in\mathfrak M_j\), the operator \(K_j\) coincides with the angular operator \((^1)\) of the subspace \(\mathfrak L_j\) \((j=1,2)\).

Further, Theorem 1 is proved by the same method as in \((^3,{}^4)\); one need only consider, instead of the set \(\mathfrak K_+\) from \((^3)\), the set of all bounded linear operators \(X\) acting from \((I-P_1)\mathfrak H_1\) into \((I-P_2)\mathfrak H_2\), such that

\[ \|K_1P_1+(K_2P_2)^*(I-P_1)+(I-P_2)X(I-P_1)\|\leq 1 \]

and apply the fixed-point theorem 1 from \((^4)\).

Similarly one proves

Theorem 2. Let the operator \(A\in\mathfrak R\) satisfy the conditions \(A\mathfrak P_+\subset\mathfrak P_+\), \(P_+AP_-\in\mathfrak S_\infty\), and let \(\mathfrak L\) be a subspace with the properties \(\mathfrak L\subset\mathfrak P_+\), \(A\mathfrak L=\mathfrak L\). Then there exists \(\mathfrak L^{\max}\in\mathfrak M_1\) such that \(\mathfrak L^{\max}\supset\mathfrak L\) and \(A\mathfrak L^{\max}\subset\mathfrak L^{\max}\).

1. We shall call an operator \(B\in\mathfrak R\) \(J\)-definite if either \([Bx,x]\geq 0\) \((x\in\mathfrak H)\), or \([Bx,x]\leq 0\) \([x\in\mathfrak H]\). We shall call a \(J\)-s.a. operator \(A\) \(J\)-definizable if there exists a nontrivial polynomial \(p_A(\lambda)\) such that the operator \(p_A(A)\) is \(J\)-definite. It is known, for example, that in the space \(\Pi_\kappa\) \((^1)\) every \(J\)-s.a. operator is \(J\)-definizable \((^5)\).

Theorem 3. Every finite commutative family \(\mathfrak A\) of \(J\)-definizable operators has at least one common invariant subspace \(\mathfrak L^{\max}\in\mathfrak M_1\) \((A\mathfrak L^{\max}\subset\mathfrak L^{\max};\ A\in\mathfrak A)\).

As M. G. Krein kindly pointed out to me, from this theorem one easily obtains

Theorem 4. Every commutative family \(\mathfrak A\) of \(J\)-definizable operators \(A\) with the property \(P_+AP_-\in\mathfrak S_\infty\) \((A\in\mathfrak A)\) has at least one common invariant subspace \(\mathfrak L^{\max}\in\mathfrak M_1\).

Theorem 4 contains the theorem of M. A. Naimark from \((^6,{}^7)\).

An operator \(F\in\mathfrak R\) with the property \(F^2=F=F^+\) is called a \(J\)-projector; two \(J\)-projectors \(F,G\) are called \(J\)-orthogonal if \(FG=0\). Theorem 3 follows from the following more general proposition.

Theorem 5. Let \(\mathfrak A\) be a commutative family of \(J\)-s.a. operators, \(\mathfrak F\) a family of mutually \(J\)-orthogonal \(J\)-projectors for which \(AF=FA\) \((A\in\mathfrak A,\ F\in\mathfrak F)\), and suppose that for every pair of operators \(A,F\) \((A\in\mathfrak A,\ F\in\mathfrak F)\) there exists a natural number \(n=n(A,F)\) such that the operator \(A^nF\) is \(J\)-definite. Then there exists \(\mathfrak L^{\max}\in\mathfrak M_1\), invariant with respect to all operators \(F\) and \(AF\) \((F\in\mathfrak F,\ A\in\mathfrak A)\).

In the proof of Theorems 3 and 5 it is essential that a \(J\)-definizable operator possesses “its own spectral function with critical points,” analogous to that considered in \((^5)\) for \(J\)-s.a. operators in \(\Pi_\kappa\).* The following two lemmas are also used; they are of independent interest.

Lemma 2. Every family \(\mathfrak F\) of mutually \(J\)-orthogonal \(J\)-projectors has property \((P)\).

In the case when the family \(\mathfrak F\) consists only of the identity operator, the assertion of Lemma 2 follows from the result of Phillips formulated above, which is also used in the proof of Lemma 2.

Lemma 3. Let \(\mathfrak A\) be a commutative family of \(J\)-s.a. operators. Then there exists a subspace \(\mathfrak N_0\subset\mathfrak H\) satisfying the conditions: \(\mathfrak N_0\subset\mathfrak P_0\), \(A\mathfrak N_0\subset\mathfrak N_0\), and \(A\mathfrak N_0^\perp\cap(A\mathfrak N_0^\perp)^\perp\subset\mathfrak N_0\) \((A\in\mathfrak A)\).

Theorem 6. Every commutative family \(\mathfrak A\) of \(J\)-definite operators has property \((P)\).

* We take this opportunity to point out, on behalf of the authors of note \((^5)\), that an error was made in writing formula (2) of that note. The correct form of this formula, in the notation of the note, is as follows:

\[ \mathscr P_{A^2}(A)E(\Delta)x=SE(\Delta)x+\int_\Delta \mathscr P_{A^2}(\lambda)\,dE_\lambda x \qquad (x\in\Pi_\kappa), \]

where \(S\) is a s.a. operator with the properties: 1) \(S^2=0\); 2) \((Sx,x)\geq 0\) \((x\in\Pi_\kappa)\), and 3) \(SE(\Delta')=0\) for every segment \(\Delta'\) containing no critical point.

The assertions of Theorems 3 and 6 are apparently new even for the case when the family \(\mathfrak A\) consists of only one operator.

1. Let us now consider the pencil \(L(\lambda)=\lambda^2 I+\lambda B+C\), where \(B\) and \(C\) are two bounded self-adjoint operators in the Hilbert space \(\mathfrak K\). The pencil \(L(\lambda)\) is called relatively strongly damped (r.s.d.) \((^8)\) if

\[ (By,y)^2>4(Cy,y)\|y\|^2 \qquad (y\in\mathfrak K;\ y\ne0) \tag{*} \]

and strongly damped (s.d.) if, in addition, \(B,C\ge0\). Condition \((*)\) means that each quadratic trinomial

\[ \varphi_y(\lambda)=(L(\lambda)y,y) \qquad (y\in\mathfrak K,\ y\ne0) \]

has two distinct real roots \(p_{1,2}(y)\), \(p_2(y)<p_1(y)\). Put
\(\alpha_1=\inf p_1(y)\), \(\alpha_2=\sup p_2(y)\) \((y\ne0)\). It turns out (cf. \((^9)\)) that
\(-\infty<\alpha_2\le\alpha_1<\infty\). We shall say that a point \(\lambda\in\sigma(L)\) \((^9)\) of the r.s.d. pencil \(L(\lambda)\) belongs to \(\sigma_1(L)\) \((\sigma_2(L))\), if for every sequence \((y_n)\) for which \(\|y_n\|=1\) and \(L(\lambda)y_n\to0\), one has \(p_1(y_n)\to\lambda\) \((p_2(y_n)\to\lambda)\). It can be shown that

\[ \sigma(L)\cap[\alpha_1,\infty)=\sigma_1(L)\cup\{\alpha_1\},\qquad \sigma(L)\cap(-\infty,\alpha_2]=\sigma_2(L)\cup\{\alpha_2\}. \]

As in \((^{9,10})\), with the s.d. pencil \(L(\lambda)\) we associate the \(J\)-s.s. operator

\[ H= \begin{pmatrix} 0 & C^{1/2}\\ -C^{1/2} & -B \end{pmatrix} \quad \text{in the } J\text{-space } \mathfrak H=\mathfrak K\oplus\mathfrak K,\quad J= \begin{pmatrix} I & 0\\ 0 & -I \end{pmatrix}. \]

We shall say \((^{11})\) that a \(J\)-s.s. operator \(A\) in the \(J\)-space \(\mathfrak H\) satisfies the Pesonen condition if from the equalities \([x,x]=0\) and \([Ax,x]=0\) it follows that \(x=0\).

Lemma 4. The pencil \(L(\lambda)\) is s.d. if and only if the operator \(H\) satisfies the Pesonen condition.

Theorem 7. Let the pencil \(L(\lambda)\) be r.s.d. Then there exist two roots \(Z_1\) and \(Z_2=-B-Z_1^*\) of the equation \(Z^2+BZ+C=0\) such that:

1) \(\sigma(Z_j)=\sigma_j(L)\cup\{\alpha_j\}\), and the eigenvectors \((^9)\) of the pencil \(L(\lambda)\) and of the root \(Z_j\), corresponding to eigenvalues from \(\sigma_j(L)\setminus\{\alpha_k\}\), coincide \((j,k=1,2;\ j\ne k)\);

2) the roots \(Z_1\) and \(Z_2\) are symmetrized by the positive operator

\[ S=Z_1+Z_1^*+B=Z_1-Z_2; \]

3) the roots \(Z_1,Z_2\) form a complete pair \((^{9,10})\);

4) for each segment \(\Delta\) of the real axis lying to the right of \(\alpha_2\) (to the left of \(\alpha_1\)), there exist two subspaces \(\mathfrak K_\Delta,\mathfrak K'_\Delta\), invariant with respect to the operator \(Z_1\) \((Z_2)\), such that
\(\mathfrak K=\mathfrak K_\Delta+\mathfrak K'_\Delta\) and
\(\sigma(Z_1|\mathfrak K_\Delta)\subset\Delta\),
\(\sigma(Z_1|\mathfrak K'_\Delta)\subset\Delta'\)
\((\sigma(Z_2|\mathfrak K_\Delta)\subset\Delta,\ \sigma(Z_2|\mathfrak K'_\Delta)\subset\Delta')\), where
\(\Delta'=(-\infty,\infty)\setminus\Delta\). The operator \(Z_1\) \((Z_2)\) in the subspace \(\mathfrak K_\Delta\) is similar to a self-adjoint operator. If the pencil \(L(\lambda)\) is strongly damped, then, in addition, the inequalities

\[ Z_1^*Z_1\le C,\qquad Z_2^*Z_2>C \]

hold.

To the proof of Theorem 7 we add the following explanations. By the change of parameter \(\lambda\to\lambda-b\) (\(b\) a sufficiently large real number), an r.s.d. pencil can be reduced to an s.d. pencil
\(L'(\lambda)=\lambda^2 I+\lambda B'+C'\), where
\(B'=2bI+B\), \(C'=b^2I+bB+C\) \((^8)\). Therefore it is enough to consider the case of an s.d. pencil. For such a pencil the operator \(H\) satisfies the Pesonen condition (Lemma 4), by virtue of which, by a lemma of R. Cohn \((^{11})\), there will exist a real \(\mu\) such that the operator \(H-\mu I\) is \(J\)-definite \((\alpha_2\le\mu\le\alpha_1)\). Then, by Theorem 3, the operator \(H\) will have a maximal

a nonnegative invariant subspace. After this, the root \(Z_1\) is obtained in the same way as in \((^9,{}^{10})\). Theorem 4 is proved with the aid of the above-mentioned “proper spectral function” of the \(J\)-definitizable operator \(H\).

Theorem 7, in comparison with the corresponding theorem from \((^9,{}^{10})\), is more general in the sense that it does not assume complete continuity of the operator \(C\).

Finally, we note that Theorem 7 can also be extended to the case of an unbounded operator \(B\).

I express my sincere gratitude to Prof. M. G. Krein for discussion and for very valuable suggestions.

Dresden Technical University
Dresden, GDR

Received
3 XI 1965

REFERENCES

\({}^{1}\) Yu. P. Ginzburg, I. S. Iokhvidov, UMN, 17, 4 (106), 3 (1962).
\({}^{2}\) R. S. Phillips, Proc. Int. Symposium Linear Spaces, Jerusalem, 1961.
\({}^{3}\) M. G. Krein, DAN, 154, No. 5, 1023 (1964).
\({}^{4}\) I. S. Iokhvidov, DAN, 159, No. 3, 501 (1964).
\({}^{5}\) M. G. Krein, G. K. Langer, DAN, 152, No. 1, 39 (1963).
\({}^{6}\) M. A. Naimark, DAN, 149, No. 6, 1261 (1963).
\({}^{7}\) M. A. Naimark, Acta Sci. Math. Szeged, 24, 177 (1963).
\({}^{8}\) R. J. Duffin, Quarterly Appl. Math., 18, 215 (1960).
\({}^{9}\) M. G. Krein, G. K. Langer, DAN, 154, No. 6, 1258 (1964).
\({}^{10}\) M. G. Krein, G. K. Langer, Proceedings of the International Symposium on the Application of the Theory of Functions of a Complex Variable in Continuum Mechanics, Moscow, 1965.
\({}^{11}\) R. Kühne, Math. Ann., 154, 56 (1964).