UDC 517.917

MATHEMATICS

A. I. PEROV

ON THE QUESTION OF THE STRUCTURE OF THE LIMIT SET

(Presented by Academician I. G. Petrovskii, December 6, 1966)

1. Types of orbits

Let \(E_x\) and \(E_y\) be real Euclidean oriented spaces, with \(E_x\) two-dimensional and \(E_y\) three-dimensional. Consider the many-dimensional differential equation

\[ y'(x)=f(y(x)), \tag{1} \]

in which the unknown is the vector function \(y(x)\) of the vector variable \(x\) \((x\in E_x,\ y(x)\in E_y)\) and \(y'(x)\in L(E_x;E_y)\), where \(L(E_x;E_y)\) is the space of all linear operators acting from \(E_x\) into \(E_y\). The operator function \(f(y)\) is continuously differentiable in some domain \(G\subset E_y\) and takes values in the space \(L(E_x;E_y)\).

The rank of a point \(\eta\in G\) shall mean the rank of the linear operator \(f(\eta)\). A point is called singular if its rank is zero. Points of rank zero or one are called singular. A point is called regular if its rank is two. A set \(A\subset G\) is regular if each of its points is regular.

We assume that for any initial condition

\[ y(0)=\eta \tag{2} \]

the equation under consideration has a unique solution, which we denote by \(\mathfrak{D}(x,\eta)\). This holds if and only if the condition
\[ \Lambda_{hk}\{f'(y)(f(y)h)k\}=0 \quad (h,k\in E_x) \]
is satisfied. The vector function \(\mathfrak{D}(x,\eta)\) is defined and continuously differentiable on some open set of the space \(E_x\oplus E_y\).

In what follows we shall use the ordinary vector differential equation (the orthogonal equation)

\[ dy/dt=g(y), \tag{3} \]

where \(g(y)=[f(y)e_1,f(y)e_2]\). Here \(e_1,e_2\) is some basis of \(E_x\), and the symbol \([u,v]\) is used, as usual, to denote the vector product of the elements \(u,v\in E_y\).

The surface given by the equation \(y=\mathfrak{D}(x,\eta)\) is called the integral manifold passing through the point \(\eta\). If the solution is defined on all of \(E_x\), then the integral manifold representing it is called complete.

The period group \(\Omega=\Omega(\eta)\) of the solution \(\mathfrak{D}(x,\eta)\) has the form \(\Omega=N+S\), where \(N\) is the maximal subspace of constancy, and \(S\) is a discrete group and \(L(S)\cap N=0\) (\(L(S)\) is the linear span of the set \(S\)).

In the case we consider, complete integral manifolds can be only of the following six types: 1) if \(N\) is two-dimensional, then the integral manifold is zero-dimensional; 2) if \(N\) is one-dimensional, then integral manifolds may be of two kinds—a simple curve and a closed curve, according as \(S=0\) or \(S\ne 0\); 3) if \(N\) is zero-dimensional, then two-dimensional integral manifolds may be of three kinds: of torus type (briefly, a torus), if \(L(S)\) is two-dimensional (\(\eta\) is a point of the first type); of cylinder type (briefly, a cylinder), if \(L(S)\) is one-dimensional (\(\eta\) is a point of the second-

type); of plane type (a simple integral manifold), if \(L(S)\) is zero-dimensional (\(\eta\) is a point of the third type).

2. Limit sets. The dynamic limit set \(\mathfrak D(\infty,\eta)\) of the orbit \(\mathfrak D(E_x,\eta)\) is the set of all points \(\zeta\in G\) for each of which one can specify a sequence \(x_j\in E_x\) such that \(\hat x_j\to\infty\) and \(\mathfrak D(x_j,\eta)\to\zeta\). Here \(\hat x\) is the adjacency class of the quotient group \(E_x/\Omega\) containing the point \(x\). If the simple integral manifold \(\mathfrak D(E_x,\eta)\) lies in a compact part of the domain \(G\), then \(\mathfrak D(\infty,\eta)\) is nonempty, compact, connected, and invariant, where the latter is understood as follows: if \(\zeta\in\mathfrak D(\infty,\eta)\), then the solution \(\mathfrak D(x,\zeta)\) is defined for all \(x\in E_x\) and \(\mathfrak D(E_x,\zeta)\subset\mathfrak D(\infty,\eta)\).

Let \(\mathfrak D(E_x,\eta)\) be an integral manifold of cylinder type and let \(\omega^+\), \(\omega\) be a basis of the space \(E_x\), with \(\omega\) a basis vector of the group \(\Omega\). The dynamic \((+)\)-limit set \(\mathfrak D(+\infty,\eta)\) is the set of all points \(\zeta\in G\), each of which is obtained in the following way: \(x_j^1\to+\infty\), \(\mathfrak D(x_j,\eta)\to\zeta\) \((x_j=x_j^1\omega^+ + x_j^2\omega)\). The definition of the \((-)\)-limit set \(\mathfrak D(-\infty,\eta)\) is given analogously. It is clear that
\[ \mathfrak D(\infty,\eta)=\mathfrak D(-\infty,\eta)\cup \mathfrak D(+\infty,\eta). \]
If \(\mathfrak D(E_x,\eta)\) is compact and lies in \(G\), then each of the limit sets \(\mathfrak D(-\infty,\eta)\) and \(\mathfrak D(+\infty,\eta)\) is nonempty, compact, connected, and invariant.

An integral manifold of torus type has no limit set.

We are interested in the behavior of integral manifolds in a neighborhood of a torus and in the structure of limit sets for integral manifolds of cylinder type and plane type. The investigation of such questions on the plane constitutes the content of the Poincaré–Bendixson theory \(({}^{1,2})\), see also \(({}^{3})\). The first attempt to transfer the main results of this theory to multidimensional differential equations of the kind considered by us is the work of B. Pini \(({}^{4})\), which, unfortunately, contains errors.

3. Transversals. A regular curve \(\mathfrak L:\ y=y(t)\) \((t\in\Delta)\), lying in the domain \(G\), is called a transversal if
\[ dy(t)/dt \not\equiv f(y(t))E_x \qquad (t\in\Delta). \tag{4} \]
Recall that regularity of a curve means that the vector function \(y(t)\) is continuously differentiable and \(dy(t)/dt\ne0\) \((t\in\Delta)\). For transversals passing through regular points, one can prove a number of assertions analogous to those which hold in the Poincaré–Bendixson theory \((({}^{3}), pp. 426–427)\).

4. Geometric properties of the torus. An integral manifold of torus type is a two-dimensional topologically compact manifold situated in three-dimensional space. Let \(\mathfrak T=\mathfrak D(E_x,\eta)\) be an integral manifold of torus type and \(\Omega\), as usual, the group of periods of the vector function \(\mathfrak D(x,\eta)\) \((x\in E_x)\). The fundamental group \(\mathfrak F(\mathfrak T)\) of the torus \(\mathfrak T\) is isomorphic to the group \(\Omega\) (for fundamental groups see, for example, \(({}^{5})\)).

According to the Jordan–Brouwer theorem \((({}^{6}), pp. 519 and 562)\), the open set \(E_y\setminus\mathfrak T\) consists of two disjoint connected open sets \(\mathfrak T_i\) and \(\mathfrak T_e\); the domain \(\mathfrak T_i\) is bounded and is called the interior of the torus, the domain \(\mathfrak T_e\) is unbounded and is called the exterior of the torus; the boundary of each of these two domains is the torus \(\mathfrak T\).

Whereas the fundamental group of the torus is invariant, the fundamental groups of its interior and exterior depend essentially on the position of this torus in three-dimensional space.

In what follows we restrict ourselves to the consideration of integral manifolds of torus type whose fundamental group of the interior is isomorphic to the group of integers.

For \(\omega\in\Omega\) let \(\overset{*}{\omega}\) denote the path \(\overset{*}{\omega}(t)=\mathfrak D(t,\omega,\eta)\) \((0\le t\le1)\), which we shall call a generator of the torus \(\mathfrak T\). Let \(\varphi(t,y)\) be a solution of the ortho-

differential equation (3), \(\varphi(0,y)=y\), and \(\varphi^\varepsilon\) is the mapping defined by the formula \(\varphi^\varepsilon(y)=\varphi(\varepsilon\,\operatorname{sgn} g,y)\), where \(\operatorname{sgn} g\) is equal to \(+1\) or \(-1\) depending on whether the vector \(g(y)\) is directed outward or inward with respect to the torus \(\mathfrak T\) for \(y\in\mathfrak T\). If \(\zeta\) is a path, then \(\zeta^\varepsilon\): \(\zeta^\varepsilon(t)=\varphi^\varepsilon\circ\zeta(t)\) \((0\leq t\leq 1)\).

The properties of the linking coefficient of two continuous cycles used below can be found in \((^6)\), Ch. 15 (for the linking coefficient we use the sign \(\times\)).

We shall call a basis \(\omega_1,\omega_2\) of the group \(\Omega\) canonical if

\[ \overset{*}{\omega}_1^{\varepsilon}\times \overset{*}{\omega}_1^{-\varepsilon}=0,\qquad \overset{*}{\omega}_1^{-\varepsilon}\times \overset{*}{\omega}_2^{\varepsilon}=1,\qquad \overset{*}{\omega}_2^{\varepsilon}\times \overset{*}{\omega}_2^{-\varepsilon}=0 \quad (\varepsilon>0); \tag{5} \]

\[ \omega_2\sim 0\qquad (\text{in }\mathfrak T_i). \tag{6} \]

It can be shown that a canonical basis exists and is unique up to replacing the basis vectors by their opposites.

The system of generators \(\overset{*}{\omega}_1,\overset{*}{\omega}_2\) corresponding to the canonical basis \(\omega_1,\omega_2\) will be called the canonical \(\overset{*}{\omega}_1\)-parallel and the \(\overset{*}{\omega}_2\)-meridian (or contractible generator).

The theorem given below is an analogue of the well-known Poincaré–Bendixson theorem (\((^3)\), pp. 436–437) and is proved using sheaf manifolds (\((^7)\), pp. 145 and 437).

Theorem 1. If the interior of an integral manifold \(\mathfrak T\) of torus type belongs to the domain \(G\), then inside the torus \(\mathfrak T\) there exists at least one singular point.

5. Structure of a neighborhood of the torus. Let the differential equation (1) have in the domain \(G\) an integral manifold \(\mathfrak T\) of torus type, and let \(y=u(x)\) be the solution representing this torus. Consider a neighborhood \(U^\varepsilon\) of the torus \(\mathfrak T\), where

\[ U^\varepsilon=\{y\in G:\ y=\varphi(t,u),\ |t|<\varepsilon,\ u\in\mathfrak T\}. \tag{7} \]

It is not hard to see that the vector function \(y=\varphi(t(x),u(x))\), \(|t(x)|<\varepsilon\), gives a parametric representation of a certain integral manifold of equation (1) if and only if it satisfies the differential equation

\[ \begin{split} t'(x)h={}&-(\Phi(t(x),u(x))f(u(x))h,\, g(\varphi(t(x),u(x))))\\ &/\,\|g(\varphi(t(x),u(x)))\|^2, \end{split} \tag{8} \]

\((\Phi(t,u)=\partial\varphi(t,u)/\partial u)\), whose right-hand side is defined for \(x\in E_x\) and \(|t|<\varepsilon\). For greater simplicity of exposition we shall restrict ourselves to studying the outer semineighborhood \(V^\varepsilon\) of the torus \(\mathfrak T\), assuming that the vector \(g(y)\) for \(y\in\mathfrak T\) is directed to the outside of the manifold \(\mathfrak T\).

1) Either in some semineighborhood of the torus \(\mathfrak T\) there are no other tori besides the one under study, or in an arbitrarily small semineighborhood there are tori distinct from \(\mathfrak T\). In the latter case all these tori contain our torus \(\mathfrak T\) and are isotopic to it.

2) Suppose that in some semineighborhood the torus \(\mathfrak T\) is isolated and that \(\omega_1,\omega_2\) is a canonical basis of the torus \(\mathfrak T\) (see item 4).

Then one can indicate a vector \(\omega=p\omega_1+q\omega_2\), where \(p,q\) are relatively prime integers, and a basis \(\omega,\omega^+\) of the space \(E_x\), such that any positive half-cylinder \(\mathfrak A^+\) intersecting a sufficiently small semineighborhood of the torus \(\mathfrak T\) has the property that, for any \(\sigma>0\), one can indicate an \(\mathscr E>0\) such that

\[ \rho\bigl(\mathfrak D(x^1\omega^+ + x^2\omega,y),\mathfrak T\bigr)<\sigma \quad\text{for }x^1>\mathscr E,\quad |x^2|<+\infty \tag{9} \]

and \(\omega\) is a basis vector of the group of periods of the half-cylinder \(\mathfrak A^+\).

3) Suppose that in some semineighborhood the torus \(\mathfrak T\) is isolated. Let the limiting set \(\mathfrak D=\mathfrak D(\infty,y)\) of a simple integral mani-

manifold \(\mathfrak H=\mathfrak D(E_x,y)\) contains \(\mathfrak T\); let \(\mathfrak L\) be a transversal, drawn through an arbitrary fixed point \(\eta\in\mathfrak T\).

Only the following cases of the behavior of \(\mathfrak H\) in a neighborhood of the torus \(\mathfrak T\) are possible:
1) there exists a half-cylinder \(\mathfrak A^+\) such that \(\mathfrak A^+\cap\mathfrak L^+=\mathfrak D\cap\mathfrak L^+\), and the points of \(\mathfrak A^+\cap\mathfrak L^+\) are two-sided limiting points for \(\mathfrak H\cap\mathfrak L^+\) (the simple case);
2) there exist two half-cylinders \(\mathfrak A^+\) and \(\mathfrak B^+\) such that \((\mathfrak A^+\cap\mathfrak L^+)\cup(\mathfrak B^+\cap\mathfrak L^+)=\mathfrak D\cap\mathfrak L^+\), and the points of \((\mathfrak A^+\cap\mathfrak L^+)\cup(\mathfrak B^+\cap\mathfrak L^+)\) are one-sided limiting points for \(\mathfrak H\cap\mathfrak L^+\) (the mixed case);
3) \(\mathfrak D\cap\mathfrak L^+\) contains some half-interval, i.e. \(\mathfrak H\) is everywhere dense in some half-neighborhood of the torus \(\mathfrak T\) (the ergodic case);
4) \(\mathfrak D\cap\mathfrak L^+\) is a perfect nowhere-dense set (the singular case).

1. Structure of the limit set. The following holds.

Theorem 2. Let the positive half-cylinder \(\mathfrak A^+=\mathfrak T(E_x,\eta)\) lie in a compact regular set \(K\subset G\). Then \(\mathfrak T=\mathfrak D(+\infty,\eta)\) is a torus.

Let an integral manifold \(\mathfrak A=\mathfrak D(E_x,\eta)\) of cylinder type lie in a compact regular set \(K\subset G\). By Theorem 2, the limit sets \(\mathfrak S=\mathfrak D(-\infty,\eta)\), \(\mathfrak T=\mathfrak D(+\infty,\eta)\) are tori. It may be shown that \(\mathfrak S\ne\mathfrak T\). Let us study the relative position of the tori \(\mathfrak S\) and \(\mathfrak T\). Let \(\xi_1^*,\xi_2^*\) be a canonical system of generators of the torus \(\mathfrak S\); let \(\omega_1^*,\omega_2^*\) be a canonical system of generators of the torus \(\mathfrak T\), and let \(\omega^*\) be a generator of the cylinder \(\mathfrak A\). Let \(\omega=m\xi_1^*+n\xi_2^*\), \(\omega=p\omega_1^*+q\omega_2^*\) (see §§ 4 and 5).

Suppose that \(\mathfrak A\) tends to \(\mathfrak S\) and \(\mathfrak T\) from the outside. Then

\[ m\xi_1^*\times \omega_1^*=q,\qquad p\xi_1^*\times \omega_1^*=n. \tag{10} \]

We see that, except for the case \(n=q=0\), the parallels are homologically, and therefore also homotopically, linked.

Suppose that \(\mathfrak A\) tends to \(\mathfrak S\) from the inside, and to \(\mathfrak T\) from the outside. Then the relations hold

\[ n\omega_1^*\times \xi_2^*=q,\qquad p\omega_1^*\times \xi_2^*=m. \tag{11} \]

The structure of the limit set \(\mathfrak D=\mathfrak D(\infty,\eta)\) of a simple integral manifold is described by the following theorem.

Theorem 3. Let the simple integral manifold \(\mathfrak H=\mathfrak D(E_x,\eta)\) lie in a compact regular set \(K\subset G\). Then only the following four cases are possible: there exist two distinct tori \(\mathfrak S\) and \(\mathfrak T\), \(\mathfrak S_i\subset \mathfrak T_i\), such that:
1) \(\mathfrak D=\mathfrak S\cup\mathfrak A\cup\mathfrak T\), where \(\mathfrak A\) is a cylinder and \(\mathfrak A\subset \mathfrak T_i\) (the simple case);
2) \(\overline{\mathfrak D}=\mathfrak T_i\setminus \mathfrak S_i\) (the ergodic case);
3) \(\mathfrak D\subset \overline{\mathfrak T_i\setminus \mathfrak S_i}\), moreover \(\mathfrak S\cup\mathfrak T\subset \mathfrak D\) and \(\mathfrak D\ne \overline{\mathfrak T_i\setminus \mathfrak S_i}\) (the singular case);
there exist two distinct tori \(\mathfrak S\) and \(\mathfrak T\) and two distinct cylinders \(\mathfrak A\) and \(\mathfrak B\) such that
4) \(\mathfrak D=\mathfrak S\cup\mathfrak A\cup\mathfrak B\cup\mathfrak T\) (the mixed case).

1. Some of the assertions set forth above arose as a result of conversations between the author and A. D. Myshkis, to whom the author expresses sincere thanks. For his constant interest in my work I express deep gratitude to Academician I. G. Petrovskii.

Voronezh State
University

Received
19 XI 1966

REFERENCES

\({}^{1}\) A. Poincaré, On curves defined by differential equations, Moscow–Leningrad, 1947.
\({}^{2}\) I. Bendixson, Acta Math., 24 (1901).
\({}^{3}\) E. A. Coddington, N. Levinson, Theory of Ordinary Differential Equations, IL, 1958.
\({}^{4}\) V. B. Fin, Rend. Sem. Mat. Univ. Padova, 22, No. 1 (1953).
\({}^{5}\) L. S. Pontryagin, Continuous Groups, Moscow, 1954.
\({}^{6}\) P. S. Aleksandrov, Combinatorial Topology, Moscow, 1947.
\({}^{7}\) Hu Sze-tsen, Homotopy Theory, Moscow, 1964.