UDC 513.83

MATHEMATICS

A. V. ARKHANGELSKII

A THEOREM ON THE REPRESENTATION OF A HOMEOMORPHISM MODULO SETS OF THE FIRST CATEGORY

(Presented by Academician P. S. Aleksandrov on 21 IV 1969)

In this paper a topological equivalent is established of a result of V. A. Rokhlin from the theory of measure-preserving transformations (see (¹), Theorem 10.13)*.

Terminology. Everywhere below: (X) is a metric space with a countable base; (f) is a homeomorphism of (X) onto (X); (f^0=f\cdot f^{-1}), (f^n=f\cdot f^{n-1}), (x^n=f^n(x)) for all (x\in X) and (n=0,\pm1,\pm2,\ldots). Further, (F(x)={x^n:n=1,2,\ldots}), (Q_n={x\in X:x^n=x}), (Q_0=\Lambda), and (C_n=Q_n\setminus Q_{n-1}) for all (n=1,2,\ldots). The points of the set (C=\bigcup{C_n:n=1,2,\ldots}) are called periodic. It is clear that each (Q_n) is closed in (X) and each (C_n) is open in (Q_n). A set (A\subset X) is called invariant if (f^{-1}(A)=A=f(A)). The sets (C) and (X\setminus C) are invariant. A set (A\subset X) is called rare if (F(x)\cap(X\setminus A)) is infinite for every point (x\in X\setminus C). The envelope of a family (\pi) of subsets of the set (X) is
[
\pi^={f^nA:A\in\pi,\ n=0,\pm1,\pm2,\ldots}.
]
A family (\mathcal P) of open subsets of (X) is called a generator of the homeomorphism (f) if the set of points of the space (X) at which the envelope (\mathcal P^) of the family (\mathcal P) does not form a base, i.e. the set
[
{x\in X:\text{ there exists an open }U\subset X,\ U\ni x,\text{ containing no element of the family }\mathcal P^\text{ that contains }x},
]
is a set of the first category in (X), i.e. is contained in the union of a countable family of sets nowhere dense in (X). The family (\mathcal P) is called a producing family if its envelope (\mathcal P^) is a base of the space (X). Finally, a family (\gamma) of nonempty open subsets of (X) is called a (\pi)-base of the space (X) (V. I. Ponomarev) if, for every nonempty open subset (U) of (X), there exists (G\in\gamma) such that (G\subset U).

Main result. A homeomorphism (f) of a separable metric space onto itself has a disjoint generator if and only if the set (C) of its periodic points is a set of the first category.

Let us note that in the space of diffeomorphisms of an arbitrary manifold (carrying the (C^{(r)})-topology), the diffeomorphisms whose set of periodic points is of the first category form an (everywhere dense) set of the second category (see (²)).

Lemma 1. If all points of (X) are nonisolated and there exists in (X) a disjoint generator (\mathcal P), then each (C_n) is a set of the first category in (X).

Proof. Suppose the contrary. Then, since (C_n) is open in ([C_n]), there exists an open subset (U) of (X), contained in (C_n), which is not a set of the first category in (X). Choose in (U) a point (x_0) at which (\mathcal P^*), the envelope of (\mathcal P), forms a base. Consider
[
A={x_0^0,x_0^1,\ldots,x_0^{\,n-1}}.
]
The set of elements of (\mathcal P) intersecting (A) is finite. Write them out, renumbering:
[
V_1,\ldots,V_k,
]
and put
[
W_i=V_i\cap \operatorname{Int} C_n,\quad i=1,\ldots,k
]
(where (\operatorname{Int} C_n) is the interior of the set (C_n) in (X)). Since (C_n) is invariant—

* I was asked to solve this problem by Ya. G. Sinai, to whom I am indebted for acquainting me with its existence, for the reference to the book (²), and for the standard construction used in the proof of Proposition 4.

is invariant, and (\mathcal P^*) is a base at (x_0); the family
(\mathcal B={f^j(W_i): j=0,\pm1,\pm2,\ldots\ \text{and}\ i=1,\ldots,k}) is also a base at (x_0). But since (f^l(W_i)=f^{l+n}(W_i)) for any integer (l), (\mathcal B) is finite—in contradiction with the non-isolatedness of (x_0). Lemma 1 is proved.

The following are evident:

Lemma 2. The set (X(\mathcal P)) of points of the space (X) at which the envelope (\mathcal P^) of the family (\mathcal P) of open sets forms a base is invariant.*

Lemma 3. With respect to the restriction of the homeomorphism (f) to the space (X(\mathcal P)), the family
[
\mathcal P'={U\cap X(\mathcal P): U\in\mathcal P}
]
forms a generating family. (The notation is the same as in Lemma 2).

Proposition 1. If (X) has a disjoint generator, then there exists a set (X') of first category in (X) such that (X''=X\setminus X') is invariant, and some disjoint open (in (X'')) cover of the space (X'') is a generating family for the restriction of (f) to (X).

Main lemma. If (A) is a closed rare set and (U) is a nonempty open set, then there exists a nonempty open set (V\subset U) such that (A\cup V) is a rare set.

Lemma 4. If (A) is a rare set, (U) is an open set, and the set (A\cup U) is not rare, then there exists a nonperiodic point (y\in U) such that (F(y)\subset A\cup U).

Proof. There exists a nonperiodic point (x\in X) for which the set
[
F(x)\cap (X\setminus (A\cup U))
]
is finite. We have: (x^{n'}\ne x^{n''}) if (n'\ne n''). Let
[
F(x)\cap (X\setminus (A\cup U))={x^{n_1},\ldots,x^{n_k}}.
]
Since (A) is a rare set, (F(x)\setminus A) is infinite. Therefore, there exists an index (N) such that (x^N\in F(x)\setminus A) and (N>n_i,\ i=1,\ldots,k). Then (x^N\in U) and
[
F(x^N)={z\in F(x): z=x^n,\ \text{where } n>N}\subset A\cup U.
]

Proof of the main lemma. If (A\cup U) is a rare set, the problem is solved. Let the set (A\cup U) be not rare. By Lemma 4 there then exists a nonperiodic point (y\in U) for which (F(y)\subset A\cup U), i.e. (F(y)\setminus A\subset U). Since (A) is a rare set, (F(y)\setminus A) is infinite; consequently, there exists (y'\in F(y)\setminus A,\ y'\ne y). Then (y'\in U\setminus A,\ y'=y^k), i.e. (y'=f^k(y)) for some (k\ge1). Choose disjoint open neighborhoods (Oy'\subset U\setminus A,\ Oy\subset U) of the points (y') and (y). By continuity of the mapping (f^k) there exists an open (W\ni y), (W\subset Oy), such that (f^k(W)\subset Oy'). Then
[
f^k(W)\cap W=\Lambda,\qquad f^k(W)\cap A=\Lambda,\qquad W\ne\Lambda,\qquad W\subset U.
]
We shall show that (A\cup W) is a rare set. Suppose the contrary. By Lemma 4 it follows that there exists a nonperiodic point (z\in W) for which (F(z)\subset A\cup W). But
[
f^k(W)\subset X\setminus (A\cup W),
]
as established above. Consequently,
[
f^k(z)\in F(z)\setminus (A\cup W),
]
a contradiction. The main lemma is proved.

Proposition 2. In (X) there exists a countable (\pi)-base, the union of the closures of any finite number of whose elements is a rare set.

Proof. We shall construct such a (\pi)-base by induction. Let
[
\mathcal B={U_n:\ n=0,1,2,\ldots}
]
be a base of the space (X), with (U_0=\Lambda) and (U_n\ne\Lambda) for (n\ge1). Put (V_0=\Lambda). Suppose that open sets (V_i) have already been defined for (i=0,\ldots,k) in such a way that (V_i\subset U_i) and
[
\cup{[V_i]: i=0,\ldots,k}
]
is a rare set. By the main lemma there then exists a nonempty open set (G\subset U_{k+1}) for which
[
(\cup{[V_i]: i=0,\ldots,k})\cup G
]
is a rare set. Fix such a (G) and choose (using the regularity of (X)) a nonempty open set (W) for which ([W]\subset G). Put (V_{k+1}=W). It is clear that
[
{V_i: i=1,2,\ldots}
]
is the desired (\pi)-base of the space (X).

Proposition 3. If the set of periodic points is empty, then there exists a disjoint generator in (X).

Proof. Choose in (X) a (\pi)-base (\mathcal P) satisfying the requirement of Proposition 2. In (X) there exists, moreover, a (\pi)-base
[
\mathcal P'={W_n:\ n=1,2,\ldots},
]
where all (W_n) are nonempty, such that
[
\lim_{n\to\infty}\operatorname{diam} W_n=0.
]

(Such a (\pi)-base can easily be constructed by induction: in the proof of Proposition 2 the sets (V_n), obviously, could have been chosen so that, in addition, the condition (\operatorname{diam} V_n < 1/n,\ n=1,2,\ldots), were satisfied.) We define by induction pairwise disjoint sets (G_n \in \mathscr P) and natural numbers (l_n) for which (f^{-l_n}(G_n) \subset W_n,\ n=1,2,\ldots). If (k=0) or (k \geqslant 1) and for all (i=1,\ldots,k) the sets (G_i \in \mathscr P) and the numbers (l_i) have already been defined, define (G_{k+1}) as follows. Consider (X_k=\cup{[G_i]: i=1,\ldots,k}). This is a rare set by the choice of (\mathscr P) (if (k=0), then (X_k=\Lambda), a rare set). Choose in (W_{k+1}) any point (x). Then there exists a natural (l_{k+1}) such that (x^{l_{k+1}}\in X\setminus X_k). We have (f^{-l_{k+1}}(x^{l_{k+1}})=x). By the continuity of the mapping (f^{-l_{k+1}}) there exists an open (U\ni x^{l_{k+1}}) for which (f^{-l_{k+1}}(U)\subset W_{k+1}) and (U\subset X\setminus X_k). There exists (G_{k+1}\in\mathscr P,\ G_{k+1}\subset U) (for (\mathscr P) is a (\pi)-base). Then (G_{k+1}\cap G_j=\Lambda) for (j\ne k+1) and (f^{-l_{k+1}}(G_{k+1})\subset W_{k+1}). Let, in accordance with the procedure described, (G_n) and (l_n) be defined for all (n=1,2,\ldots). Put (\Gamma_n=f^{-l_n}(G_n)). From (\Gamma_n\subset W_n) it follows that (\lim_{n\to\infty}\operatorname{diam}\Gamma_n=0). Consider the open sets
[
O_i=\cup{\Gamma_n:\operatorname{diam}\Gamma_n<1/i},\quad i=1,2,\ldots
]
From the fact that (\mathscr P') is a (\pi)-base and from the nonemptiness of all (G_i) it follows that (\gamma={\Gamma_n:n=1,2,\ldots}) is a (\pi)-base (of the space (X)). Hence every (O_i) is everywhere dense in (X). Thus the set (A=\cap{O_i:i=1,2,\ldots}) is the complement of a set of first category in (X). From (x\in A) it follows that there exists a sequence ({\Gamma_{n(i)}: i=1,2,\ldots}), where (\operatorname{diam}\Gamma_{n(i)}<1/i), and each (\Gamma_{n(i)}) contains (x). But then ({\Gamma_{n(i)}: i=1,2,\ldots}) is a base at the point (x). From the definition of the sets (\Gamma_n) it follows that all of them belong to the hull of the family ({G_i:i=1,2,\ldots}). Proposition 3 is proved. The main result follows easily from it.

Definition 1. A homeomorphism (f) is called scattered if there exists a disjoint open cover of the space (X) which is a generating family.

From Propositions 1 and 3 it follows that

Main Proposition. Let (f) be a homeomorphism of a separable metric space (X) onto (X), and suppose that the set (C) of periodic points is a set of first category in (X).

Then there exists an invariant set (X^\subset X) such that: a) (X\setminus X^) is a set of first category and b) the restriction (f^) of the homeomorphism (f) to the subspace (X^) of the space (X) is a scattered homeomorphism of (X^) onto (X^).

Let (N) be the set of all integers and let (N^\infty) be the space of all two-sided sequences of integers, endowed with the product topology. In other words, (N^\infty) is the set of all mappings of (N) into (N), endowed with the topology of pointwise convergence. An arbitrary element of the set (N^\infty) will be denoted by (z={z_k:k=0,\pm1,\pm2,\ldots}), where (z_k\in N). As usual, the shift (T:N^\infty\to N^\infty) is the mapping described by the rule (T(z)=z'\Longleftrightarrow z_k=z'_{k+1}). Obviously, (T) is a homeomorphism of the space (N^\infty) onto itself. By a representation of a homeomorphism (f) in (N^\infty) we mean any homeomorphism (\varphi:X\to N^\infty) (not necessarily (\varphi(X)=N^\infty)) such that (\varphi f=T\varphi). If a representation exists for (f), one says that (f) is representable. A homeomorphism (f) is called almost representable if there exist an invariant set (X'), whose complement is a set of first category in (X), and a homeomorphism (\varphi:X'\to N^\infty) which is a representation of the restriction (f) to (X') in (N^\infty).

Proposition 4 (Ya. G. Sinai). Every scattered homeomorphism (f:X\to X) is representable.

Proof. Let (D={U_i:i=1,2,\ldots}) be a disjoint open cover of the space (X), which is a generating family of sets ((D) is countable, since (X) is separable). Define the mapping (\varphi:X\to N^\infty) by the rule:
[
\varphi(x)=z\Longleftrightarrow x^{-k}\in U_{z_k},\quad k=0,\pm1,\pm2,\ldots
]
Then, if (z'=\varphi(f(x))), then (U_{z'_{k+1}}\ni (f(x))^{-(k+1)}=x^{-k}), whence it follows,

that (z_k=z'_{k+1}). Hence, (z'=T(z)=T(\varphi(x))) and, finally, (T\varphi=\varphi f). We shall show that (\varphi) is a homeomorphism.

I. The mapping (\varphi) is one-to-one. Indeed, if (x'\ne x''), then there exist (U_{i_1}\in D) and an integer (k_1) such that (x'\in f^{k_1}(U_{i_1})\subset X\setminus{x''}). Then ((x')^{-k_1}\in U_{i_1}) and ((x'')^{-k_1}\notin U_{i_1}). This means that (z'{k_1}=i_1) and (z''\ne i_1), where (z'=\varphi(x')), (z''=\varphi(x'')). Consequently, (z'\ne z'').

II. Let (z^=\varphi(x^)) and let ({k_1,\ldots,k_l}) be a collection of natural numbers. Put
[
W={z\in\varphi(X):\ z_{k_1}=z^{k_1},\ldots,z=z^{k_l}}
]
and
[
U=f^{k_1}(U{z^{k_1}})\cap\cdots\cap f^{k_l}(U{z^_{k_l}}).
]
Then (x^*\in U), and (U) is an open set. It is clear that
[
x\in U \Longleftrightarrow \varphi(x)\in W.
]
Since (W\subset\varphi(X)), we conclude that (\varphi(U)=W). This proves that the mapping (\varphi:X\to\varphi(X)\subset N^\infty) is continuous and open. Thus (see I), (\varphi) is a homeomorphism.

From the main proposition and Proposition 4 it follows that

Theorem 1. Every homeomorphism of a separable metric space (X) onto itself for which the set of periodic points is a set of the first category in (X) is almost representable.

Corollary. If (f) is a homeomorphism of a complete separable metric space (X) onto itself for which the set of periodic points is a set of the first category in (X), then there exist an invariant subset (X^) of the second category in (X) and an invariant under (T) subset (Y^\subset N^\infty) of the second category in ([Y^]) such that the restriction (f^=f|X^:X^\to X^) and the restriction (T^=T|Y^:Y^\to Y^) are topologically equivalent.*

Remark. Let us consider to what extent the last result is final. The set of periodic points of a homeomorphism (T:N^\infty\to N^\infty) is countable. Since the property of being a set of the second category is topologically invariant, we conclude that the restriction we imposed on the set of periodic points is not only sufficient but also necessary (see also Proposition 2). Further, the conclusion of the last assertion cannot be strengthened to the requirement that (Y^) be everywhere dense in (N^\infty), for in the latter case there would be a point in (Y^) whose trajectory has closure equal to (N^\infty), and then the homeomorphism (f) would also have an everywhere dense trajectory in (X) (i.e., would be topologically transitive). However, the following generalization of a known result of M. A. Lavrent’ev ((^3)) holds:

Theorem 2. Let (X,Y) be metric spaces, let (X^) and (Y^) be everywhere dense subspaces of the spaces (X) and (Y), respectively, and let (\varphi:X^\to Y), (f:X^\to X^), (g:Y^\to Y^) be homeomorphisms onto their images for which (g\varphi=\varphi f). Then there exist subspaces (\widetilde X^\supset X^), (\widetilde Y^\supset Y^) of type (G_\delta) in (X) and (Y), respectively, and homeomorphisms onto their images (\widetilde\varphi:\widetilde X^\to\widetilde Y^), (\widetilde f:\widetilde X^\to\widetilde X^), (\widetilde g:\widetilde Y^\to\widetilde Y^) such that (\widetilde f|X^=f), (\widetilde g|Y^=g), (\widetilde\varphi|X^=\varphi).

Theorem 2 reduces the question of weak topological equivalence of homeomorphisms to the question of topological equivalence of some restrictions of these homeomorphisms to everywhere dense subspaces.

Faculty of Mechanics and Mathematics
Moscow State University
named after M. V. Lomonosov

Received
26 III 1969

REFERENCES

(^1) V. A. Rokhlin, UMN, 22, no. 5 (137), 3 (1967). (^2) R. Abraham, J. Robbin, Transversal Mappings of Flows, N. Y., 1967. (^3) M. A. Lavrent’ev, Fund. Math., 6 (1924).