B. I. PLOTKIN

ON A CERTAIN RADICAL OF THE AUTOMORPHISM GROUP OF A GROUP WITH THE MAXIMAL CONDITION

(Presented by Academician A. I. Mal'tsev on 27 X 1959)

In the present note, for automorphism groups of groups with the maximal condition, a radical is constructed, analogous to the radical of the endomorphism ring. At the end of the note some remarks are made on the radical in automorphism groups of arbitrary groups. Everywhere in the note, by an automorphism group of a group \(\mathfrak G\) we shall mean, generally speaking, some subgroup of the group of all automorphisms of the group \(\mathfrak G\).

Let \(\mathfrak G\) be a group and \(\Phi\) its automorphism group. \(\Phi\) is called externally nilpotent (\(\mathfrak G\)-nilpotent) if in \(\mathfrak G\) there is a finite normal series of \(\Phi\)-admissible subgroups, in whose factors the elements of \(\Phi\) induce identical automorphisms. If there is an analogous infinite series, then \(\Phi\) is called a stable automorphism group. The corresponding series in \(\mathfrak G\) will be called nilpotent (stable) relative to \(\Phi\).

Let \(g \in \mathfrak G\) and \(\sigma \in \Phi\). Denote
\[ [g,\sigma]=g^{-1}\cdot \sigma(g). \]
By
\[ [\mathfrak G,\Phi]=[\mathfrak G,\Phi(1)] \]
we shall denote the mutual commutator of \(\mathfrak G\) and \(\Phi\), i.e. the subgroup of \(\mathfrak G\) generated by all possible commutators \([g,\sigma]\), \(g \in \mathfrak G\), \(\sigma \in \Phi\). It is clear that in exactly the same way one may define the mutual commutator of a group and an arbitrary set of its automorphisms.

Denoting further
\[ [\mathfrak G,\Phi(n)]=[[\mathfrak G,\Phi(n-1)],\Phi], \]
we obtain a decreasing series of \(\Phi\)-commutators of the group \(\mathfrak G\). It is known that this series is normal and that \(\Phi\) is externally nilpotent if and only if this series reaches the identity of the group \(\mathfrak G\) after a finite number of steps.

We shall also introduce the notation
\[ [g,\sigma]=[g,\sigma(1)];\ldots;\ [g,\sigma(n)]=[[g,\sigma(n-1)],\sigma]. \]
An automorphism \(\sigma\) is called a nilautomorphism if for every \(g \in \mathfrak G\) there exists an \(n=n(\sigma,g)\) such that
\[ [g,\sigma(n)]=e. \]
If \(\Phi\) is externally nilpotent, then all elements of \(\Phi\) are nilautomorphisms. Groups with the maximal condition will be called briefly \(M\)-groups. By \(R(\mathfrak G)\) we denote the locally nilpotent radical of the group \(\mathfrak G\).

Lemma 1. Let \(\mathfrak G\) be an \(M\)-group and let \(\sigma\) be its nilautomorphism. Then
\[ [\mathfrak G,\sigma]\subset R(\mathfrak G). \]

Proof. Denote by \(\bar{\mathfrak G}=\{\mathfrak G,\sigma\}\) the subgroup in the holomorph of the group \(\mathfrak G\) generated by the group \(\mathfrak G\) and the automorphism \(\sigma\). It is easy to see that the automorphism \(\sigma\), considered as an element of \(\bar{\mathfrak G}\), is a nil-element of this group*. In \((^{1,5})\) it is proved that every nil-element of an \(M\)-group lies in its radical. Hence \(\sigma \in R(\bar{\mathfrak G})\). If \(g \in \mathfrak G\), then
\[ [g,\sigma]\in R(\bar{\mathfrak G})\cap \mathfrak G=R(\mathfrak G), \]
which proves the lemma.

From the lemma we obtain, in particular, the following fact. If an \(M\)-group admits a nontrivial nilautomorphism, then it has a nontrivial nilpotent normal divisor.

Theorem 1. If \(\mathfrak G\) is an \(M\)-group and \(\Phi\) is its automorphism group, then: a) if \(\Phi\) is \(\mathfrak G\)-nilpotent, then \([\mathfrak G,\Phi]\) is a nilpotent normal divisor in \(\mathfrak G\); b) \(\Phi\) is externally nilpotent if and only if every automorphism in \(\Phi\) is a nilautomorphism.

Proof. Part a) follows directly from the lemma.

* Here and below, for brevity, when passing to the holomorph we preserve the notation for the groups \(\mathfrak G\) and \(\Phi\), as well as for their elements.

Suppose now that every automorphism from \(\Phi\) is a nil-automorphism. Consider in \(\mathfrak G\) the series
\[ E=H_0\subset H_1\subset\cdots\subset\cdots\subset H_i\subset H_{i+1}\subset\cdots\subset H_n=R(\mathfrak G)\subset\mathfrak G, \]
where the subgroups \(H_i\) are the terms of the upper central series of the radical. Since all factors \(H_{i+1}/H_i\) are abelian \(M\)-groups and on each of these factors \(\Phi\) induces a group of nil-automorphisms, it follows, by (4), that \(\Phi\) is nilpotent with respect to each factor group \(H_{i+1}/H_i\). If one takes into account that the automorphisms from \(\Phi\) induce identities in \(\mathfrak G/R(\mathfrak G)\), it follows that \(\Phi\) is \(\mathfrak G\)-nilpotent.

Relying on part a) of Theorem 1, V. G. Vil’yatsar proved (2):

Theorem 2. If \(\mathfrak G\) is an \(M\)-group and \(\Phi\) is a \(\mathfrak G\)-nilpotent group of automorphisms, then \(\Phi\) is a nilpotent group.

Theorem 3. Let \(\mathfrak G\) be an \(M\)-group and let \(\Phi\) be its locally nilpotent group of automorphisms. Then the set of all elements of \(\Phi\) that are nil-automorphisms of the group \(\mathfrak G\) is a subgroup of \(\Phi\).

Proof. Denote by \(\Phi'\) the subgroup of \(\Phi\) generated by all nil-automorphisms of the group \(\mathfrak G\) lying in \(\Phi\). Let further
\[ \overline{\mathfrak G}=\mathfrak G\Phi' \]
be the subgroup generated by \(\mathfrak G\) and \(\Phi'\) in the holomorph of the group \(\mathfrak G\). If \(\sigma\) is a nil-automorphism of the group \(\mathfrak G\) and \(\sigma\in\Phi'\), then, since \(\Phi'\) is a locally nilpotent group, \(\sigma\) is also a nil-element in \(\overline{\mathfrak G}\). But \(\overline{\mathfrak G}\) is an \(LM\)-radical group, and therefore (6) \(\sigma\in R(\overline{\mathfrak G})\). From the definition of the subgroup \(\Phi'\) it now follows that \(\Phi'\subset R(\overline{\mathfrak G})\). But every element of \(R(\overline{\mathfrak G})\) is a nil-element in \(\overline{\mathfrak G}\). Hence it follows that every automorphism from \(\Phi'\) is a nil-automorphism of the group \(\mathfrak G\), as required.

Definition. Let \(\mathfrak G\) be an \(M\)-group and let \(\Phi\) be its group of automorphisms. By \(R_{\mathfrak G}(\Phi)\) we shall denote the subgroup in \(\Phi\) generated by all its invariant \(\mathfrak G\)-nilpotent subgroups. We shall call \(R_{\mathfrak G}(\Phi)\) the external radical of the group \(\Phi\) (as distinguished from the internal radical \(R(\Phi)\)).

Theorem 4. Let \(\mathfrak G\) be an \(M\)-group and let \(\Phi\) be its group of automorphisms. Then \(R_{\mathfrak G}(\Phi)\) is an externally nilpotent group, is contained in \(R(\Phi)\), and coincides with the set of all elements of \(R(\Phi)\) that are nil-automorphisms of the group \(\mathfrak G\). \(R_{\mathfrak G}(\Phi)\) is also an \(M\)-group.

Proof. From Theorem 2 it follows that \(R_{\mathfrak G}(\Phi)\subset R(\Phi)\). By the preceding theorem every element of \(R_{\mathfrak G}(\Phi)\) is a nil-automorphism of the group \(\mathfrak G\). But then, by Theorem 1, \(R_{\mathfrak G}(\Phi)\) is an externally nilpotent group. It is known that an externally nilpotent group of automorphisms of an \(M\)-group is itself an \(M\)-group (6, 7). Hence \(R_{\mathfrak G}(\Phi)\) is an \(M\)-group. Suppose now that \(\sigma\) is a nil-automorphism of some group \(\mathfrak G\), and \(\varphi\) is an arbitrary automorphism of this group. Then the automorphism \(\varphi^{-1}\sigma\varphi\) is also a nil-automorphism of the group \(\mathfrak G\). This fact follows from the following directly verified formula:
\[ [g,\varphi^{-1}\sigma\varphi(n)]=\varphi^{-1}[\varphi(g),\sigma(n)]. \]
It follows that the set of all nil-automorphisms of the group \(\mathfrak G\) lying in \(R(\Phi)\) is a normal divisor in \(\Phi\). It is now clear that this normal divisor must coincide with \(R_{\mathfrak G}(\Phi)\).

Let us note here that, if one requires that \(\mathfrak G\) be a solvable \(M\)-group, then, relying on a well-known result of A. I. Mal’cev (4), one can show that in \(R(\Phi)\) there is a subgroup of finite index whose commutant is contained in \(R_{\mathfrak G}(\Phi)\). The following fact is easily verified. Let \(\mathfrak G\) be an \(M\)-group, \(\Phi\) its group of automorphisms, and \(\Sigma\) a normal divisor in \(\Phi\). Then
\[ R_{\mathfrak G}(\Sigma)=R_{\mathfrak G}(\Phi)\cap\Sigma. \]

Lemma 2. Let \(\mathfrak G\) be an \(M\)-group and let \(\Phi\) be its group of automorphisms. Then: a) every \(\mathfrak G\)-nilpotent subgroup of \(\Phi\) is contained in some maximal \(\mathfrak G\)-nilpotent subgroup of \(\Phi\); b) every subgroup of \(\Phi\) conjugate to an externally nilpotent subgroup of \(\Phi\) is itself externally nilpotent.

Proof. Let \(\Phi_1 \subset \Phi_2 \subset \cdots \subset \Phi_n \subset \cdots\) be a sequence of \(\mathfrak G\)-nilpotent subgroups of \(\Phi\), and let \(\Phi'=\bigcup_n \Phi_n\). Then every element of \(\Phi'\) is a nil-automorphism, and therefore \(\Phi'\) is \(\mathfrak G\)-nilpotent. But then \(\Phi'\) is an \(M\)-group, and, consequently, there is an \(n\) such that \(\Phi'=\Phi_n\). This implies assertion a). Assertion b) follows from the fact that an automorphism conjugate to a nil-automorphism is also a nil-automorphism.

Theorem 5. If \(\mathfrak G\) is an \(M\)-group and \(\Phi\) is its automorphism group, then \(R_{\mathfrak G}(\Phi)\) coincides with the intersection of all maximal \(\mathfrak G\)-nilpotent subgroups of \(\Phi\).

Proof. From the preceding lemma it follows that if \(\Sigma\) is the intersection of all maximal \(\mathfrak G\)-nilpotent subgroups of \(\Phi\), then \(\Sigma \subset R_{\mathfrak G}(\Phi)\). To prove the reverse inclusion, it is enough to show that if \(\Sigma_1\) is an invariant \(\mathfrak G\)-nilpotent subgroup in \(\Phi\) and \(\Sigma_2\) is an arbitrary \(\mathfrak G\)-nilpotent subgroup in \(\Phi\), then the subgroup \(\Sigma_1\Sigma_2\) is also nilpotent. It is not difficult to see that for the subgroup \(\Sigma_1\) one can construct a normal series of the group \(\mathfrak G\), nilpotent relative to this subgroup, consisting of \(\Phi\)-admissible subgroups. This latter series can be refined to a series nilpotent relative to \(\Sigma_2\), and, consequently, to \(\Sigma_1\Sigma_2\). Thus \(R_{\mathfrak G}(\Phi)\) is contained in every maximal \(\mathfrak G\)-nilpotent subgroup of \(\Phi\), as was required. Let us note, in particular, that in the case when \(\mathfrak G\) is a finite \(p\)-group, \(R_{\mathfrak G}(\Phi)\) coincides with the intersection of all Sylow \(p\)-subgroups of \(\mathfrak G\).

Let us give another approach to the radical \(R_{\mathfrak G}(\Phi)\). Every ascending normal \(\Phi\)-admissible series of the group \(\mathfrak G\) will, for short, be called a \(\Phi\)-series. A composition \(\Phi\)-series is a series that admits no refinements by \(\Phi\)-series. The \(\Phi\)-centralizer of a \(\Phi\)-series is the set of all elements of \(\Phi\) inducing identities on all factors of the given \(\Phi\)-series. It is known that for any \(\Phi\)-series \([H_\alpha]\) its \(\Phi\)-centralizer \(\mathfrak z_{\Phi}[H_\alpha]\) is invariant in \(\Phi\), and moreover \(\Phi/\mathfrak z_{\Phi}[H_\alpha]\) is a subdirect product of certain automorphism groups of the factors of the series \([H_\alpha]\).

Denote by \(\mathfrak z_{\mathfrak G}(\Phi)\) the set-theoretic sum of the \(\Phi\)-centralizers of all \(\Phi\)-series of the group \(\mathfrak G\). It is not difficult to show that \(\mathfrak z_{\mathfrak G}(\Phi)\) is a normal divisor in \(\Phi\), and, moreover, if \(\mathfrak G\) has composition \(\Phi\)-series, then the \(\Phi\)-centralizers of all these series coincide with \(\mathfrak z_{\mathfrak G}(\Phi)\).

Theorem 6. If \(\mathfrak G\) is an \(M\)-group and \(\Phi\) is its automorphism group, then
\[ R_{\mathfrak G}(\Phi)=\mathfrak z_{\mathfrak G}(\Phi). \]

Proof. Since every element of \(\mathfrak z_{\mathfrak G}(\Phi)\) is a nil-automorphism of the group \(\mathfrak G\), \(\mathfrak z_{\mathfrak G}(\Phi)\) is a \(\mathfrak G\)-nilpotent normal divisor in \(\Phi\). Hence \(\mathfrak z_{\mathfrak G}(\Phi)\subset R_{\mathfrak G}(\Phi)\). Conversely, for any \(\mathfrak G\)-nilpotent normal divisor \(\Sigma\) of \(\Phi\) one can construct a \(\Phi\)-series nilpotent relative to \(\Sigma\). It follows that every \(\mathfrak G\)-nilpotent normal divisor of \(\Phi\) belongs to the \(\Phi\)-centralizer of some \(\Phi\)-series and, consequently, \(R_{\mathfrak G}(\Phi)\subset \mathfrak z_{\mathfrak G}(\Phi)\).

It follows from this theorem that for an \(M\)-group \(\mathfrak G\) the factor group \(\Phi/R_{\mathfrak G}(\Phi)\) can be characterized as a subdirect product of certain automorphism groups. In particular, if \(\mathfrak G\) is a finite group and \(\Phi\) is its automorphism group, then \(\Phi/R_{\mathfrak G}(\Phi)\) is a subdirect product of automorphism groups that are irreducible in a certain sense.

In the preceding constructions, an essential role was played by the fact established by Lemma 1. We do not know whether an analogous assertion is true for arbitrary groups. In this connection, upon passing to arbitrary groups, the role of nil-automorphisms will be played by the locally stable automorphisms defined below.

An automorphism \(\sigma\) of an arbitrary group \(\mathfrak G\) is called stable if in \(\mathfrak G\) there is an ascending normal series stable with respect to \(\sigma\). An automorphism \(\sigma\) is called locally stable if in \(\mathfrak G\) there is a local system of \(\sigma\)-admissible subgroups, in each member of which \(\sigma\) acts as a stable automorphism. Every locally stable automorphism is a nil-automorphism. For the case of \(LM\)-radical groups, including, for example, \(M\)-groups, locally nilpotent and soluble groups, one can prove the equivalence of both notions. Whether this is true in the general case is unknown.

The following generalization of Lemma 1 holds.

Theorem 7. Let \(\sigma\) be a locally stable automorphism of some group \(\mathfrak G\). Then \([\mathfrak G,\sigma]\subset R(\mathfrak G)\).

Corollary. If \(\mathfrak G\) is a group and \(\Phi\) is its stable group of automorphisms, then \([\mathfrak G,\Phi]\subset R(\mathfrak G)\).

The group \(\Phi\) itself need not be locally nilpotent.

Relying on Theorem 7, one can prove, analogously to Theorem 1:

Theorem 8. If \(\mathfrak G\) is a group of finite special rank and \(\Phi\) is its group of automorphisms, then \(\Phi\) is stable if and only if every element of \(\Phi\) is a stable automorphism.

In the case of commutative groups this is a well-known fact.

Theorem 9. Let \(\mathfrak G\) be a group and \(\Phi\) its locally nilpotent group of automorphisms. Then the set of all locally stable automorphisms from \(\Phi\) is a subgroup of \(\Phi\). This subgroup is locally stable.

We call a group of automorphisms \(\Phi\) of a group \(\mathfrak G\) locally stable if, for every subgroup with a finite number of generators \(\Phi_1\subset\Phi\), there is in \(\mathfrak G\) a local system of \(\Phi_1\)-admissible subgroups, in each member of which \(\Phi_1\) acts as a stable group.

Theorem 9 suggests the following possible definition of the outer radical of the automorphism group of an arbitrary group. Let \(\mathfrak G\) be a group and \(\Phi\) its group of automorphisms. By the outer radical \(R_{\mathfrak G}(\Phi)\) of the group \(\Phi\) with respect to the group \(\mathfrak G\) we shall mean the set of all locally stable automorphisms of the group \(\mathfrak G\) lying in \(R(\Phi)\). \(R_{\mathfrak G}(\Phi)\) is a locally nilpotent and locally stable invariant subgroup of \(\Phi\), containing all subgroups of \(\Phi\) possessing the same properties. Such an outer radical may, however, fail to contain all invariant locally stable subgroups of \(\Phi\). The radical just defined can also be characterized as follows.

Theorem 10. Let \(\mathfrak G\) be a group, \(\Phi\) its group of automorphisms, and \(\Gamma\) a subgroup of the holomorph of the group \(\mathfrak G\) generated by \(\mathfrak G\) and \(\Phi\). Then
\[ R_{\mathfrak G}(\Phi)=R(\Gamma)\cap\Phi . \]

Further, let \(\mathfrak G\) be a group, \(\Phi\) its group of automorphisms, \(g\in\mathfrak G\), and \(N=(\sigma_1,\sigma_2,\ldots,\sigma_n)\) a finite set of elements of \(\Phi\). The set \(N\) is called locally nilpotent if, for every \(g\in\mathfrak G\), there exists only a finite number of elements different from the identity of the form
\[ [[\ldots[[g,\sigma_{i_1}],\sigma_{i_2}],\ldots],\sigma_{i_k}], \]
where the \(\sigma_{i_s}\) belong to \(N\) and, in general, may be repeated. A group of automorphisms \(\Phi\) is called outer locally nilpotent if every finite set of elements of \(\Phi\) is locally nilpotent. Locally stable groups of automorphisms are outer locally nilpotent groups, and for an \(LM\)-radical group \(\mathfrak G\) the converse can also be proved. What the situation is for arbitrary groups is unknown.

Received
23 X 1959

REFERENCES

1. R. Baer, Math. Ann., 133, 3, 256 (1957).
2. V. G. Vilyatser, DAN, 131, No. 4 (1960).
3. L. A. Kaluzhnin, Ukr. matem. zhurn., 11, No. 1, 38 (1959).
4. A. I. Mal’cev, Matem. sborn., 28, 3, 567 (1951).
5. B. I. Plotkin, Izv. vyssh. uchebn. zav., Matematika, No. 1, 130 (1958).
6. B. I. Plotkin, Uspekhi matem. nauk, 13, issue 4 (82), 89 (1958).
7. D. M. Smirnov, Matem. sborn., 32 (74), 1, 365 (1953).