Reports of the Academy of Sciences of the USSR
1960. Volume 135, No. 3

MATHEMATICS

A. I. KOSTRIKIN

ON THE ENGEL PROPERTIES OF GROUPS WITH THE IDENTITY RELATION \(x^{p^\alpha}=1\)

(Presented by Academician A. I. Mal’tsev on 17 VI 1960)

In the present note we report some facts connected with the notion of Engel-ness and pertaining to the groups \(B_{k,p^\alpha}\) and \(\overline{B}_{k,p}\). Let \(F\) be a free group with \(k\) free generators \((k \geqslant 2)\); let \(F^m\) be the normal divisor generated by the \(m\)-th powers of all elements of \(F\). By definition, \(B_{k,m}=F/F^m\). A remarkable theorem of P. S. Novikov \((^{1})\) asserts that \(B_{k,m}\) is a group of infinite order for every \(m \geqslant 72\). In the case when the exponent \(m\) is a power of a prime, \(m=p^\alpha\), this means that the \(p\)-group \(B_{k,p^\alpha}\) is not nilpotent. But is it Engel, i.e., does the identity \([g,h;n]=1\) hold for all elements \(g,h\in B_{k,p^\alpha}\) and some \(n\)? Here \([g,h;n]=[[g,h;n-1],h]\), \([g,h;1]=[g,h]=ghg^{-1}h^{-1}\). This question arises in connection with the circumstance that it is still unknown whether a group satisfying the \(n\)-th Engel condition is always locally nilpotent (even for \(n=3\)).

Theorem 1. The group \(B_{k,p^\alpha}\), \(p^\alpha>72\), is not Engel.

It must be emphasized that the proof of Theorem 1 proposed here is based entirely on the paper \((^{1})\) and is, in essence, only a small remark on it. P. S. Novikov proved—and this is one of the main points of his work—that an irreducible word \(w\) in \(F\), representing an arbitrary element of the subgroup \(F^m\) \((m \geqslant 72)\), contains at least two adjacent identical segments of length \(\geqslant 1\), or, in the terminology adopted in \((^{1})\), \(w\) contains a segment which is a periodic word with exponent \(\geqslant 2\) and period of length \(\geqslant 1\) (in fact, the exponent grows together with \(m\)). It turns out that the irreducible word \(w(n)\) corresponding to the element \([a,b;n]\) contains not a single periodic segment, and therefore cannot belong to the subgroup \(F^{p^\alpha}\).

The proof is by induction on \(n\). For \(n=1,2\) the assertion is not difficult to verify directly. Moreover, from the inductive definition of \([a,b;n]\) it follows rather simply that the length of a possible period must be not less than four. Suppose that the absence of periods has already been proved for the irreducible word \(w(n)\). Note that
\[ w(n)=[a,b;n]=\prod_1^{2^{\,n-1}}\left(ab^{\nu_i}a^{-1}b^{\mu_i}\right), \]
where \(\nu_i,\mu_i=\pm1\). The regular alternation of the letters \(a\) and \(a^{-1}\) makes it possible subsequently to use the following device. Denote
\[ d=aba^{-1},\quad d^{-1}=ab^{-1}a^{-1},\quad f=a^{-1}ba,\quad f^{-1}=a^{-1}b^{-1}a. \]
Neither \(d\) nor \(f\) can cancel with the letter \(b\). We have
\[ [a,b;2]=aba^{-1}bab^{-1}a^{-1}b^{-1}=dbd^{-1}b^{-1}=[d,b] \]
\[ =abfb^{-1}f^{-1}a^{-1}=a[b,f]a^{-1}. \]
Obviously,
\[ [a,b;n+1]=[d,b;n]=a[b,f;n]a^{-1}. \]

Suppose that the irreducible word \(w(n+1)\) contains a periodic segment \(T^2\). If
\[ T=b^\varepsilon a\ldots a^{-1}b^\delta \quad (\varepsilon,\delta=0 \text{ or } \pm1,\ |\varepsilon|+|\delta|=1), \]
then, obviously, the segment \(T\) must be a word in \(d\) and \(b\). But in that case the induction on \(n\) in the word \([d,b;n]\) is applicable. On the other hand, a segment \(T\) of the form
\[ b^\varepsilon a^{-1}\ldots ab^\delta \]
is a word in relation to

respectively, \(f\) and \(b\), which gives us the right to use induction in the word \(a[b,f;n]a^{-1}\), or, what is the same, in the word \([b,f;n]\), since a segment of the form under consideration cannot occupy either the extreme left or the extreme right position in the sequence \([a,b;n+1]\). There can be no segments different from those considered by us, for quite clear reasons. The theorem is proved.

Next, let \(B_i\) be the terms of the lower central series of the group \(B_{k,p}\),

\[ B_{\infty}=\bigcap_{i=1}^{\infty} B_i. \]

Then, by definition, \(\overline{B}_{k,p}=B_{k,p}/B_{\infty}\). In paper \((^2)\) it was proved that \(\overline{B}_{k,p}\) is a finite group. Let \(C_p\) be its nilpotency class, depending on \(p\) and, possibly, on the number of generators \(k\). \(B_{\infty}=B_{C_p}\) is an infinite \(p\)-group (if \(p>72\)) with a finite number of generators, with the defining relation \(x^p=1\), coinciding with its commutant and having no subgroups of finite index—a conclusion obtained by combining the results of papers \((^1)\) and \((^2)\) and mentioned as a problem already by Grün \((^3)\). The group \(\overline{B}_{k,p}\) is an Engel group, as is every finite \(p\)-group. The Engel index \(n_p\)—the least natural number for which \([g,h;n_p]=1\) for all elements \(g,h\in\overline{B}_{k,p}\)—is uniquely determined by the prime \(p\). In what follows we take \(k=2\). The trivial estimate \(n_p\ge p-1\) is sharp for \(p=2\) and \(3\), and this, in essence, is all that is known about \(n_p\).

To obtain exact values of the invariant \(n_p\) for any \(p\), or at least to find upper estimates for it, is apparently very difficult. In any case, it follows from Theorem 1 that purely combinatorial arguments in the free group \(F\), without knowledge of certain essential properties of the group \(\overline{B}_{2,p}\), are insufficient.

Theorem 2. \(n_p\ge \dfrac{3p-5}{2}\) for all primes \(p>3\).

For the proof of this inequality we use the connection between groups and rings \((^{4-6})\), taking the generators \(a\) and \(b\) in the free group \(F\) in the form \(a=e^x,\ b=e^y\). By the Baker—Hausdorff formula we obtain

\[ [e^x,e^y;n]=e^{K_n}, \]

where \(K_n=K_{n,1}+K_{n,2}+\cdots\) is an infinite series of Lie monomials, taken with rational coefficients; \(K_{n,i}\) is the collection of Lie monomials of degree \(i\) with respect to \(x\). If \(\varphi(y)=\sum_{0}^{\infty} a_i y^i\) is some formal series, then by the symbol \([x\varphi(y)]\) we shall mean the series \(\sum a_i[xy^i]\), where

\[ [xy^i]=[\ldots[[xy]y]\ldots y]. \]

Denote \(u=1-e^{-y}\). From the expression

\[ [e^x,e^y;n]=e^{K_{n-1}} e^{-[K_{n-1}e^{-y}]} \]

it is not hard to obtain the recurrence formulas

\[ K_{n,2}=[K_{n-1,2}u]+\frac{1}{2}[xu^{n-1}[xu^n]] =\frac{1}{2}\sum_{i=0}^{n-1}[xu^i[xu^{i+1}]]u^{\,n-1-i}; \tag{1} \]

\[ \begin{aligned} K_{n,3}={}&[K_{n-1,3}u]+\frac{1}{12}\{[xu^{n-1}[xu^n]^2]+2[xu^n[xu^{n-1}]^2] \\ &+6[K_{n-1,2}[xu^n]]-6[K_{n-1,2}u[xu^{n-1}]]\}. \end{aligned} \tag{2} \]

Suppose that \(n_p\le n_0=\dfrac{3p-7}{2}\). Since \(C_p\ge 2p\) \((^6)\), in particular,

\[ [e^{\alpha x},e^y;n_0]\equiv 1\pmod{F^pF_{2p-2}}, \]

where \(\alpha\) is any integer. The series \(K_n\) contains no Lie monomials of degree \(<n+1\), and the coefficients of the monomials of degrees \(n+1,n+2,\ldots,n+p-1\) do not contain \(p\) in the denominators (see \((^{5,6})\)). It should also be noted that the minimal degree with respect to \(y\) of the Lie monomials in \(K_{n,i}\) is equal to \(n\). Let \(\overline{K}_{n,i}\) be the component \(K_{n,i}\), considered modulo Lie polynomials of degree \(\ge 2p-2\). Consider the product of the ...

several elements \([e^{\alpha_i}x, e^y; n_0]\) with different numbers \(\alpha_i \not\equiv 0(p)\), chosen in such a way that as a result one obtains the element
\(w_0=e^{\nu \bar K_{n_0,2}+R_{2p-2}}\); \(\nu\) is some integer \(\not\equiv 0(p)\); \(R_{2p-2}\) is the collection of all monomials of Lie degree \(\geqslant 2p-2\). This is possible, since the law of composition of the terms \(K_{n_0,i}\) will be additive (the product \([\bar K_{n_0,i}\bar K_{n_0,j}]\) no longer contains terms of degree \(<2p-2\)). By assumption, \(w_0 \equiv 1(F^pF_{2p-2})\). If we now use the fact that the homogeneous components \(I_k\) and \(A_k\) of the ideals \(I\) and \(A\) (defined, for example, in \((^5,^6)\)) coincide for \(k\leqslant 2p\), then from this one can conclude that \(\nu \bar K_{n_0,2}\equiv 0(I)\). But this is false. Indeed, the unique monomial of least degree occurring with a nonzero coefficient in \(\nu\bar K_{n_0,2}\) and not explicitly containing products of the type \([wy^k]\), \(k\geqslant p-1\), is

\[ z=\left[xy^{\frac{p-}{2}}\left[xy^{\frac{5p-3}{2}}\right]y^{p-2}\right] \equiv (-1)^{\frac{p-3}{2}}[xy^{p-4}xy^{p-2}]\ (I). \]

Since

\[ \langle[xy^{p-1}],\,x,\,(p-2)y\rangle \equiv -2z+[xy^{p-2}xy^{p-4}]\equiv 0\ (I), \]

\[ \psi_{2,2p-6}-\varphi_{2,2p-6} =\left[\frac{2p-5}{2}\right]-2\left[\frac{p-4}{2}\right]-1=1 \]

(see § 3 of \((^6)\)), it follows that \(z\not\equiv 0(I)\). Thus,

\[ [e^x,e^y;n_0]\not\equiv 1(F^pF_{2p-2}). \]

The theorem is proved.

If one uses the recurrent formula (2), then in the case \(p=5\) not very long computations show that

\[ \bar K_{5,3}\equiv \frac14\{[x[xy][xy^2]y^3]-[x[xy]y[xy]y^3]\}\equiv 2f^1_{9,3}\ (I). \]

Here \(f^1_{9,3}\) is the notation for the monomial \([xy^2xyxy^3]\), taken from the paper \((^7)\). From Theorem 6 of \((^6)\) it follows that \(f^1_{9,3}\not\equiv 0(I)\). Therefore arguments analogous to those just described lead to the estimate \(n_5\geqslant 6\). It turns out further that all monomials of Lie algebra occurring in \(K_6\) are \(\equiv 0(I)\). One can prove that \([e^x,e^y;6]\equiv 1(F^5F_{13})\), and since \(c_5\leqslant 13\), the following assertion is true:

Theorem 3. \(n_5=6\).

Is the identity \([a,b;6]\equiv 1(F^5)\) satisfied in the free group \(F\)? In view of Theorem 3 this question is directly related to the Burnside problem for exponent 5, which, despite the result of P. S. Novikov, is of definite interest, especially in the case of its negative solution. The possibility of a deeper study of the situation arising in this case gives hope of shedding light on the structure of the mysterious group \(B_\infty\).

Steklov Mathematical Institute
Academy of Sciences of the USSR

Received
16 VI 1960

CITED LITERATURE

\(^{1}\) P. S. Novikov, DAN, 127, No. 4 (1959).
\(^{2}\) A. I. Kostrikin, Izv. AN SSSR, ser. matem., 23, No. 1, 3 (1959).
\(^{3}\) O. Grün, J. reine u. angew. Math., 182, 158 (1940).
\(^{4}\) W. Magnus, Ann. of Math., 52, No. 1 (1950).
\(^{5}\) I. N. Sanov, Izv. AN SSSR, ser. matem., 16, 23 (1952).
\(^{6}\) A. I. Kostrikin, Izv. AN SSSR, ser. matem., 21, 289 (1957).
\(^{7}\) A. I. Kostrikin, Izv. AN SSSR, ser. matem., 19, 233 (1955).