CYBERNETICS AND CONTROL THEORY

V. I. LEVENSHTEIN

ON SOME PROPERTIES OF CODE SYSTEMS

(Presented by Academician P. S. Novikov on 24 V 1961)

In the present paper effective criteria are studied for recognizing certain properties \((^{1-6})\) of code systems. Algorithms simpler than the known ones \((^{5,6})\) are constructed for determining whether a code system has the property of unique decipherability or the property of bounded delay; an algorithm is also found for determining whether a code system has the synchronization property.

1. Let \(B=\{b_1,\ldots,b_r\}\) be some alphabet. An arbitrary (ordered) set \(V=\{v_1,\ldots,v_m\}\), consisting of nonempty words in the alphabet \(B\), will be called a dictionary over \(B\). The length of a word \(\beta\) in the alphabet \(B\) will be denoted by \(\lambda(\beta)\). A code system \(\{A,U;B,V\}\) will mean the totality of a dictionary \(U=\{u_1,\ldots,u_m\}\) over some alphabet \(A=\{a_1,\ldots,a_q\}\) and a dictionary \(V=\{v_1,\ldots,v_m\}\) over some alphabet \(B=\{b_1,\ldots,b_r\}\). The words \(u_i\in U\) will be called elementary messages, and the words \(v_i\in V\) elementary codes. Words in the alphabet \(A\) representable in the form of products \(u_{i_1}\cdots u_{i_k}\) will be called messages, and words in the alphabet \(B\) representable in the form of products \(v_{j_1}\cdots v_{j_l}\) will be called codes. The one-to-one correspondence
   \(u_i\leftrightarrow v_i,\ 1\le i\le m,\)
   induces a mapping of the set of messages onto the set of codes, under which each message \(u_{i_1}\cdots u_{i_k}\) is mapped into the code \(v_{i_1}\cdots v_{i_k}\). Since in a number of cases messages or codes may be decomposed into elementary components in more than one way, the indicated mapping, generally speaking, is not single-valued in either direction. Everywhere in what follows it is assumed that every message is decomposed into elementary messages in a unique way, and therefore the mapping of the set of messages onto the set of codes (encoding) is single-valued. A code system is said to have the property of unique decipherability if different messages are mapped into different codes.

We shall say that a code system has the property of bounded delay (cf. \((^4)\)) if it has the property of unique decipherability and there exists a number \(T\) such that, from the first \(T\) letters of any (not completely known to us) code, one can uniquely determine its first elementary code. If a code system has the property of bounded delay, then the least of the numbers \(T\) \(\bigl(T\ge \max \lambda(v_i)\bigr)\) satisfying the condition indicated above will be denoted by \(T_d\) and called the deciphering delay. The property of bounded delay makes it possible to carry out the deciphering of messages in a device with finite memory. From the first \(T_d\) letters of any code \(v_{i_1}v_{i_2}\cdots v_{i_k}\) that have entered its memory, this device determines the first elementary code \(v_{i_1}\) and remembers the place from which the deciphering of the next code should begin; then, from the first \(T_d\) letters of the code \(v_{i_2}\cdots v_{i_k}\), it determines the elementary code \(v_{i_2}\) and remembers the place from which the deciphering of the next code should begin, and so on.

We shall say that a code system has the synchronization property if it has the property of unique ...

decoding, and there exists a number \(T\) such that every word of length \(T\) or more which is the beginning of some code and the end of some code forms a code. If a code system has the synchronization property, then the smallest of the numbers \(T\) (\(T \geqslant 0\)) satisfying the condition stated above will be denoted by \(T_s\) and will be called the synchronization delay. The meaning of these definitions is as follows. If the decoding device at some moment in time begins to proceed from incorrect information about where the next code begins (a synchronization failure), then, at most after \(T_s+\max \lambda(v_i)-1\) units of time, either it will start operating exactly as if this failure had not occurred (see Fig. 1**), or a word will enter its memory that is not the beginning of any code, which will signal the failure that has occurred.

Fig. 1

1. Consider an arbitrary dictionary \(V=\{v_1,\ldots,v_m\}\) over some alphabet \(B=\{b_1,\ldots,b_r\}\). The dictionary \(V^*=\{v_1^*,\ldots,v_m^*\}\), where \(v_i^*\), \(1\leqslant i\leqslant m\), denotes the reversal of the word* \(v_i\), will be called the *reversal of the dictionary \(V\), and the code system \(\{A,U;B,V^*\}\) the reversal of the code system \(\{A,U;B,V\}\). We shall call the dictionary \(V\): 1) free; 2) strongly free; 3) completely free, if in the alphabet \(B\) there are no equalities of the form: 1) \(v_{i_1}\ldots v_{i_k}=v_{j_1}\ldots v_{j_l}\), \(i_1\ne j_1\); 2) \(v_{i_1}v_{i_2}\ldots=v_{j_1}v_{j_2}\ldots\), \(i_1\ne j_1\); 3) \(\beta v_{i_1}\ldots v_{i_k}=v_{j_1}\ldots v_{j_l}\beta\), \(0<\lambda(\beta)<\lambda(v_{j_1})\), respectively. Denote by \(K(V)\) (respectively \(H(V)\)) the set of all nonempty ends (beginnings) of words of the dictionary \(V\) which do not belong to \(V\). It is easy to see that* \(H(V)=K^*(V^*)\) and \(K(V)=H^*(V^*)\). Denote the number of elements of the set \(K(V)\) by \(N(V)\). Obviously, the number of elements of the set \(H(V)\) is equal to \(N(V^*)\).

Lemma 1. Let \(v_{p_1},\ldots,v_{p_{m_0}}\) be all those words of the dictionary \(V=\{v_1,\ldots,v_m\}\) over \(B=\{b_1,\ldots,b_r\}\), each of which is not the end of any other word from \(V\). Then

\[ N(V)\leqslant \sum_{n=1}^{m_0}\lambda(v_{p_n})-m_0[\log_r m_0]-m+r\,\frac{r^{[\log_r m_0]}-1}{r-1}, \tag{1} \]

where the estimate (1) is attained for any \(m\geqslant m_0\geqslant 1\), \(r\geqslant 2\).

For the dictionary \(V=\{v_1,\ldots,v_m\}\) over \(B\) and an arbitrary set \(V_0\) consisting of nonempty words in the alphabet \(B\), construct the right clas-

* It will be shown below that a code system possessing the synchronization property has the property of bounded delay.

** This failure may occur, for example, as a result of a malfunction in the operation of the decoding device or an error in the transmission of the code.

*** In Fig. 1 it is shown that if a segment \(\beta\) of the incoming code \(v_{i_1}v_{i_2}\ldots v_{i_k}\) is the beginning of a code, the end of a code, and has length at least \(T_s\), then, by the synchronization property, it forms some code \(v_{j_1}v_{j_2}\ldots v_{j_l}\). In this case the decoding device will operate exactly as if the code \(v_{j_1}\ldots v_{j_l}v_{i'_{k+1}}\ldots v_{i_k}\) had entered its memory.

**** The reversal of the word \(\beta=b_{i_1}b_{i_2}\ldots b_{i_t}\) is the word \(\beta^*=b_{i_t}b_{i_{t-1}}\ldots b_{i_1}\).

***** Here \(K^*(V^*)=(K(V^*))^*\), \(H^*(V^*)=(H(V^*))^*\), etc.

with \(R_n(V,V_0)\). As the initial class \(R_0(V,V_0)\) take the set \(V_0\). The classes \(R_n(V,V_0)\), \(n=1,2,\ldots\), will be defined successively as the sets of all nonempty words \(\beta\) in the alphabet \(B\) for which the equality \(\beta_{n-1}\beta=v_i\) holds, or the equality \(v_i\beta=\beta_{n-1}\), where \(v_i\in V\), \(\beta_{n-1}\in R_{n-1}(V,V_0)\). In an analogous way the left classes \(L_n(V,V_0)\) are defined, and it is easy to show that \(L_n(V,V_0)=R_n^{*}(V^{*},V_0^{*})\). We introduce the following notation:
\[ R_n(V)=R_n(V,V),\quad L_n(V)=L_n(V,V),\quad \widetilde R_n(V)=R_n(V,K(V)), \]
\[ \widetilde L_n(V)=L_n(V,H(V)). \]

Example 1. \(A=U=\{a_1,a_2,a_3,a_4\}, B=\{0,1\}, V=\{10,101,001,0111\}\).

The principal auxiliary assertion is

Lemma 2. A word \(\beta\) belongs to \(R_n(V,V_0)\), \(n=0,1,\ldots\), if and only if there exist words \(\beta_0\in V_0\), \(v_{i_1},\ldots,v_{i_k}, v_{j_1},\ldots,v_{j_l}\in V\), \(k+l=n\), such that the equality*
\[ \beta_0v_{i_1}\cdots v_{i_k}\beta=v_{j_1}\cdots v_{j_l} \]
holds, where \(\lambda(\beta)<\lambda(v_{j_l})\), or the equality
\[ \beta_0v_{i_1}\cdots v_{i_k}=v_{j_1}\cdots v_{j_l}\beta, \]
where \(\lambda(\beta)<\lambda(v_{i_k})\) (if \(k\geq 1\)); moreover, for no \(k'\) and \(l'\), \(0\leq k'\leq k\), \(0\leq l'\leq l\), does the equality
\[ \beta_0v_{i_1}\cdots v_{i_{k'}}=v_{j_1}\cdots v_{j_{l'}} \]
hold.

1. Let a code system \(\{A,U;B,V\}\) be given (as was already said earlier, the dictionary \(U\) is assumed to be free). It is obvious that the code system has the property of unique decoding if and only if the dictionary \(V\) is free. On the other hand, it can be shown that the code system has the property of bounded delay (the synchronization property) if and only if the dictionary \(V\) is strongly free (respectively, completely free).

Fig. 2. Graph \(G_R(V)\) for the code system of Example 1

Theorem 1**. In order that a code system have the property of unique decoding, it is necessary and sufficient that no class \(R_n(V)\), for
\[ 1\leq n\leq \min\bigl(N(V),N(V^{*})\bigr)+1 \]
contain elementary codes.

Theorem 2. In order that a code system possessing the property of unique decoding have the property of bounded delay, it is necessary and sufficient that some class \(R_n(V)\) with \(n\leq N(V)+1\) be empty; moreover, if it has the property of bounded delay, then
\[ \left[\frac{n_1}{2}\right]\lambda_{\min}<T_d\leq \left[\frac{n_1+1}{2}\right]\lambda_{\max}, \tag{2} \]
where \(n_1\) is the number of the first empty class \(R_n(V)\),
\[ \lambda_{\min}=\min_i\lambda(v_i),\quad \lambda_{\max}=\max_i\lambda(v_i). \]

Theorem 3. In order that a code system possessing the property of unique decoding have the synchronization property, it is necessary and sufficient that some class \(\widetilde R_n(V)\) with

* Here, as usual, the word \(v_{i_1}\cdots v_{i_k}\) is equal to the empty word if \(k=0\).

** The conditions of Theorem 1 were known earlier \((^{2})\), but the question of the fact that, in order to verify these conditions, it is sufficient to examine a bounded number of classes \(R_n(V)\), remained open.

\(n \leqslant \min (N(V), N(V^*)) + 1\) was empty; moreover, if the code system has the synchronization property, then it and its inverse have the property of bounded delay, and for \(\lambda_{\max} > 1\)

\[ \max (T_d, T_d^*) - 2(\lambda_{\max} - 1) \leqslant T_s \leqslant \left[\frac{n_2}{2}\right]\lambda_{\max}, \tag{3} \]

where \(n_2\) is the number of the first empty class \(\widetilde R_n(V)\), and \(T_d^*\) is the decoding delay of the inverse code system.

Remark 1. Theorems 1 and 3 will remain valid if, in their formulations, the classes \(R_n(V)\) and \(\widetilde R_n(V)\) are replaced by the classes \(L_n(V)\) and \(\widetilde L_n(V)\), respectively*.

From Theorems 1 and 3, Remark 1, and the obvious equalities \(R_n(V^*) = L_n^*(V)\), \(\widetilde R_n(V^*) = \widetilde L_n^*(V)\), it follows that: 1) the inverse of a code system possessing the property of unique decodability possesses the property of unique decodability; 2) the inverse of a code system possessing the synchronization property possesses the synchronization property**.

4. There exists another, graphical method for recognizing the indicated properties of code systems, close to the method of Al. A. Markov \((^{5,6})\), used by him for recognizing the first two of these properties. Construct, for the dictionary \(V\), a directed graph \(G_R(V)\). To do this, first take as the vertices of the graph the elements of the set \(K(V) \cup V\), and connect the vertex \(\beta_i\) with the vertex \(\beta_j\) by an edge directed from \(\beta_i\) to \(\beta_j\), if the equality \(\beta_i\beta_j = v\) or the equality \(\beta_i = v\beta_j\) holds, where \(v \in V\), and then merge all vertices \(\beta_i \in V\) into one common vertex and denote it by \(\bigwedge\). In an analogous way (replacing the set \(K(V)\) by \(H(V)\) and changing the direction of the edges) the graph \(G_L(V)\) is constructed.

Theorem 4. In order that a code system possess any one of the following properties: 1) the property of unique decodability; 2) the property of bounded delay; 3) the synchronization property, it is necessary and sufficient that the graph \(G_R(V)\) contain, respectively: 1) no directed cycles passing through the vertex \(\bigwedge\); 2) no directed cycles that can be reached from the vertex \(\bigwedge\); 3) no directed cycles at all.

With the aid of the graph \(G_L(V)\) one can recognize the corresponding properties of the inverse code system, since the graphs \(G_R(V^*)\) and \(G_L(V)\) are isomorphic under the correspondence \(\beta \leftrightarrow \beta^*\) between their vertices.

Received
22 V 1961

CITED LITERATURE

\(^{1}\) D. A. Huffman, Proc. IRE, 40, No. 9, 1098 (1952).
\(^{2}\) A. A. Sardinas, G. W. Patterson, Conv. Rec. IRE, IT, part 8, 104 (1953).
\(^{3}\) M. P. Schützenberger, Trans. IRE, IT-2, No. 3, 47 (1956).
\(^{4}\) E. N. Gilbert, E. F. Moore, Bell Syst. Techn. J., 38, No. 4, 933 (1959).
\(^{5}\) Al. A. Markov, DAN, 132, No. 3, 521 (1960).
\(^{6}\) Al. A. Markov, DAN, 139, No. 3 (1961).
\(^{7}\) S. W. Golomb, B. Gordon, L. R. Welch, Canad. J. Math., 10, No. 2, 202 (1958).
\(^{8}\) E. N. Gilbert, Trans. IRE, IT-6, No. 4, 470 (1960).

* In terms of the classes \(L_n(V)\) one can also answer a number of questions pertaining to other properties of code systems. In particular, it can be shown that (in the terminology of article \((^{4})\)) an exhaustive code system is (for \(\lambda_{\max} > 1\)) completely self-synchronizing if and only if

\[ H(V) = \bigcup_{n=1}^{n_3} L_n(V) \]

for some \(n_3 \leqslant \dfrac{m-r}{r-1}\).

** The inverse of a code system possessing the property of bounded delay, generally speaking, does not have this property. In particular, the inverse of a completely self-synchronizing code system mentioned in the preceding footnote never has the property of bounded delay and, consequently, the latter does not have the synchronization property. Moreover, it can be shown that all other optimal Huffman code systems \((^{1})\) likewise do not have the synchronization property (for \(m \geqslant 2r - 1\)). Examples of code systems possessing the synchronization property are systems whose dictionary \(V\) contains a word without a comma \((^{7})\) (in particular, a dictionary with a synchronizing prefix \((^{8})\)).