CYBERNETICS AND CONTROL THEORY

V. I. VARSHAVSKII

ON THE COMPLEXITY OF NETWORKS OF DEPTH TWO CONSTRUCTED FROM THRESHOLD ELEMENTS

(Presented by Academician P. S. Novikov on 31 III 1961)

A network of depth two that realizes some logical function of \(n\) variables consists of \(k\) input TEs*, each having no more than \(n\) inputs, and one output TE having \(k\) inputs, which are the outputs of the input TEs. The behavior of each TE is described by the expression

\[ y_l=\operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}x_j-\eta_l\right) \qquad (l=0,1,2,\ldots,k-1). \tag{1} \]

The behavior of the output TE is described by the expression

\[ z=\operatorname{sign}\left(\sum_{l=0}^{k-1}v_l y_l-\widetilde{\eta}\right). \tag{2} \]

Let the logical function be specified by the set of vertices of the unit hypercube \(T\), at which the function assumes the value one, and by the set \(F\) for the values of the function zero. Let the vectors be

\[ t=(t_0,t_1,\ldots,t_{n-1})\in T,\qquad f=(f_0,f_1,\ldots,f_{n-1})\in F. \]

Then

\[ \operatorname{sign}\left[\sum_{l=0}^{k-1}v_l\operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}t_j-\eta_l\right)-\widetilde{\eta}\right]=1, \]

\[ \operatorname{sign}\left[\sum_{l=0}^{k-1}v_l\operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}f_j-\eta_l\right)-\widetilde{\eta}\right]=0. \tag{3} \]

Taking into account the integrality condition for the coordinates and coefficients, we obtain

\[ \sum_{l=0}^{k-1}v_l\operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}t_j-\eta_l\right)-\widetilde{\eta}\geq 0, \]

\[ -\sum_{l=0}^{k-1}v_l\operatorname{sing}\left(\sum_{j=0}^{n-1}\xi_{lj}f_j-\eta_l\right)+\widetilde{\eta}\geq 1. \tag{4} \]

A solution of system (4) with respect to \(v_l,\ \xi_{lj},\ \eta_l,\ \widetilde{\eta},\ k\) \((l=0,1,2,\ldots,k-1;\ j=0,1,2,\ldots,n-1)\) gives a complete description of the network of depth two. Of essential interest is an estimate of the quantity \(k\) from above.

* TE is a threshold element—an element whose behavior is determined by an expression

\[ y=\operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_j x_j-\eta\right) \]

(see (1)); in the foreign literature, an element “of majority decision” or a “ballot-box” element.

We shall call the \(j\)-th tier relative to a vertex \(l\) in the unit \(n\)-dimensional hypercube the set of vertices \(\mathcal{R}_l^j\) that differ from \(l\) in \(j\) coordinates (\(l\) is the number of the vertex \(i\)). A vertex is its own zero tier. The vertex \(\bar{i}\) is the \(n\)-th tier of the vertex \(i\).

Theorem. For arbitrary \(T\) and \(F\), system (4) has a solution for \(k \leq n\) (any logical function of \(n\) variables is realized by a network of depth two consisting of no more than \((n+1)\) threshold elements).

Proof. Consider four cases.

Case 1. Let the function \(\varphi(x_0,x_1,\ldots,x_{n-1})\) be specified by a set \(T\) consisting of \(2^{n-1}\) isolated vectors \(t\). Such a function can be represented as the sum of \(n\) variables modulo 2.

If \(n=2p\), then \(T=\bigcup_{j=0}^{p-1}\mathcal{R}_l^{2j+1}\) and \(F=\bigcup_{j=0}^{p}\mathcal{R}_l^{2j}\), where \(l\) is the number of some vertex \(f\in F\). If \(n=2p+1\), then \(T=\bigcup_{j=0}^{p}\mathcal{R}_l^{2j+1}\), and \(F=\bigcup_{j=0}^{p}\mathcal{R}_l^{2j}\) under the same assumptions concerning \(l\). Let \(i_a=(a_0,a_1,\ldots,a_{n-1})\). If \({}^{r}S_l\) is a star with vertex \(l\), separated by the hyperplane with equation:

\[ \sum_{j=0}^{n-1}(-1)^{1-a_j}x_j-\sum_{j=0}^{n-1}a_j+r=0, \tag{5} \]

then it is not difficult to verify \((^1)\) that

\[ {}^{r}S_l=\bigcup_{j=0}^{r}\mathcal{R}_l^j, \]

whence

\[ \mathcal{R}_l^r={}^{r}S_l\cap \overline{{}^{\,r-1}S_l}. \]

Let \(n=2p\); then

\[ T=\bigcup_{j=0}^{p-1}\mathcal{R}_l^{2j+1} =\bigcup_{j=0}^{p-1}\left({}^{2j+1}S_l\cap \overline{{}^{\,2j}S_l}\right). \tag{6} \]

Let us note that

\[ {}^{r}S_l\cap {}^{\,r+q}S_l={}^{r}S_l,\qquad {}^{r}S_l\cap \overline{{}^{\,r+q}S_l}=0. \]

Then in (6), under the union sign, all possible intersections of the \(n\) stars entering into (6), taken three at a time, can be formed. It is not difficult to observe that such a function of \(n\) stars is realized by a threshold element with the excitation condition

\[ -\sum_{j=0}^{n-1}(-1)^j\cdot {}^{j}S_l \geq 3-p, \tag{7} \]

where \(n=2p\). Now let \(n=2p+1\); then

\[ T=\bigcup_{j=0}^{p}\mathcal{R}_l^{2j+1} =\bigcup_{j=0}^{p}\left({}^{2j}S_l\cap \overline{{}^{\,2j-1}S_l}\right). \tag{8} \]

Expression (8), analogously to expression (6), can be represented in the form of the union of \(C_n^3\) possible intersections of \(n\) stars taken three at a time. In this case the output threshold element must have the excitation condition

\[ \sum_{j=0}^{n-1}(-1)^j\cdot {}^{j}S_l \geq 3-p, \tag{9} \]

where \(n=2p+1\).

Case 2. Let the function \(\varphi(x_0,x_1,\ldots,x_{n-1})\) be specified by a set \(T\) consisting of \(m<2^{n-1}\) vectors \(t\). In this case there always exists a vertex \(i\in F\) such that

\[ T\cap {}^{\,n-1}S_l=T. \]

Let us complete the definition of the function \(\varphi(x_0,x_1,\ldots,x_{n-1})\) on levels no higher than the \((n-1)\)-st relative to \(I\) in such a way that a star, minimal from the point of view of the number of vertices and realizable by one threshold element, is obtained. If \(D_1\) is the set of vertices on which the completion was carried out, then \(T \cup D_1 = {}^1 C_I\), where \({}^1 C_I\) is the above-mentioned star. Then \(T = {}^1 C_I \cap \overline{D}_1\). The set \(\overline{D}_1\) contains the vertex \(i\), which is the vertex of the star \({}^2 C_I\), including all vertices belonging to \(\overline{D}_1\) and lying on the \((n-1)\)-st level relative to the vertex \(I\). The star \({}^2 C_I\) also contains vertices belonging to \(\overline{D}_1\) and lying on the \((n-2)\)-nd level relative to the vertex \(I\). The set \(\overline{D}_1\) can be represented in the form \(\overline{D}_1 = {}^2 C_I \cup D_2\). The latter partition is, obviously, not unique. Among all possible partitions there exists at least one such that, if \(\overline{D}_1 \cap \mathcal{Я}_I^{\,n-2} \ne 0\), then
\({}^2 C_I \cap \overline{D}_1 \cap \mathcal{Я}_I^{\,n-2} = \overline{D}_1 \cap \mathcal{Я}_I^{\,n-2}\); then, since \(\overline{D}_1 \cap \overline{{}^2 C_I}=D_2\), we have \(D_2 \cap \mathcal{Я}_I^{\,n-2}=0\).

Let us complete the definition of the set \(D_2\) to a star realizable by a threshold element so that
\({}^3 C_I \cap \mathcal{Я}_I^{\,n-2}=0\), i.e. \({}^3 C_I = D_2 \cup D_3\) and \(D_2 = {}^3 C_I \cap \overline{D}_3\). Then, analogously to what was said above, \(\overline{D}_3 = {}^4 C_I \cup D_4\), with \(D_4 \cap \mathcal{Я}_I^{\,n-4}=0\). It is obvious that in no more than \(n\) steps we completely define the set \(T\) as a function of \(n\) stars, i.e.

\[ T = {}^1 C_I \cap \left({}^2 C_I \cup \left({}^3 C_I \cap \left({}^4 C_I \cup (\cdots\right.\right.\right. \tag{10} \]

Formula (10) defines a function realizable by one threshold element. It follows from formula (10) that in this case the network consists of no more than \(n+1\) threshold elements.

Case 3. Let the function \(\varphi(x_0,x_1,\ldots,x_{n-1})\) be given by a set \(T\) containing \(m>2^{n-1}\) vectors. In this case the set \(F\) contains \(2^{n-1}-m<2^{n-1}\) vectors \(f\). Then, according to the preceding case, the function
\(\overline{\varphi(x_0,x_1,\ldots,x_{n-1})}\) is realized by a network consisting of no more than \(n+1\) threshold elements. The structure of this network is described by the expression

\[ \overline{\varphi(x_0,x_1,\ldots,x_{n-1})} = \operatorname{sign}\left[ \sum_{l=0}^{k-1} v_l \operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}x_j-\eta_l\right) -\tilde{\eta} \right], \]

where \(k \le n\). Note that \(1-\operatorname{sign} A=\operatorname{sign}(-A-1)\), and since

\[ \overline{\varphi(x_0,x_1,\ldots,x_{n-1})} = 1-\varphi(x_0,x_1,\ldots,x_{n-1}), \]

it follows that

\[ \varphi(x_0,x_1,\ldots,x_{n-1}) = \operatorname{sign}\left[ -\sum_{l=0}^{k-1} v_l \operatorname{sign}\left(\sum_{j=0}^{n-1}\xi_{lj}x_j-\eta_l\right) +\tilde{\eta}-1 \right], \]

i.e. in this case as well no more than \(n+1\) threshold elements are required.

Case 4. Let the function \(\varphi(x_0,x_1,\ldots,x_{n-1})\) be given by a set \(T\) containing \(2^{n-1}\) nonisolated vectors \(t\). Then there is at least one one-dimensional subcube such that, after its deletion, we obtain a function equal to one on no more than \(n-2\) levels relative to some vertex \(i\) and requiring for its realization, according to Case 2, no more than \(n\) threshold elements, i.e. the original function can be realized by a network of depth two consisting of no more than \(n+1\) threshold elements.

Thus the theorem is proved for any logical function of \(n\) variables.

Consider an example. Let the function \(\varphi(x_0,x_1,x_2,x_3,x_4)\) be given by the set
\(T=\{31,29,23,28,26,25,19,13,7,20,18,17,10,6,3,8\}\).
In Table 1 the vertices of the cube are arranged by levels relative to the vertex 31. The synthesis is carried out according to Case 2 of the theorem. The table gives the values

Table 1

stars and the completing sets \(D\). In the last column of the table are given the combinations of values of the functions corresponding to the stars, which determine the behavior of the output TE. Using the method of paper \({}^{1}\), it is easy to obtain the parameters of all the TEs of the network:

\[ \begin{aligned} y_{1} &= \operatorname{sign}(x_{0}+2x_{1}+x_{2}+3x_{3}+2x_{4}-3),\\ y_{2} &= \operatorname{sign}(-x_{0}-x_{1}-2x_{2}-4x_{3}-2x_{4}+4),\\ y_{3} &= \operatorname{sing}(x_{0}+3x_{1}+2x_{2}+3x_{3}+x_{4}-4),\\ y_{4} &= \operatorname{sign}(-2x_{0}-3x_{1}-2x_{2}-3x_{3}-x_{4}+7),\\ y_{5} &= \operatorname{sign}(x_{0}+2x_{1}+2x_{2}+x_{3}+2x_{4}-7),\\ \varphi(x_{0},x_{1},x_{2},x_{3},x_{4}) &= \operatorname{sign}(y_{1}+y_{2}+y_{3}+y_{4}+y_{5}-3). \end{aligned} \]

Leningrad Branch
of the Mathematical Institute named after V. A. Steklov
Academy of Sciences of the USSR

Received
25 III 1961

CITED LITERATURE

\({}^{1}\) V. I. Varshavskii, DAN, 139, No. 5 (1961).