CYBERNETICS AND CONTROL THEORY

M. N. VAINŠVAIG

ON THE POWER OF CIRCUITS OF FUNCTIONAL ELEMENTS

(Presented by Academician A. I. Berg on 10 VIII 1960)

One of the basic problems of cybernetics is the problem of synthesizing control systems. The specifics of this problem depend essentially on the choice of the class of control systems, and also on the method of estimating their complexity. In the present paper, the class of circuits of functional elements is taken as the class of control systems under study. As an index of the simplicity of such a circuit, a certain quantity characterizing the functioning of the circuit is chosen. In particular, this may be the average power of the circuit, the maximum power of the circuit, the maximum number of simultaneously excited elements, etc. Starting from this index, the Shannon function is introduced in the usual way. It is shown that, when the finite basis is changed, this function varies within the limits from \(C_1 n\) to \(C_2 \dfrac{2^n}{n}\), where \(C_1\) and \(C_2\) are certain constants depending on the basis. It is established that these estimates are exact.

1. We shall consider circuits \(S\) of functional elements \(\mathfrak A_i\) belonging to some set \(\mathfrak A=\{\mathfrak A_i\}\) and realizing, respectively, functions \(f_i(x_1,\ldots,x_{k_i})\) of the algebra of logic. (It is assumed that the system of functions \(\{f_i\}\) is complete.) To each element \(\mathfrak A_i\) and to a set of states of its inputs \((\sigma_1,\ldots,\sigma_{k_i})\) we assign the number
   \[ e_i(\sigma_1,\ldots,\sigma_{k_i})=e_i f_i(\sigma_1,\ldots,\sigma_{k_i}), \]
   where \(e_i\) is a positive constant.

Consider some circuit \(S\) over the basis \(\mathfrak A\). Number all its elements. At the output of each \(\mathfrak A_{ij}\) (the \(j\)-th element of the \(i\)-th type) there is realized some function
\[ \varphi_{ij}(\sigma_1,\ldots,\sigma_n) \]
of the states of the input poles (inputs) of the circuit, and, consequently, to the element \(\mathfrak A_{ij}\) and to the set \((\sigma_1,\ldots,\sigma_n)\) there will be assigned the number
\[ e_{ij}(\sigma_1,\ldots,\sigma_n)=e_i\cdot \varphi_{ij}(\sigma_1,\ldots,\sigma_n). \]
Introduce the notation
\[ E_{\mathfrak A}(S,\sigma_1,\ldots,\sigma_n)=\sum_{ij} e_{ij}(\sigma_1,\ldots,\sigma_n). \]

As indices of the simplicity of a circuit, introduce the numbers
\[ E_{\mathfrak A}(S)=\frac{1}{2^n}\sum_{(\sigma_1,\ldots,\sigma_n)} E_{\mathfrak A}(S,\sigma_1,\ldots,\sigma_n),\qquad E_{\mathfrak A}^{*}(S)=\max_{(\sigma_1,\ldots,\sigma_n)} E_{\mathfrak A}(S,\sigma_1,\ldots,\sigma_n). \]

In content, they may be interpreted, for example, as the average and, respectively, maximum power of the circuit \(S\).

Introduce the functions
\[ E_{\mathfrak A}(f)=\min_S E_{\mathfrak A}(S),\qquad E_{\mathfrak A}^{*}(f)=\min_S E_{\mathfrak A}^{*}(S), \]
where the minimum is taken over all circuits \(S\) realizing the function \(f\);
\[ E_{\mathfrak A}(n)=\max_f E_{\mathfrak A}(f),\qquad E_{\mathfrak A}^{*}(n)=\max_f E_{\mathfrak A}^{*}(f), \]

the maximum is taken over all functions \(f\) of \(n\) arguments. Since all the numbers \(e_i\) are finite and therefore do not affect the order of the functions \(E_{\mathfrak A}(n)\) and \(E_{\mathfrak A}^{*}(n)\), for simplicity we shall assume \(e_i=1\) \((i=1,2,\ldots)\). In this case the simplicity index of a circuit \(S\) will be determined by the number of its excited elements*.

1. It is obvious that \(E_{\mathfrak A}(n)\leq E_{\mathfrak A}^{*}(n)\leq L_{\mathfrak A}(n)\), where \(L_{\mathfrak A}(n)\) is the least number such that any function \(f(x_1,\ldots,x_n)\) of the algebra of logic in \(n\) arguments can be realized by a circuit \(S\) over \(\mathfrak A\) in which the number of elements does not exceed \(L_{\mathfrak A}(n)\).

Theorem 1. For any basis \(\mathfrak A\),

\[ E_{\mathfrak A}^{*}(n)\lesssim \frac{2^n}{n}. \]

Indeed, \(L_{\mathfrak A}(n)\lesssim 2^n/n\) (4) for any basis \(\mathfrak A\), and hence \(E_{\mathfrak A}^{*}(n)\lesssim 2^n/n\).

Theorem 2**. For any finite basis \(\mathfrak A\) there is a constant \(C>0\), depending on the basis, such that

\[ Cn\lesssim E_{\mathfrak A}(n). \]

Proof. Consider a circuit \(S\) over an arbitrary but finite basis \(\mathfrak A\), realizing the function \(x_1+x_2+\cdots+x_n\pmod 2\). This circuit has the following property: for each fixed state of any \(n-1\) inputs \(x_1,\ldots,x_{i-1},x_{i+1},\ldots,x_n\) of the circuit, when the input \(x_i\) is changed the state of the output of the circuit changes, and consequently the state changes of at least one element directly connected with the input \(x_i\). Let \(k\) be the maximum number of inputs of elements from \(\mathfrak A\). Then each element of the circuit, and hence each excited element as well, can be directly connected with no more than \(k\) inputs of the circuit \(S\). It follows that the average number of excited elements directly connected with the inputs of the circuit is not less than \(n/2k\), i.e., a fortiori,

\[ E_{\mathfrak A}(n)\geq \frac{n}{2k}. \]

The theorem is proved.

1. Theorem 3. There exists a finite basis \(\mathfrak A\) such that

\[ E_{\mathfrak A}^{*}(n)<Cn, \]

where \(C\) is some positive constant***.

Proof. Consider a basis \(\mathfrak A'\) containing the following elements: \(\mathfrak A_0\) realizes the disjunction \(x\vee y\); \(\mathfrak A_1\) realizes the negation \(\bar x\); \(\mathfrak A_2\) realizes the conjunction \(x\&y\).

* It can be shown that if to each set of variables there is assigned some positive probability of its being supplied to the input of the circuit, then all the main results remain valid.

** For an infinite basis Theorem 2 is false. An example is the basis \(\mathfrak A\) consisting of neurons (1). Indeed, in this basis any conjunction of the form \(x_1^{\sigma_1},\ldots,x_n^{\sigma_n}\) of arbitrary but fixed length \(n\) is realized by one element; moreover, in a multi-pole realizing all such conjunctions, for any state of its inputs exactly one element will be excited. A disjunction of any length is also realized by one element. Consequently, any function \(f(x_1,\ldots,x_n)\) (\(n\) arbitrary) can be realized by a circuit \(S\) over \(\mathfrak A\) corresponding to its representation in perfect disjunctive normal form and having, by what has been said, no more than two excited elements, i.e. \(E_{\mathfrak A}^{*}(n)\leq 2\).

*** It can be shown that if \(e_i=1\) \((i=1,2,\ldots)\), then for any \(\varepsilon>0\) there exists a basis \(\mathfrak A\) such that \(E_{\mathfrak A}^{*}(n)<\varepsilon n\); if, however, \(e_i\) is equal to the number of inputs of the element \(\mathfrak A_i\), then there exists a constant \(C\), independent of the basis, such that \(E_{\mathfrak A}(n)>Cn\).

Lemma 1. There exists a multi-output circuit \(S'\) over \(\mathfrak A\) with \(n\) inputs and \(2^n\) outputs, realizing all conjunctions of the form \(x_1^{\sigma_1}, \ldots, x_n^{\sigma_n}\) and such that

\[ E_{\mathfrak A'}^{*}(S') < 2n . \]

Proof will be by induction on \(n\). For \(n=1\) the assertion of the lemma is valid. Indeed, in this case the multi-output circuit \(S'\) can be constructed from one element \(\mathfrak A_1\), and therefore
\[ E_{\mathfrak A'}^{*}(S') = 1 < 2n . \]

Suppose that there exists a multi-output circuit \(S_1\) with \(n=n_1-1\) inputs, realizing all conjunctions of length \(n_1-1\), such that \(E_{\mathfrak A'}^{*}(S_1)<2(n_1-1)\). We shall prove that the lemma is true for \(n=n_1\). We extend the multi-output circuit \(S_1\) to \(S'\) as follows: 1) add to \(S_1\) the input \(x_{n_1}\); 2) realize \(\overline{x}_{n_1}\) by an element \(\mathfrak A_1\); 3) realize all conjunctions of length \(n_1\) by elements \(\mathfrak A_2\), noting that
\[ x_1^{\sigma_1}\cdots x_{n_1-1}^{\sigma_{n_1-1}}x_{n_1}^{\sigma_{n_1}} = A\cdot x_{n_1}^{\sigma_{n_1}}, \]
where \(A\) is a conjunction of length \(n_1-1\); 4) declare the outputs of the corresponding elements \(\mathfrak A_2\) to be outputs of the multi-output circuit \(S'\).

Obviously, under this construction
\[ E_{\mathfrak A'}^{*}(S')=E_{\mathfrak A'}^{*}(S_1)+2<2n_1=2n . \]
The lemma is proved.

Lemma 2. Let the circuit \(S'\) have \(\mu\) outputs and realize a system of distinct conjunctions \(A_1,\ldots,A_\mu\), where
\[ A_i=x_1^{\sigma_{i1}}\cdots x_n^{\sigma_{in}} . \]
Then there exists a circuit \(S''\) with \(\mu\) inputs and one output such that, upon connecting the outputs of the circuit \(S'\) to the corresponding inputs of the circuit \(S''\), one obtains a circuit \(S\) realizing
\[ \bigvee_{i=1}^{\mu} A_i . \]
Moreover,
\[ E_{\mathfrak A'}^{*}(S)\le E_{\mathfrak A'}^{*}(S')+\rceil \log_2 \mu \lceil^{*}. \]

Proof. From elements \(\mathfrak A_0\) we construct a tree of length \(\rceil \log_2\mu \lceil\). This tree has one output, \(2^{\rceil \log_2\mu \lceil}\ge \mu\) inputs, and
\[ 2^{\rceil \log_2\mu \lceil-1}<\mu \]
input elements. For some input elements we identify the inputs so that the total number of inputs is equal to \(\mu\) (the inputs of one element cannot be identified with the inputs of another). We declare the inputs of the tree obtained after identification to be the inputs of the circuit \(S''\), and the output of the tree to be the output of the circuit \(S''\). Since all conjunctions are distinct, for any assignment of the variables at most one input element can be excited, and hence the total number of excited elements in \(S''\) does not exceed \(\rceil \log_2\mu \lceil\), i.e.
\[ E_{\mathfrak A_1}^{*}(S)\le E_{\mathfrak A_1}^{*}(S')+\rceil \log_2\mu \lceil . \]

The lemma is proved.

Proof of the theorem. Representing an arbitrary function \(f(x_1,\ldots,x_n)\) in the form of a perfect disjunctive normal form and noting that the number \(\mu\) of terms in it does not exceed \(2^n\), we obtain, by Lemmas 1 and 2,
\[ E_{\mathfrak A}(n)<3n^{**}. \]
The theorem is proved.

Theorem 4. There exists a finite basis \(\mathfrak A\) such that

\[ E_{\mathfrak A}(n)\succcurlyeq C\frac{2^n}{n}, \]

where \(C\) is some positive constant.

Proof. We introduce the following definitions. By a path from the \(i\)-th input to an output of a circuit \(S\) over \(\mathfrak A\) we shall mean any sequence of elements such that the output of the preceding element

\(*\) \(\rceil r\lceil\) is the least integer not less than \(r\).

\(**\) Similarly one can show that this estimate is valid for the basis of I. I. Zhegalkin \((^6)\) \(xy,\ x+y\pmod 2,\ 1\).

is connected to the input of the next one, with the input of the first element connected to the \(i\)-th input of the circuit \(S\), and the output of the last one being the output of the circuit \(S\). We shall call a circuit regular if through each of its elements there passes at least one path from some input to the output. It is obvious that for any circuit \(S\) realizing a function \(f(x_1,\ldots,x_n)\), one can construct a regular circuit \(S'\) realizing the same function and such that \(E_{\mathfrak A}(S)\ge E_{\mathfrak A}(S')\).

Consider the basis \(\mathfrak A''\), consisting of an element realizing the Sheffer stroke, i.e. \(x/y=\bar x\vee \bar y\). Circuits over this basis have the following property: if on some path the preceding element is not excited, then the following one is excited. Consider a regular circuit \(S\) over \(\mathfrak A''\). In it, obviously, all elements except one have successors. Suppose that for some state of the inputs the circuit \(S\) has \(k\) unexcited elements. Then, by virtue of the two-place nature of the function, the number of elements following them is not less than \(\left]\frac{k-1}{2}\right[\). Let \(L(S)\) be the number of elements in \(S\). Then the number of excited elements \(E(S)\) satisfies the condition

\[ E(S)=L(S)-k\ge \frac{1}{3}L(S)-1. \]

This estimate does not depend on the state of the inputs of the circuit \(S\). It is known that \(L_{\mathfrak A''}(n)\simeq \frac{2^n}{n}\) \({}^{(2)}\). This relation remains valid if one restricts oneself to the realization of functions by regular circuits. Hence

\[ E_{\mathfrak A''}(n)\gtrsim \frac{1}{3}\frac{2^n}{n}. \]

The theorem is proved.*

4. Theorem 5. For the basis \(\mathfrak A'''=\{\mathfrak A_0,\mathfrak A_1\}\), where \(\mathfrak A_0\) realizes the disjunction \(x\vee y\); \(\mathfrak A_1\) realizes negation \(\bar x\),

\[ \sqrt n\cdot 2^{n/2}\gtrsim E_{\mathfrak A'''}^{*}(n)\ge E_{\mathfrak A'''}(n)\gtrsim 2^{n/2}. \]

The proof of the lower estimate is carried out analogously to how it was done in Theorem 4; here one uses the existence of functions of \(n\) arguments such that circuits \(S\) over \(\mathfrak A'''\) realizing them contain no fewer than \(2^{n/2}\) elements \(\mathfrak A_1\).** The proof of the upper estimate is carried out by decomposing \(f(x_1,\ldots,x_n)\) with respect to the first \(k\) arguments, using Lemma 2.

Institute of Biological Physics
Academy of Sciences of the USSR

Received
5 III 1960

REFERENCES

1. S. K. Kleene, in: Automata Studies, IL, 1956, pp. 15–67.
2. O. B. Lupanov, Radiophysics, 1, No. 1, 120 (1958).
3. O. B. Lupanov, DAN, 111, No. 6, 1171 (1956).
4. D. E. Muller, I. R. E. E C-5, No. 1, 15 (1956).
5. C. E. Shannon, Bell Syst. Techn. J., 28, No. 1, 59 (1949).
6. I. I. Zhegalkin, Matem. sborn., 35, No. 3–4, 311 (1928).

* It can be shown that for the basis \(\mathfrak A''\)

\[ \frac{2}{3}\frac{2^n}{n}\gtrsim E_{\mathfrak A''}^{*}(n)\ge E_{\mathfrak A''}(n)\gtrsim \frac{1}{3}\frac{2^n}{n}. \]

Estimates of the same orders hold for all Sheffer functions.

** Proved by O. B. Lupanov by a method analogous to that set forth in \({}^{(5)}\).