Full Text
UDC 62-506.3
MATHEMATICS
V. V. PETROV, A. S. USKOV
INFORMATION ASPECTS OF THE REGULARIZATION PROBLEM
(Presented by Academician A. N. Tikhonov, May 8, 1970)
In solving problems of statistical optimization of control systems, the question of the correctness of the formulation of the problem plays an important role. A problem is considered to be correctly posed if its solution exists, is unique, and is stable with respect to small variations of the initial data.
In this sense, integral equations of the first kind, to which many problems of statistical dynamics reduce, are ill-posed.
For the effective solution of these equations, A. N. Tikhonov and M. M. Lavrent'ev developed regularization methods \((^{1-4})\). In problems of statistical dynamics these methods were further developed in the works of V. V. Solodovnikov and V. L. Lenskii \((^{5,6})\).
In the present note an attempt is made to show the informational content of the regularization problem and to establish its direct connection with informational methods in control theory.
Information estimates (throughputs) characterize the limiting capabilities of a system, for the realization of which very large time delays would be required \((^{7-9})\).
These limiting capabilities correspond to physically unrealizable filters \((^{7,8})\). Thus, in order to solve the problem posed and to obtain simple informational relations, it seems most expedient to consider the regularization method using such filters as an example.
At the input of a device of the reproduction type there arrives a useful signal \(m(t)\) with noise \(n(t)\) superposed on it, so that the input signal \(\varphi(t)\) has the form
\[ \varphi(t)=m(t)+n(t). \]
Here \(m(t)\) and \(n(t)\) are stationary, random, Gaussian, mutually uncorrelated processes with known correlation functions \(R_m(t)\), \(R_n(t)\) (or spectral densities \(S_m(f)\), \(S_n(f)\)) and zero mathematical expectations.
The optimal characteristic must reproduce at the output, with minimum mean-square error (m.s.e.), the useful signal entering the input.
The regularizing functional
\[ I_1=\int_{-\infty}^{+\infty} k^2(t)\,dt \tag{1} \]
reduces the optimization problem, without taking into account the condition of physical realizability, to the minimization of the expression
\[ I=\lambda \int_{-\infty}^{+\infty} k^2(t)\,dt+\overline{\varepsilon^2}, \tag{2} \]
where
\[ \overline{\varepsilon^2} = R_m(0) + \int_{-\infty}^{+\infty}\int_{-\infty}^{+\infty} R_\varphi(t-\tau)k(t)k(\tau)\,dt\,d\tau - 2\int_{-\infty}^{+\infty} R_m(t)k(t)\,dt . \tag{3} \]
mean square random error.
\[ R_{\varphi}(t)=R_m(t)+R_n(t). \]
Here \(\lambda\) is the Lagrange multiplier, and \(k(t)\) is the desired optimal impulse response.
A necessary and sufficient condition for the minimum of the functional (2) is the integral equation of the second kind
\[ \lambda k(t)+\int_{-\infty}^{+\infty} R_{\varphi}(t-\tau)k(\tau)d\tau=\dot R_m(t). \tag{4} \]
The solution of equation (4) in the frequency domain can be represented in the form
\[ \Phi(jf)=S_m(f)/(S_m(f)+S_n(f)+\lambda). \tag{5} \]
The transfer function (5) may be regarded as the characteristic of a Shannon filter \((^{7,8})\), to whose input there is applied interference with spectral density
\[ S_n^{*}(f)=S_n(f)+\lambda. \tag{6} \]
The Lagrange multiplier \(\lambda\) corresponds to the spectral density of white noise
\[ \lambda=N^2. \tag{7} \]
The rms error corresponding to the optimal characteristic (5) is determined by the formula
\[ \overline{\varepsilon^2}=\int_{-\infty}^{+\infty} S_{\varepsilon}(f)\,df, \tag{8} \]
where
\[ S_{\varepsilon}(f)=S_m(f)[S_n(f)+N^2]/(S_m(f)+S_n(f)+N^2). \tag{9} \]
The minimum rms error \((\overline{\varepsilon^2})_{\min}\) occurs for \(N=0\). The level of white noise is found from equation (8) for a prescribed value of the rms error \((\overline{\varepsilon^2})_0\) satisfying the condition
\[ (\overline{\varepsilon^2})_0>(\overline{\varepsilon^2})_{\min}. \tag{10} \]
We shall regard the signal \(\varepsilon(t)\) as approximately limited in spectrum (frequency-band width \(W\)) and in time (duration in time \(T\)).
In a narrow frequency band \(\Delta f\) of the spectrum, the entropy of the error is \((^{3,10})\)
\[ H_{\varepsilon\Delta f}=T\Delta f\log 2\pi e S_{\varepsilon}(f)\Delta f. \tag{11} \]
The information rate of the error signal in the frequency band \(\Delta f\) is
\[ R_{\varepsilon\Delta f}=\Delta f\log 2\pi e S_m(f)\Delta f-\Delta f\log\frac{S_m(f)}{S_{\varepsilon}(f)}. \tag{12} \]
The total rate of transmission of information by the error signal is determined by summation over all frequencies
\[ R_{\varepsilon}=\int_W \log[2\pi e S_m(f)\Delta f]\,df-\int_W \log\frac{S_m(f)}{S_{\varepsilon}(f)}\,df. \tag{13} \]
By virtue of formulas (9), (13), the channel capacity can be represented in the form
\[ C=\int_W \log\frac{S_m(f)+S_n(f)+N^2}{S_n(f)+N^2}\,df. \tag{14} \]
The information per one degree of freedom of the error signal is
\[ H_n(\varepsilon)=H_n(m)-\frac{1}{W}\int_W \log\frac{S_m(f)+S_n(f)+N^2}{S_n(f)+N^2}\,df, \tag{15} \]
where
\[ H_n(m)=\frac{1}{W}\int_W \log [2\pi e S_m(f)\Delta f]\,df . \tag{16} \]
The quantity \(H_n(\varepsilon)\) characterizes the loss of information when the signal passes through the filter.
The minimum of \(H_n(\varepsilon)\) corresponds to \(N=0\).
As the level of white noise increases, the error entropy increases and, as \(N\to\infty\), tends to the entropy of the useful signal. In this case the throughput \(C\to 0\).
Analogous results can be obtained when the condition of physical realizability is taken into account,
\[ k(t)=0 \quad \text{for } t<0 . \tag{17} \]
In this case the regularizing functional (1) reduces the Kolmogorov–Wiener problem to an integral equation of the second kind,
\[ \lambda k(t)+\int_0^\infty R_\varphi(t-\tau)k(\tau)\,d\tau =R_m(t), \quad t>0 . \tag{18} \]
The equations (4), (18) obtained by the regularization method satisfy A. N. Tikhonov’s correctness conditions, and their numerical solution is simpler than that of equations of the first kind.
From the informational point of view, this is equivalent to feeding into the input of the system an additional disturbance in the form of white noise \((^{4-7,\,18})\), whose level is determined by a certain prescribed admissible error.
A decrease in dynamic accuracy in the process of regularization (10) leads to a corresponding change in the informational characteristics. The throughput decreases (14), the error entropy increases (15), and the passband of the system decreases (5). That is, by introducing a tolerance on dynamic accuracy, and consequently also on information loss, the regularization method makes it possible to simplify the problem of the physical realization of the system.
In applied information theory, such methods, using the ideas of regularization, have found broad application. The developing information theory of control has recently been based on the desire to take into account the specific features of system operation (dynamic errors due to limited energy resources) \((^{8,\,11})\). In this case an error is allowed (and is not made sufficiently small), which makes it possible to narrow the channel bandwidth and reduce the amount of auxiliary energy, and consequently the dimensions and weight of the instruments and devices entering into the system.
Several variants of the application of such an approach to questions of the design of information devices and systems \((^{6,\,8,\,11})\) may be indicated. The regularization method underlies all these variants.
Moscow Aviation Institute
named after S. Ordzhonikidze
Received
10 I 1970
CITED LITERATURE
- A. N. Tikhonov, DAN, 151, No. 3 (1963).
- A. N. Tikhonov, DAN, 153, No. 1 (1963).
- A. N. Tikhonov, DAN, 162, No. 4 (1965).
- M. M. Lavrent’ev, On Certain Incorrect Problems of Mathematical Physics, Novosibirsk, 1962.
- V. V. Solodovnikov, V. L. Lenskii, Izv. AN SSSR, Technical Cybernetics, No. 2 (1966).
- Technical Cybernetics, ed. V. V. Solodovnikov, Book 2, Ch. VIII, Moscow, 1967.
- V. N. Petrov, V. V. Petrov et al., Stochastic Process in Control and Information Systems. Survey Paper 63, IFAC, Warszawa, 1969.
- V. V. Petrov, A. S. Uskov, Reports of the seminar: Information Methods in Control, Measurement, and Monitoring Systems, Far Eastern Central Bureau of Technical Information, Vladivostok, 1968.
- T. Bodé, K. Shannon, Theory of Information and Its Applications, collection of translations edited by A. A. Kharkevich, Moscow, 1959.
- S. Goldman, Information Theory, IL, 1957.
- V. V. Petrov, V. M. Ageev, V. V. Zaporozhets, Modern Methods for Designing Automatic Control Systems, Ch. 13, Moscow, 1967.