M. S. Pinsker

INFORMATIONAL STABILITY OF GAUSSIAN RANDOM VARIABLES AND PROCESSES

(Presented by Academician A. N. Kolmogorov on 7 III 1960)

As shown in \((^{1-4})\), the validity of the fundamental theorems of information theory (Shannon’s theorems) is determined by the presence of informational stability in random variables and processes. In the present note, necessary and sufficient conditions for informational stability are indicated for Gaussian random variables and processes. In addition, conditions are formulated for the applicability of the central limit theorem to the information density of such random variables and processes.

\(1^\circ\). Informational stability of random variables. Let \(\xi\) and \(\eta\) be random variables taking values in the spaces \((X,S_X)\), \((Y,S_Y)\); let \(P_{\xi\eta}(\cdot)\), \(P_\xi(\cdot)\), \(P_\eta(\cdot)\) be, respectively, the distributions of the random variables \((\xi,\eta)\), \(\xi\), \(\eta\), and let \(P_{\xi\times\eta}(\cdot)=P_\xi\times P_\eta(\cdot)\) be the direct product of the measures \(P_\xi(\cdot)\), \(P_\eta(\cdot)\).

If \(P_{\xi\eta}(\cdot)\) is absolutely continuous with respect to the measure \(P_{\xi\times\eta}\), then set

\[ i_{\xi\eta}(x,y)=\ln \frac{dP_{\xi\eta}(\cdot)}{dP_{\xi\times\eta}(\cdot)},\qquad I(\xi,\eta)=\iint_{XY} i_{\xi\eta}(x,y)\,P_{\xi\eta}(dx\,dy). \tag{1} \]

If, however, the measure \(P_{\xi\eta}(\cdot)\) is not absolutely continuous with respect to the measure \(P_{\xi\times\eta}(\cdot)\), then set \(I(\xi,\eta)=\infty\), and in the case of singularity of the measure \(P_{\xi\eta}(\cdot)\) with respect to the measure \(P_{\xi\times\eta}(\cdot)\),

\[ \frac{i_{\xi\eta}(x,y)}{I(\xi,\eta)}=1. \]

\(I(\xi,\eta)\), \(i_{\xi\eta}(\cdot,\cdot)\) are called, respectively, the information and the information density of the pair of random variables \((\xi,\eta)\).

The expression \(i_{\xi\eta}(\xi,\eta)=i(\xi,\eta)\) is a measurable function of the pair \((\xi,\eta)\) of random variables and, consequently, is itself a random variable. Obviously,

\[ I(\xi,\eta)=Mi(\xi,\eta). \tag{2} \]

A sequence of random variables \(\{(\xi^t,\eta^t)\}\), \(t=t_1,t_2,\ldots,\ \lim\limits_{n\to 8} t_n=\infty\), will be called informationally stable if, in probability,

\[ \lim_{t\to\infty}\frac{i(\xi^t,\eta^t)}{I(\xi^t,\eta^t)}=1. \]

A random variable \(\xi=\{\xi_\tau\},\ \tau\in N\), where \(N\) is some set and which is formed from one-dimensional random variables \(\xi_\tau\), is called Gaussian if, for any finite set \(\tau_1,\ldots,\tau_n\in N\) of values of the parameter \(\tau\), the joint distribution of the random variables \(\xi_{\tau_1},\ldots,\xi_{\tau_n}\) is normal.

Theorem 1. For a sequence of Gaussian random variables \((\xi^t,\eta^t)\), \(t=t_1,t_2,\ldots\), to be informationally stable, it is necessary and sufficient that the condition

\[ \lim_{t\to\infty} I(\xi^t,\eta^t)=\infty \tag{3} \]

be satisfied.

Moreover, if

\[ \lim_{t\to\infty} Di(\xi^t,\eta^t)= \lim_{t\to\infty} M\bigl(i(\xi^t,\eta^t)-I(\xi^t,\eta^t)\bigr)^2=\infty, \tag{4} \]

then the distribution of the information density \(i(\xi^t,\eta^t)\) converges to the normal distribution.

\(2^\circ\). Information stability of Gaussian random processes. Let \(\xi=\{\xi(\cdot)\}\) be a random process. Denote by \(\xi_0^T\) the segment of this process—the random variable formed from the random variables \(\xi(t)\), \(0\le t\le T\), if \(\xi\) is an ordinary random process, and from \(\xi(\varphi)\), \(\varphi\in\Phi\), \(\varphi(t)=0\) for \(t\notin[0,T]\), if \(\xi\) is a generalized random process.

We shall call a pair \((\xi,\eta)\) of random processes \(\xi=\{\xi(\cdot)\}\) and \(\eta=\{\eta(\cdot)\}\) information stable if either

\[ \bar I(\xi,\eta)=\lim_{T\to\infty}\frac{1}{T} I(\xi_0^T,\eta_0^T)=0, \]

or every sequence of random variables \((\xi_0^{t_1},\eta_0^{t_1}),(\xi_0^{t_2},\eta_0^{t_2}),\ldots\), \(\lim_{n\to\infty} t_n=\infty\), is information stable.

Theorem 2. A pair of random processes \(\xi\) and \(\eta\) forming a Gaussian random process \((\xi,\eta)\) is always information stable.

Let now \(\xi\) and \(\eta\) be one-dimensional random processes, stationary in the broad sense. Introduce the notation

\[ |r_{\xi\eta}(\lambda)|^2= \frac{|f_{\xi\eta}(\lambda)|^2}{f_{\xi\xi}(\lambda)f_{\eta\eta}(\lambda)}, \tag{5} \]

where \(f_{\xi\xi}(\lambda)\), \(f_{\eta\eta}(\lambda)\), and \(f_{\xi\eta}(\lambda)\) are, respectively, the derivatives of the spectral and cross-spectral functions of the processes \(\xi\) and \(\eta\);

\[ Q_{\xi\eta}=\frac{1}{\pi}\int |r_{\xi\eta}(\lambda)|^2\,d\lambda, \tag{6} \]

where the integral is taken over the limits from \(0\) to \(\pi\) if \(\xi\) and \(\eta\) are random processes of a discrete argument, and over the limits from \(0\) to \(\infty\) if \(\xi\) and \(\eta\) are random processes of a continuous argument or generalized random processes.

Theorem 3. Let \(\xi\) and \(\eta\) be one-dimensional random processes forming a two-dimensional Gaussian random process.

Then

\[ \lim_{T\to\infty}\frac{1}{T}Di(\xi_0^T,\eta_0^T) = \lim_{T\to\infty}\frac{1}{T}M\bigl|i(\xi_0^T,\eta_0^T)-I(\xi_0^T,\eta_0^T)\bigr|^2 = Q_{\xi\eta} \tag{7} \]

in the following cases:

1) \(\xi\) and \(\eta\) are random processes of a discrete argument, one of which is nonsingular;

2) \(\xi\) and \(\eta\) are random processes of a continuous argument or generalized random processes, one of which has rational spectral densities.

From Theorems 1 and 3 there follows the corollary:

Corollary. If \(Q_{\xi\eta}>0\), then under the conditions of Theorem 3

\[ \lim_{T\to\infty} P\left\{ a< \frac{i(\xi_0^T,\eta_0^T)-I(\xi_0^T,\eta_0^T)} {TQ_{\xi\eta}} <b \right\} = \frac{1}{\sqrt{2\pi}}\int_a^b e^{-x^2/2}\,dx. \tag{8} \]

In writing down the formula generalizing (9) to multidimensional random processes, we shall use the result of the following lemma, which is also of independent interest.

Lemma. Let \(\xi=(\xi_1,\ldots,\xi_n)\), \(\eta=(\eta_1,\ldots,\eta_m)\) be, respectively, \(n\)- and \(m\)-dimensional random processes forming an \((n+m)\)-dimensional Gaussian stationary random process \((\xi,\eta)\).

Then there exist, respectively, \(n\)- and \(m\)-dimensional Gaussian random processes \(\xi'=(\xi'_1,\ldots,\xi'_n)\) and \(\eta'=(\eta'_1,\ldots,\eta'_m)\) such that:

1) the random processes \(\xi'_1,\ldots,\xi'_n,\eta'_1,\ldots,\eta'_m\), with the exception of the pairs \((\xi'_j,\eta'_j)\), \(j=1,\ldots,k\leqslant \min(n,m)\), are mutually independent;

2) the random processes \(\xi\) and \(\xi'\), \(\eta\) and \(\eta'\) are stationarily related and subordinate to one another.

Put now

\[ Q_{\xi\eta}=Q_{\xi'\eta'}=\sum_{j=1}^{n} Q_{\xi'_j\eta'_j} =\frac{1}{\pi}\sum_{j=1}^{k}\int \left|r_{\xi'_j\eta'_j}(\lambda)\right|^2\,d\lambda, \tag{9} \]

where the limits of integration are determined in the same way as in formula (6).

Theorem 4. Let \(\xi=(\xi_1,\ldots,\xi_n)\) and \(\eta=(\eta_1,\ldots,\eta_m)\) be \(n\)- and \(m\)-dimensional random processes forming a Gaussian \((n+m)\)-dimensional stationary random process \((\xi,\eta)\).

Then in the following cases formula (7) holds:

1) \(\xi\) and \(\eta\) are random processes of a discrete argument, and \(\xi\) has regularity rank equal to \(n\);

2) \(\xi\) and \(\eta\) are random processes of a continuous argument or generalized random processes, one of which has rational spectral densities.

From Theorems 1 and 4 the following corollary follows.

Corollary. If \(Q_{\xi\eta}>0\), then under the conditions of Theorem 4 formula (8) holds.

Remark. It can be shown that always

\[ \lim_{T\to\infty}\frac{1}{T}Di\bigl(\xi_0^T,\eta_0^T\bigr)\geqslant Q_{\xi\eta}. \tag{10} \]

Laboratory of Information Transmission Systems
Academy of Sciences of the USSR

Received
7 III 1960

CITED LITERATURE

¹ C. E. Shannon, W. Weaver, University of Illinois Press, Urbana Ill, 1949 (incomplete Russian trans. in the collection Theory of the Transmission of Electrical Signals in the Presence of Noise, IIL, 1953).
² B. MacMillan, Ann. Math. Statist., 24, No. 2, 196 (1953).
³ A. Ya. Khinchin, UMN, 11, no. 1 (67), 17 (1956).
⁴ R. L. Dobrushin, UMN, 14, no. 6 (90), 3 (1959).