MATHEMATICS

M. S. PINSKER

EXTRAPOLATION OF HOMOGENEOUS RANDOM FIELDS AND THE AMOUNT OF INFORMATION ABOUT A GAUSSIAN RANDOM FIELD CONTAINED IN ANOTHER GAUSSIAN RANDOM FIELD

(Presented by Academician A. N. Kolmogorov, 20 IX 1956)

Let \(\xi(\mathbf t)=\xi(t_1,\ldots,t_n)\) be an \(n\)-dimensional homogeneous random field \((^1)\). In the case when the arguments \(t_1,\ldots,t_n\) run through all possible integer values, we shall call \(\xi(\mathbf t)\) a field with a discrete argument and denote it by \(\xi_{\mathbf m}=\xi_{m_1,\ldots,m_n}\).

If, however, \(t_1,\ldots,t_n\) run through all possible real values, then \(\xi(\mathbf t)\) will be called a field with a continuous argument. For \(n=1\) the concept of a homogeneous random field coincides with the concept of a stationary random process (sequence).

Introduce the notation:
\[ \sigma_k^2=\lim_{l\to\infty}\inf_{a_{m_1,m_2,\ldots,m_n}} \mathbf M\left|\xi_{0,0,\ldots,0}- \right. \]
\[ \left. -\sum_{m_1=k}^{l}\sum_{m_2,\ldots,m_n=-l}^{l} a_{m_1,m_2,\ldots,m_n}\xi_{m_1,m_2,\ldots,m_n}\right|^2, \tag{1} \]
\[ \sigma_{(R)}^2=\lim_{k\to\infty}\sigma_k^2, \tag{2} \]
\[ \xi^{(1)}_{s_1,m_2,\ldots,m_n} = \xi_{s_1,m_2,\ldots,m_n} - P_{H_{s_1}}\xi_{s_1,m_2,\ldots,m_n}, \]
where \(H_{s_1}\) is the closed linear hull of the random variables
\(\xi_{m_1,m_2,\ldots,m_n}\), \(m_1>s_1\); \(-\infty<m_2,\ldots,m_n<\infty\), and \(P_{H_{s_1}}\) is the projection operator of the space \(H_{s_1}\). According to \((^4)\), a homogeneous random field \(\xi_{\mathbf m}=\xi_{m_1,\ldots,m_n}\) is called regular, nonsingular, singular, respectively, if
\(\sigma_{(R)}^2=\mathbf M|\xi_{\mathbf m}|^2\);
\(\sigma_{(R)}^2\ne0\);
\(\sigma_{(R)}^2=0\).

For \(n=1\) these definitions turn into the definitions given by A. N. Kolmogorov \((^2)\).

The concepts of regularity and singularity adopted in \((^{1,2})\) naturally presuppose that one of the directions in the space of the arguments of the field is distinguished (plays the role of time).

For the purposes of the present note it is convenient, for \(n>1\), to adopt other definitions of these concepts, which, as will be seen below, are not connected with singling out one of the axes. Namely, taking for \(n=1\) the usual definition of regularity, nonsingularity, and singularity, we agree, by induction, to call a homogeneous random field \(\xi_{\mathbf m}=\xi_{m_1,\ldots,m_n}\) regular (respectively nonsingular*) if it is regular (nonsingular) in the direction \(m_1\) and \(\xi^{(1)}_{1,m_2,\ldots,m_n}\) is a regular (nonsingular) \((n-1)\)-dimensional homogeneous random field; further, \(\xi_{\mathbf m}\) is a singular homogeneous random field if \(\xi_{\mathbf m}\) is singular in the direction \(m_1\) or \(\xi^{(1)}_{1,m_2,\ldots,m_n}\) is a singular \((n-1)\)-dimensional homogeneous random field.

The following theorems are a generalization of results of A. N. Kolmogorov \((^2)\).

* Regularity, singularity, and nonsingularity, as defined in \((^4)\), should naturally be called regularity, singularity, and nonsingularity in the direction \(m_1\).

Theorem 1. Every nonsingular homogeneous random field
\(\xi_{\mathbf m}=\xi_{m_1,\ldots,m_n}\) can be decomposed into an uncorrelated sum

\[ \xi_{\mathbf m}=\xi_{\mathbf m}^{(R)}+\xi_{\mathbf m}^{(S)}, \tag{3} \]

where \(\xi_{\mathbf m}^{(R)},\xi_{\mathbf m}^{S}\) are regular and singular homogeneous random fields, homogeneously correlated with \(\xi_{\mathbf m}\),

\[ M\bigl(\xi_{\mathbf m'}^{(R)}\cdot \xi_{\mathbf m'}^{(S)}\bigr)=0, \qquad \xi_{\mathbf m}^{(R)}\in H_{\mathbf m}=H_{m_1,m_2,\ldots,m_n}, \tag{4} \]

\(H_{m_1,m_2,\ldots,m_n}\) is the closed linear span of the vectors

\[ \begin{gathered} \xi_{l_1,l_2,\ldots,l_n};\quad l_1>m_1;\quad -\infty<l_2,\ldots,l_n<\infty,\\ \xi_{m_1,l_2,l_3,\ldots,l_n};\quad l_2>m_2;\quad -\infty<l_3,\ldots,l_n<\infty,\\ \cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\\ \xi_{m_1,\ldots,m_{n-1},l_n};\quad l_n>m_n. \end{gathered} \]

Theorem 2. A regular homogeneous random field \(\xi_{\mathbf m}\) admits the representation

\[ \xi_{\mathbf m}=\xi_{m_1,\ldots,m_n} = \sum_{l_1=1}^{\infty} \sum_{l_2,\ldots,l_n=-\infty}^{\infty} a_{l_1,l_2,\ldots,l_n}\, \eta_{l_1+m_1,l_2+m_2,\ldots,l_n+m_n} + \tag{5} \]

\[ + \sum_{l_2=1}^{\infty} \sum_{l_3,\ldots,l_n=-\infty}^{\infty} a_{0,l_2,\ldots,l_n}\, \eta_{m_1,l_2+m_2,\ldots,l_n+m_n} +\ldots+ \sum_{l_n=0}^{\infty} a_{0,0,\ldots,l_n}\, \eta_{m_1,m_2,\ldots,l_n+m_n}, \]

where \(\eta_{\mathbf m}\) is a homogeneous random field, homogeneously correlated with \(\xi_{\mathbf m}\),

\[ M(\eta_{\mathbf m}\cdot \overline{\eta}_{\mathbf m})=1, \tag{6} \]

\[ M(\eta_{\mathbf m}\cdot \overline{\eta}_{\mathbf m'})=0,\qquad \mathbf m\ne \mathbf m', \tag{7} \]

\[ \eta_{\mathbf m}\in H_{\mathbf m}=H_{m_1,\ldots,m_n}; \qquad H_{m_1,\ldots,m_n}\ \text{is defined above.} \]

For any two representations of the form considered,

\[ \eta'_{\mathbf m}=\alpha\eta_{\mathbf m}; \qquad \alpha=\mathrm{const},\quad |\alpha|=1,\quad a'_{l_1,\ldots,l_n}=\overline{\alpha}\,a_{l_1,\ldots,l_n}. \]

Theorem 3. In order that a homogeneous random field \(\xi_{\mathbf m}\) be regular, it is necessary and sufficient that the following conditions hold.

1) The spectral function \(F(\lambda_1,\ldots,\lambda_n)\) is absolutely continuous and the spectral density
\(f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)=F_{\lambda_1,\ldots,\lambda_n}^{(n)}(\lambda_1,\ldots,\lambda_n)\) is nonzero almost everywhere,

2)

\[ \int_{-\pi}^{\pi}\cdots\int_{-\pi}^{\pi} \left|\log f(\lambda_1,\ldots,\lambda_n)\right| \,d\lambda_1\cdots d\lambda_n<\infty. \tag{8} \]

Theorem 4. In order that a homogeneous random field \(\xi_{\mathbf m}\) be nonsingular, it is necessary and sufficient that the following conditions hold:

1) The derivative
\(f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)=F_{\lambda_1,\ldots,\lambda_n}^{(n)}(\lambda_1,\ldots,\lambda_n)\) of the spectral function \(F(\lambda_1,\ldots,\lambda_n)\) is nonzero almost everywhere,

2)

\[ \int_{-\pi}^{\pi}\cdots\int_{-\pi}^{\pi} \left|\log f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\right| \,d\lambda_1\cdots d\lambda_n<\infty. \tag{9} \]

Let

\[ \sigma_{\xi\xi}^{2}=|a_{0,0,\ldots,0}|^{2} = \lim_{l\to\infty}\inf_{a_{m_1,m_2,\ldots,m_n}} M\left|\xi_{0,0,\ldots,0}-\right. \]

\[ \begin{gathered} -\sum_{m_1=1}^{l}\sum_{m_2,\ldots,m_n=-l}^{l} a_{m_1,m_2,\ldots,m_n}\xi_{m_1,m_2,\ldots,m_n}\\ -\sum_{m_2=1}^{l}\sum_{m_3,\ldots,m_n=-l}^{l} a_{0,m_2,\ldots,m_n}\xi_{0,m_2,m_3,\ldots,m_n}-\cdots\\ \cdots-\sum_{m_n=1}^{l} a_{0,\ldots,m_n}\xi_{0,\ldots,m}\big|^2 . \end{gathered} \tag{10} \]

Then the following theorem holds:

Theorem 5. If \(\xi_{\mathbf m}\) is a regular or nonsingular homogeneous random field, then

\[ \sigma_{\xi\xi}^{2}=(2\pi)^n\exp\left\{ \frac{1}{(2\pi)^n}\int_{-\pi}^{\pi}\cdots\int_{-\pi}^{\pi} \log f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\, d\lambda_1,\ldots,d\lambda_n \right\}. \tag{11} \]

It is clear that Theorems 3 and 4 follow from Theorem 5, since in the case of a regular or nonsingular field \(\sigma_{\xi\xi}^2\ne0\).

Let us dwell on the proof of the last theorem. From the results of note \((^4)\) it follows that

\[ f_{\xi^{(1)}\xi^{(1)}}(\lambda_2,\ldots,\lambda_n) = 2\pi\exp\left\{ \frac{1}{2\pi}\int_{-\pi}^{\pi} \log f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\,d\lambda_1 \right\} \]

is the mixed derivative of order \((n-1)\) of the spectral function of the field
\(\xi^{(1)}_{1,m_2,\ldots,m_n}\).

Similarly, passing from the random field \(\xi^{(1)}_{1,m_2,\ldots,m_n}\) to \((n-2)\)-, \((n-3)\)-, …, 1-dimensional random fields of arguments \(m_3,\ldots,m_n;\ m_4,\ldots,m_n;\ \ldots;\ m_n\), we obtain for the spectral functions and for \(\sigma_{\xi\xi}^{2}\) the expressions:

\[ 4\pi^2\exp\left\{ \frac{1}{4\pi^2}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi} \log f_{\xi\xi}(\lambda_1,\lambda_2,\ldots,\lambda_n)\, d\lambda_1\,d\lambda_2 \right\}, \]

\[ 8\pi^3\exp\left\{ \frac{1}{8\pi^3}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi}\int_{-\pi}^{\pi} \log f_{\xi\xi}(\lambda_1,\lambda_2,\lambda_3,\ldots,\lambda_n)\, d\lambda_1\,d\lambda_2\,d\lambda_3 \right\}, \]

\[ \cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots \]

\[ \sigma_{\xi\xi}^{2}=(2\pi)^n\exp\left\{ \frac{1}{(2\pi)^n}\int_{-\pi}^{\pi}\cdots\int_{-\pi}^{\pi} \log f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\, d\lambda_1\cdots d\lambda_n \right\}. \]

Theorem 5 is proved.

We turn to the consideration of the amount of information contained in one random field relative to another.

Let \(\xi_{\mathbf m}\) and \(\eta_{\mathbf m}\) be random fields of a discrete argument. The amount of information per unit volume contained in one of the random fields relative to the other is, by definition, equal to

\[ I(\{\xi_m\};\{\eta_m\})=\lim_{k\to\infty}\frac{1}{k^n} I(\xi_{k(n)};\eta_{k(n)}), \tag{12} \]

where \(I(\xi_{k(n)};\eta_{k(n)})\) is the amount of information \((^3)\) about the random vector

\[ \eta_{k(n)}=(\eta_{1,1,\ldots,1};\eta_{2,1,\ldots,1};\eta_{n,\ldots,n}), \]

contained in the random vector

\[ \xi_{k(n)}=(\xi_{1,1,\ldots,1};\xi_{2,1,\ldots,1};\cdots;\xi_{n,n,\ldots,n}). \]

For a pair of random fields \(\xi(t),\eta(t)\) of a continuous argument it is natural to introduce into consideration pairs of random fields of a discrete argument

\[ \xi_{\mathbf m}^{\,n}=\xi(\mathbf m h)=\xi(m_1h;\ldots;m_nh), \qquad \eta_{\mathbf m}^{\,n}=\eta(\mathbf m h)=\eta(m_1h;\ldots;m_nh), \]

\(h\) is a positive number. One variant of the definition of the amount of information \(I(\{\xi(t)\};\{\eta(t)\})\) per unit volume about one of the random fields \(\xi(t)\), \(\eta(t)\), contained in the other, is

\[ I(\{\xi(t)\};\{\eta(t)\})=\lim \frac{1}{h^n} I(\{\xi_{\mathbf m}^h\};\{\eta_{\mathbf m}^h\}). \tag{13} \]

On the basis of Theorem 5, just as in note (3), it is easy to obtain the following theorem.

Theorem 6. If \(\xi_{\mathbf m}=(\xi_{m_1},\ldots,\xi_{m_n})\), \(\eta_{\mathbf m}=(\eta_{m_1},\ldots,\eta_{m_n})\) are Gaussian homogeneous and homogeneously correlated random fields of a discrete argument, of which at least one is nonsingular, then the amount of information per unit volume concerning one of these random fields, contained in the other, can be computed by the formula

\[ I(\{\xi_{\mathbf m}\};\{\eta_{\mathbf m}\}) = \tag{14} \]

\[ = \frac{1}{2(2\pi)^n} \int_{-\pi}^{\pi}\cdots\int_{-\pi}^{\pi} \log \frac{ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n) }{ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n) - \left|f_{\xi\eta}(\lambda_1,\ldots,\lambda_n)\right|^2 } \, d\lambda_1 \cdots d\lambda_n, \]

where the integrand is taken to be equal to zero when

\[ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n)=0; \]

\(f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\), \(f_{\eta\eta}(\lambda_1,\ldots,\lambda_n)\), \(f_{\xi\eta}(\lambda_1,\ldots,\lambda_n)\) are the \(n\)-th mixed derivatives with respect to \(\lambda_1,\ldots,\lambda_n\) of the spectral functions and mutual spectral functions of the fields \(\xi_{\mathbf m}\), \(\eta_{\mathbf m}\).

For what follows it is expedient to introduce the following concept: a homogeneous random field \(\xi(t)\) of a continuous argument is quasi-nonsingular if the homogeneous random fields \(\xi_{\mathbf m}^h=\xi(\mathbf m h)\) of a discrete argument are nonsingular.

Theorem 7. If \(\xi(t)=\xi(t_1,\ldots,t_n)\), \(\eta(t)=\eta(t_1,\ldots,t_n)\) are Gaussian homogeneous and homogeneously correlated random fields of a continuous argument, of which at least one is quasi-nonsingular, then the amount of information per unit volume concerning one of these fields, contained in the other, can be computed by the formula

\[ I(\{\xi(t)\};\{\eta(t)\}) = \tag{15} \]

\[ = \frac{1}{2(2\pi)^n} \int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\infty} \log \frac{ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n) }{ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n) - \left|f_{\xi\eta}(\lambda_1,\ldots,\lambda_n)\right|^2 } \, d\lambda_1 \cdots d\lambda_n, \]

where the integrand is taken to be equal to zero when

\[ f_{\xi\xi}(\lambda_1,\ldots,\lambda_n) f_{\eta\eta}(\lambda_1,\ldots,\lambda_n)=0, \]

\(f_{\xi\xi}(\lambda_1,\ldots,\lambda_n)\), \(f_{\eta\eta}(\lambda_1,\ldots,\lambda_n)\), \(f_{\xi\eta}(\lambda_1,\ldots,\lambda_n)\) are the \(n\)-th mixed derivatives with respect to \(\lambda_1,\ldots,\lambda_n\) of the spectral functions and the mutual spectral function of the fields \(\xi(t)\), \(\eta(t)\).

Laboratory for the Development
of Scientific Problems of Wire Communication
Academy of Sciences of the USSR

Received
20 IX 1956

REFERENCES

1. A. M. Yaglom, Uspekhi Mat. Nauk, 7, No. 5 (51), 3 (1952).
2. A. N. Kolmogorov, Bull. Moscow State Univ., 2, No. 6 (1941).
3. M. S. Pinsker, Dokl. Akad. Nauk SSSR, 99, No. 2, 213 (1954).
4. Chiang Tse-pei, Dokl. Akad. Nauk SSSR, 112, No. 2 (1957).