UDC 519.3 + 519.212.3

MATHEMATICS

D. B. YUDIN

ON THE EXISTENCE OF A SOLUTION OF GENERALIZED PROBLEMS OF SMOOTHING AND EXTRAPOLATION OF RANDOM FUNCTIONS

(Presented by Academician A. A. Dorodnitsyn, 4 IV 1967)

The problem of interpolation, extrapolation, and smoothing of random sequences and random functions, important for various applications, was posed and solved in the works of A. N. Kolmogorov and others \((^{1-4})\). The problem has been comprehensively studied for the case when the quality index of the process is the variance of the interpolation (extrapolation, smoothing) errors. The present note is devoted to an analysis of the conditions for the existence of a solution of the smoothing and extrapolation problem for more complicated quality indices of these processes.

1. Consider, on a probability space \((\Omega, F, P)\), a random function \(\xi(t)\) with first moment \(\bar{\xi}(t)\) and correlation function \(k(t_1,t_2)=k_{\xi,\xi}(t_1,t_2)\). Associate with \(k(t_1,t_2)\) the set of functions \(\mathscr{P}(\tau)\) on \([0,T]\) for which

\[ \int_0^T\int_0^T k(t_0-\tau_1,t_0-\tau_2)\mathscr{P}(\tau_1)\mathscr{P}(\tau_2)\,d\tau_1\,d\tau_2<\infty, \tag{1} \]

where \(t_0\) and \(T>0\) are fixed numbers.

If \(\xi(t)\) is a purely random process (“white noise”), then \(k(t_1,t_2)=\delta(t_1-t_2)\), and the set of functions \(\mathscr{P}(\tau)\) is the space \(L_2\) of square-integrable functions on the interval \([0,T]\). If \(k(t_1,t_2)\) is a continuous symmetric positive definite function, then the corresponding eigenvalues \(\lambda_i>0\) and eigenfunctions \(f_i(t)\) generate the set of functions \(\mathscr{P}(\tau)=\sum_{i=1}^{\infty}c_i f_i(\tau)\), where the \(c_i\) satisfy the condition \(\sum_{i=1}^{\infty}\lambda_i c_i^2<\infty\). This set of functions coincides with the set of functions satisfying condition (1), and is a Hilbert space with scalar product

\[ (x^{(1)},x^{(2)})= \int_0^T\int_0^T k(t_0-\tau_1,t_0-\tau_2)x^{(1)}(\tau_1)x^{(2)}(\tau_2)\,d\tau_1\,d\tau_2. \]

We shall call this space the space \(H^{(k)}(t_0,T)\).

Consider, on the probability space \((\Omega,F,P)\), the Hilbert space \(\hat{H}\) of random variables \(\zeta\) with bounded mathematical expectation and bounded variance \((M\zeta<\infty,\ M\zeta^2<\infty)\), with scalar product \((\zeta^{(1)},\zeta^{(2)})=M(\zeta^{(1)}\zeta^{(2)})\). Denote by \(L(t_i,T_i)\), \(i=1,2,\ldots,m\) (\(t_i\) and \(T_i>0\) are fixed numbers), the subset of elements of \(\hat{H}\) generated by the random function \(\xi(t)\) for \(t_i-T_i\le t\le t_i\). A random variable \(\zeta\in L(t_i,T_i)\) if

\[ \zeta=\int_0^{T_i}\xi(t_i-\tau)\mathscr{P}(t_i,\tau)\,d\tau. \tag{2} \]

If \(\mathscr P(t_i,\tau) \in H^{(k)}(t_i,T_i)\), then \(L(t_i,T_i)\) is a subspace of the space \(H\). For any random variable \(\xi \in L(t_i,T_i)\) there exists a unique function \(\mathscr P(t_i,\tau) \in H^{(k)}(t_i,T_i)\) for which formula (2) is valid.

This assertion is easily extended to the multidimensional case. In this case, instead of a random function \(\xi(t)\), one considers a random vector function \(\xi(t)=\{\xi^\alpha(t)\}\), \(\alpha=1,2,\ldots,n\), characterized by the correlation matrix \(\|k_{\xi^\alpha,\xi^\beta}(t_1,t_2)\|\). Instead of the Hilbert space \(H\) of random variables \(\zeta\), one considers the Hilbert space \(H^n\) of random vectors \(\zeta=\{\zeta^\alpha\}\). The space \(H^n\) is the direct product of the spaces \(H\) of random variables \(\zeta^\alpha\), the components of the vector \(\zeta\). Instead of weight functions \(\mathscr P(\tau)\), one considers matrices of weight functions \(\|\mathscr P_{\alpha\beta}(\tau)\|\). With the random vector function \(\xi(t)\) there is associated the Hilbert space \(H^{(K)}(t_0,T)\) of functions \(\mathscr P_{\alpha\beta}(\tau)\) with scalar product

\[ (x_{\alpha\beta}^{(1)},x_{\alpha\beta}^{(2)})= \sum_{\alpha=1}^{n}\sum_{\mu=1}^{n}\sum_{\nu=1}^{n}\int_{0}^{T}\int_{0}^{T} k_{\xi^\mu,\xi^\nu}(t_0-\tau_1,t_0-\tau_2) x_{\alpha\mu}^{(1)}(\tau_1)x_{\alpha\nu}^{(2)}(\tau_2)\,d\tau_1\,d\tau_2, \]

and formula (2) is replaced by a relation of the form

\[ \xi^\alpha= \sum_{\beta=1}^{n}\int_{0}^{T} \xi^\beta(t_i-\tau)\mathscr P_{\alpha\beta}(t_i,\tau)\,d\tau, \qquad \alpha=1,2,\ldots,n. \tag{3} \]

Let us denote by \(H_0\) the Hilbert space of random variables with zero mathematical expectation and bounded variance. In \(H_0\) the scalar product of the variables \(\xi_0^{(1)}\) and \(\xi_0^{(2)}\) is equal to their correlation moment, and the norm \(\|\xi_0\|\) coincides with the root-mean-square value of the random variable \(\xi_0\). In analogous fashion one introduces and interprets the Hilbert space \(H_0^n\) of random \(n\)-dimensional vectors with zero mathematical expectation and bounded variance. We single out in \(H_0^n\) the subspace \(L_0^n(t_i,T_i)\), whose elements \(\xi_0=\{\xi_0^\alpha\}\) are random vectors determined by the random vector function \(\xi_0(t)=\{\xi_0^\alpha(t)\}\) \((\mathrm M\xi_0^\alpha(t)=0;\ \mathrm M[\xi_0^\alpha(t)]^2<\infty)\),

\[ \xi_0^\alpha= \sum_{\beta=1}^{n}\int_{0}^{T_i} \xi_0^\beta(t_i-\tau)\mathscr P_{\alpha\beta}(t_i,\tau)\,d\tau, \]

where \(\mathscr P_{\alpha\beta}(t_i,\tau)\in H^{(K)}(t_i,T_i)\).

The set \(L^n(t_i,T_i)\) of random vectors \(\xi=\xi_0+\bar\xi\), where \(\xi_0\in L_0^n(t_i,T_i)\), is a subspace of the Hilbert space \(H^n\) of random vectors \(\xi\).

We also introduce the Hilbert space \(H_r^n\) of elements \(h=\{\xi_0,\gamma_1,\ldots,\gamma_r\}\). Here \(\xi_0=\{\xi_0^\alpha\}\), \(\alpha=1,2,\ldots,n\), is a random vector with zero mathematical expectation and bounded variance \((\xi_0\in H_0^n)\); \(\gamma_j=\{\gamma_j^\alpha\}\), \(\alpha=1,2,\ldots,n\), for \(j=1,\ldots,r\), is a nonrandom vector whose components are real numbers \((\gamma_j\in E^n)\). The scalar product in \(H_r^n\) is defined by the relation

\[ (h^{(1)},h^{(2)})= \mathrm M(\xi_0^{(1)}\xi_0^{(2)})+ \sum_{j=1}^{r}(\gamma_j^{(1)},\gamma_j^{(2)}) = \mathrm M(\xi_0^{(1)}\xi_0^{(2)})+ \sum_{j=1}^{r}\sum_{\alpha=1}^{n}\gamma_j^{\alpha(1)}\gamma_j^{\alpha(2)}. \]

The space \(H_r^n\) is the Cartesian product of \(H_0^n\) and \(r\) Euclidean spaces \(E^n\).

p. 2. Consider a family of random vector functions with components of the form

\[ \eta^\alpha(t)=\eta_0^\alpha(t)+\sum_{j=1}^{r}z_j^\alpha\varphi_j^\alpha(t), \qquad \alpha=1,2,\ldots,n, \tag{4} \]

where \(\eta_0^\alpha(t)\) are given random functions with zero mathematical expectations and bounded variances, \(\varphi_j^\alpha(t)\), \(j=1,\ldots,r\), are fixed nonrandom functions, and \(z_j^\alpha\) are arbitrary parameters.

The components of the vector functions \(\eta(t)\) must, at the point \(t_0+t_y\) (\(t_y\) is the lead time), be approximated by linear combinations of the components \(\xi^\alpha(t)\) \((t_0-T \le t \le t_0)\) of the form

\[ \xi^\alpha(t)=\xi_0^\alpha(t)+\sum_{j=1}^{r} z_j^\alpha \psi_j^\alpha(t),\qquad \alpha=1,2,\ldots,n. \]

Here \(\xi_0^\alpha(t)\) are fixed random functions with zero mathematical expectations and bounded variances, \(\psi_j^\alpha(t)\), \(j=1,2,\ldots,r\), are nonrandom functions specified for each \(\alpha\) and linearly independent on \([t_0-T,t_0]\), and \(z_j^\alpha\) are arbitrary parameters, the same as in (4). The cross-correlation functions \(k_{\xi^\alpha,\xi^\beta}(t_1,t_2)\) of the components \(\xi^\alpha(t)\) and \(\xi^\beta(t)\) are assumed to be given.

Let us consider subspaces \(L_r^n(t_i,T_i)\), \(i=1,2,\ldots,m\), of the space \(H_r^n\). The subspaces \(L_r^n(t_i,T_i)\) are generated by the random vector function \(\xi(t)=\{\xi^\alpha(t)\}\) for \(t_i-T_i \le t \le t_i\). An element \(h=\{\zeta_0,\gamma_1,\ldots,\gamma_r\}\) of the space \(H_r^n\) belongs to \(L_r^n(t_i,T_i)\) if

\[ \zeta_0^\alpha=\sum_{\beta=1}^{n}\int_{0}^{T_i} \xi_0^\beta(t_i-\tau)\mathcal{P}_{\alpha\beta}(t_i,\tau)\,dt, \qquad \alpha=1,2,\ldots,n, \]

\[ \gamma_j^\alpha=\sum_{\beta=1}^{n}\int_{0}^{T} \psi_j^\beta(t_i-\tau)\mathcal{P}_{\alpha\beta}(t_i,\tau)\,d\tau, \qquad \alpha=1,2,\ldots,n;\quad j=1,2,\ldots,r, \]

where \(\mathcal{P}_{\alpha\beta}(t_i,\tau)\in H^{(K)}(t_i,T_i)\), \(\alpha,\beta=1,2,\ldots,n\).

In the subspace \(L_r^n(t_0,T)\) let us single out the set \(G^\alpha\) of elements of the form \(h^\alpha=\{\zeta_0^\alpha,\varphi_1^\alpha(t_0+t_y),\ldots,\varphi_r^\alpha(t_0+t_y)\}\). The random variables \(\zeta_0^\alpha\) belong to the subspace \(L_0^n(t_0,T)\) of the Hilbert space \(H_0^n\). The first moments of the random variables determined by the elements \(h^\alpha\in G^\alpha\) coincide with the first moments of the components \(\eta^\alpha(t)\) at the point \(t_0+t_y\) for any values of the parameters \(z_j^\alpha\).

1. The concepts introduced in paragraphs 1 and 2 make it possible to generalize and study problems of smoothing and extrapolation of random functions.

The extrapolation errors
\[ x^\alpha(t_0)=\eta^\alpha(t_0+t_y)-\xi^\alpha(t_0)\equiv \eta^\alpha-\xi^\alpha,\qquad \alpha=1,2,\ldots,n. \]
The first and second moments of the distribution of the forecast errors are respectively
\(m^\alpha=M(\eta^\alpha-\xi^\alpha)\);
\(k^{\alpha\beta}=M\{(x^\alpha-m^\alpha)(x^\beta-m^\beta)\}\).

We shall consider two kinds of quality criteria for solving smoothing and extrapolation problems: \(R(k^{\alpha\beta})\) and \(R(m^\alpha,k^{\alpha\beta})\), depending on whether it is required to ensure elimination of the first moments of the forecast errors. The quality criterion for the solution of the generalized problem I of smoothing and extrapolation is

\[ R(k^{\alpha\beta})=\bar{R}(h^\alpha)=\bar{R}(\zeta^\alpha)=\hat{R}(\mathcal{P}_{\alpha\beta}). \tag{5} \]

We shall give three formulations of problem I in terms of the Hilbert spaces \(H_r^n\), \(H^m\), and \(H^{(K)}\).

Problem I. \(1^\circ\). It is required to choose a system of elements \(h^\alpha\in G^\alpha\), \(\alpha=1,2,\ldots,n\), on which the forecast quality criterion \(R(k^{\alpha\beta})=\bar{R}(h^\alpha)\) of the random vector function \(\eta(t)=\{\eta^\alpha(t)\}\) at the point \(t_0+t_y\) attains its upper bound.

\(2^\circ\). It is required to choose a random vector \(\xi_R=\{\xi_R^\alpha\}\in L^n(t_0,T)\) on which the forecast quality criterion \(R(k^{\alpha\beta})=\bar{R}(\zeta^\alpha)\) of the random vector function \(\eta(t)=\{\eta^\alpha(t)\}\) at the point \(t_0+t_y\) attains its upper bound under the conditions

\[ M\{\eta^\alpha(t_0+t_y)-\xi^\alpha\}=0 \quad \text{for any } z_j^\alpha,\quad \alpha=1,2,\ldots,n;\quad j=1,2,\ldots,r. \tag{6} \]

3°. It is required to choose a system of weight functions \(\{\mathcal P_{\alpha\beta}^{R}(t_0,\tau)\}\in H^{(K)}(t_0,T)\) on which the forecast-quality index \(R(k^{\alpha\beta})=\bar R(\mathcal P_{\alpha\beta})\) of the random vector-function \(\eta(t)=\{\eta^\alpha(t)\}\) attains its supremum at the point \(t_0+t_y\) under the conditions

\[ \varphi_j^\alpha(t_0+t_y)= \sum_{\beta=1}^{n}\int_{0}^{T}\psi_j^\beta(t_0-\tau)\mathcal P_{\alpha\beta}(t_0,\tau)\,d\tau, \qquad \alpha,\beta=1,\ldots,n;\ j=1,\ldots,r . \tag{7} \]

By virtue of the assumed linear independence of the functions \(\psi_j^\alpha(t)\), \(j=1,2,\ldots,r\), on \([t_0-T,t_0]\) for each \(\alpha=1,2,\ldots,n\), there exist points \(t_1^\alpha,\ldots,t_r^\alpha\subset[t_0-T,t_0]\) such that \(\|\psi_i^\alpha(t_k^\alpha)\|\ne0\). Therefore the sets \(G^\alpha\) are nonempty. For the same reason, the systems (6) and (7) have solutions.

We restrict the choice of quality criteria \(R(k^{\alpha\beta})\) for the solution of problem I by the following conditions:

(a) \(R(k^{\alpha\beta})\) is a continuous function of its arguments.

(b) Let the sequences \(\{\xi_s^\alpha\}\), \(\alpha=1,2,\ldots,n\), be such that

\[ \sum_{\alpha=1}^{n} M|\xi_s^\alpha|^2 \to \infty \quad \text{as } s\to\infty . \]

Then, for all sufficiently large \(s\),

\[ R_s=\bar R(k_s^{\alpha\beta})< \sup_{\xi\in L^n(t_0,T)} \widetilde R(\xi^\alpha). \]

Theorem 1. Problem I, under conditions (a) and (b), has a solution.

p. 4. In many problems of smoothing and extrapolation of random functions there is no need to reduce to zero the first moments of the forecast errors. In such problems the quality index of the solution is a function of the first and second moments of the extrapolation errors

\[ R(m^\alpha,k^{\alpha\beta})=\bar R(h^\alpha)=\widetilde R(\xi^\alpha)=\hat R(\mathcal P_{\alpha\beta}). \tag{8} \]

We subject the quality index \(R(m^\alpha,k^{\alpha\beta})\) to conditions analogous to conditions (a) and (b) of the preceding paragraph. Consider the subspace \(L=L^n(t_0,T)\) of the Hilbert space \(H^n\) of random vectors \(\xi=\{\xi^\alpha\}\) and formulate the following smoothing and extrapolation problem.

Problem II. It is required to choose a system of random variables \(\xi^\alpha\in L\), \(\alpha=1,2,\ldots,n\) (or, equivalently, a system of weight functions \(\mathcal P_{\alpha\beta}(\tau)\in H^{(K)}(t_0,T)\), \(\alpha,\beta=1,2,\ldots,n\)), on which the quality index (8) attains its supremum.

Theorem 2. Under conditions naturally adapting requirements (a) and (b) of p. 3 to the quality index (8), problem II has a solution.

The concepts introduced in pp. 1 and 2 are used to prove Theorems 1 and 2 and to construct effective schemes for smoothing and extrapolating random functions in accordance with the quality indices (5) and (8).

A qualitative investigation of the solution of optimization problems of smoothing and forecasting in the case of a number of subspaces \(L_r^n(t_i,T_i)\) (or \(L^n(t_i,T_i)\)) reduces to the analysis of the mutual arrangement of these subspaces. In particular, the assertions of the present article carry over directly to the case of orthogonal subspaces \(L_r^n(t_i,T_i)\) (or \(L^n(t_i,T_i)\)), i.e., to the case when the extrapolation errors determined by the behavior of the random vector-functions \(\eta(t)\) and \(\xi(t)\) on different intervals \((t_i-T_i,t_i)\), \(i=1,\ldots,m\), can be correlated only through factors not connected with the choice of the weight functions \(\mathcal P_{\alpha\beta}(t_i,\tau)\).

Received
30 III 1967

REFERENCES

1. A. N. Kolmogorov, Bull. Moscow State Univ., 2, issue 6 (1941).
2. A. N. Kolmogorov, Izv. Acad. Sci. USSR, Mathematics Series, 5, No. 1 (1941).
3. N. Wiener. Extrapolation, Interpolation and Smoothing of Stationary Time Series, N. Y., 1949.
4. L. A. Zadeh, J. R. Ragazzini, J. Appl. Phys., 21, No. 7 (1950).