MATHEMATICS

Corresponding Member of the Academy of Sciences of the USSR A. N. TIKHONOV

ON ILL-POSED PROBLEMS OF LINEAR ALGEBRA AND A STABLE METHOD FOR SOLVING THEM

1. Consider the system of linear algebraic equations

\[ Az=\bar u,\qquad A=\{a_{ij}\},\qquad z=\{z_j\},\qquad u=\{u_i\}, \]

\[ i=1,\ldots,m;\quad j=1,\ldots,n. \tag{1} \]

Such a system is not always solvable. Denote
\(\bar U_A=\{u:u=Az,\ z\in R_n\}\subset U=R_m,\)
\(N_A=\{z:Az=0\}\), and let \(Z_A\) be the orthogonal complement to \(N_A\) in the space \(Z=R_n\). Clearly, system (1) is solvable if and only if the right-hand side of the equation \(\bar u\in U_A\). If \(N_A=0\), then the solution of the system is determined uniquely. If, however, \(N_A\ne 0\), then the solution \(z\) of system (1) is not unique: if \(z^{(0)}\) is any one of its solutions, then \(\{z\}=\{z^{0}+z:z\in N_A\}\) represents the complete set of solutions of system (1). The set of conditions

\[ A\bar z=\bar u,\qquad (\bar z\cdot e_k)=0, \]

where \(\{e_k\}\) is a basis of \(N_A\), uniquely determines the element \(z^{(0)}\). We shall call \(z^{(0)}\) the normal solution of system (1). If the system is nonsingular, then the normal solution coincides with the unique solution of this system.

We note for what follows that the normal solution can also be determined from the conditions

\[ Az^{(0)}=\bar u,\qquad \|z^{(0)}\|=\min \|\bar z\|,\quad \text{if } A\bar z=\bar u. \]

The purpose of the present paper is to construct a stable algorithm for determining the normal solution of system (1).

2. Let the initial data of system (1), i.e. the matrix \(A\) and the vector \(\bar u\), be given with some approximation; moreover, we shall measure the error of \(A\), \(u\), and \(z\) by means of the norms

\[ \|u\|=\left(\sum_{i=1}^{m}u_i^2\right)^{1/2},\qquad \|z\|=\left(\sum_{j=1}^{n}z_j^2\right)^{1/2},\qquad \|A\|=\left(\sum (a_{ij}^{2})\right)^{1/2}. \tag{2} \]

It is not difficult to verify that the problem of determining the normal solution of system (1) is ill-posed in the sense of Hadamard.

Consider the example:

\[ (x-y)=u_1,\qquad (y-z)=u_2,\qquad \xi x+\eta y+\zeta z=u_3. \]

The determinant of this system \(A\) and the value of the unknown \(x\) are equal to

\[ \Delta=(\xi+\eta+\zeta),\qquad x=\frac{u_3+\zeta u_2+(\zeta+\eta)u_3}{\xi+\eta+\zeta}=\frac{X}{\Delta}. \]

If the system is singular, then \(\xi+\eta+\zeta=0\), and the vanishing of \(X\) is one of the conditions for solvability. Suppose, however, that \(\xi,\eta,\zeta\) are irrational and, in computations with a certain number of decimal digits, are specified as \(\xi_n,\eta_n,\zeta_n\). We have that \(\Delta_n\) and \(X_n\) are in general different from zero and

the value \(x_n\), as a ratio of small numbers, may take any value. Naturally, attempts to “refine” the value of \(x\) by increasing the number of decimal digits are hopeless.

The situation is analogous for any singular system.

Thus, solutions of systems that are arbitrarily close approximations of singular systems give a large scatter, and the problem of determining normal solutions of singular systems is ill-posed in the sense of Hadamard.

1. If in system (1) \(m > n\), but the system can admit only a unique solution \((N_A = 0)\), then the method of least squares solves this problem, and it leads to a definite value \(\bar z\) for any \(\bar u\), independently of the consistency condition. Let us consider what \(\bar z\) represents if \(\bar u\) does not satisfy the consistency conditions. The set \(u_A\) forms a linear space. Denote by \(U_A\) the projection of \(\bar u\) onto \(U_A\). Obviously, the solution determined by the method of least squares satisfies the equation

\[ A\bar z = \hat u_A . \]

Indeed, the method of least squares consists in the fact that \(\bar z\) is determined as the element realizing the minimum of the quadratic form

\[ N[z,\bar u] = \|Az - \bar u\|^2 . \]

In our case

\[ N[z,\bar u] = \|Az - \hat u_A\|^2 + \|\hat u_A - \bar u\|^2, \]

since \(\hat u_A\) is the orthogonal projection of \(\bar u\) onto \(U_A\), and, obviously, the minimum of this system is realized by the element \(\bar z\) satisfying the equation \(A\bar z = \hat u_A\). However, if the matrix \(A\) is singular \((N_A \ne 0)\), then \(\bar z\) is not determined uniquely.

1. Consider the parametric functional

\[ M^\alpha[z,\tilde A,\tilde u] = \|\tilde A z - \tilde u\|^2 + \alpha \Omega[z], \qquad \Omega[z] = \|z\|^2, \]

where \(\tilde A,\tilde u\) are an arbitrary matrix and vector, and \(\alpha > 0\) is a parameter. It is not hard to see that for any \(\tilde A,\tilde u\) and \(\alpha > 0\) there exists a unique element \(z^\alpha\) realizing the minimum of this functional.

Theorem. Let \(A\) be a matrix; \(\bar u\) a vector satisfying the consistency condition of the equation \(Az=\bar u\); \(\bar z^{(0)}\) the normal solution. Let \(\tilde A,\tilde u\) be some \(\delta\)-approximations of \(A,\bar u\); let \(\varepsilon(\delta), \alpha(\delta)\) be some decreasing functions of \(\delta\), tending to zero as \(\delta \to 0\), and such that \(\delta^2 \leq \varepsilon(\delta)\alpha(\delta)\).

For any \(\varepsilon > 0\), there exists a \(\delta_0 = (\varepsilon,\|\bar z^0\|)\) such that the vector \(z^\alpha\), realizing the minimum of the functional

\[ M^\alpha[z,\tilde A,\tilde u] = \|\tilde A z - \tilde u\|^2 + \alpha \Omega[z], \]

where \(\alpha\) is any number such that

\[ \frac{1}{\varepsilon(\delta)}\delta^2 \leq \alpha \leq \alpha_0(\delta), \tag{\(\alpha^0\)} \]

satisfies the inequality

\[ \|z^\alpha - \bar z^{(0)}\| \leq \varepsilon, \]

provided only that \(\delta \leq \alpha_0(\varepsilon,\|\bar z^{(0)}\|)\).

This theorem holds for both singular and nonsingular matrices.

Denote by \(\hat u_{\tilde A}\) the projection of \(\tilde u\) onto the linear space

\[ U_A = \{u = \tilde A z,\ z \in Z = R_n\}. \]

In this case

\[ \|\tilde u - \hat u_{\tilde A}\| \leq \|\tilde u - \tilde A z\| \qquad (z \in Z = R_n), \]

\[ \|\tilde A z - \tilde u\|^2 \leq \|\tilde A z - \hat u_{\tilde A}\|^2 + \|\hat u_{\tilde A} - \tilde u\|^2. \]

Thus,

\[ M^\alpha[z,\tilde A,\tilde u] = \|\tilde u - \hat u_{\tilde A}\|^2 + M^\alpha[z,\tilde A,\hat u_{\tilde A}], \]

and the functionals on the right- and left-hand sides of this equality have a common element \(z^\alpha\) minimizing them.

We shall use the inequality

\[ \alpha\Omega[z^\alpha]\leq M^\alpha[z^\alpha,\widetilde A,\widehat u_{\widetilde A}] \leq M^\alpha[\bar z^{(0)},\widetilde A,\widehat u_{\widetilde A}] = \|\widetilde A\bar z^{(0)}-\widehat u_{\widetilde A}\|^2 +\alpha\Omega[\bar z^{(0)}], \]

as well as the inequality

\[ \begin{aligned} \|\widetilde A\bar z^{(0)}-\widehat u_{\widetilde A}\| &\leq \|\widetilde A\bar z^{(0)}-A\bar z^{(0)}\| +\|\bar u-\widehat u_{\widetilde A}\| \\ &\leq \delta\|\bar z^{(0)}\|+\|\bar u-\widetilde u\| +\|\widetilde u-\widehat u_{\widetilde A}\| \\ &\leq \delta(\|\bar z^{(0)}\|+1) +\|\widetilde u-\widetilde A[\bar z^{(0)}]\| \\ &\leq \delta(\|\bar z^{(0)}\|+1) +\{\|\widetilde u-\bar u\|+\|A\bar z^{(0)}-\widetilde A\bar z^{(0)}\|\} \\ &\leq C\delta,\qquad C=2(1+\|\bar z^{(0)}\|). \end{aligned} \]

Thus,

\[ \alpha\Omega[z^\alpha]\leq M^\alpha[z^\alpha,\widetilde A,\widehat u_{\widetilde A}] \leq \alpha\left(\frac{C^2\delta^2}{\alpha} +\Omega[\bar z^{(0)}]\right) \leq \alpha\bigl(C^2\varepsilon(\delta)+\Omega[\bar z^{(0)}]\bigr). \]

From the last inequality it follows that

\[ \|z^\alpha\|\leq \|\bar z^{(0)}\|+\varepsilon_1(\delta) \quad (\varepsilon_1(\delta)\to 0,\ \delta\to 0), \]

i.e., \(\{z^\alpha\}\) forms a compact set.

Let us now verify that for any \(\varepsilon>0\) there exists a \(\delta_0(\varepsilon,\|\bar z^{0}\|)\) such that if \(\|\widetilde A-A\|\leq\delta\) and \(\|\widetilde u-\bar u\|\leq\delta\), then \(\|z^\alpha-\bar z^{(0)}\|<\varepsilon\), provided \(\delta<\delta_0\) and \(\alpha\) satisfies condition \((\alpha^0)\).

Suppose that this is false and that there exist \(\varepsilon_0>0\) and \(A_n,\widetilde u_n\) and \(\delta_n\to 0\) such that \(\|z_n^{\alpha_n}-\bar z^{(0)}\|\geq\varepsilon_0\). By the compactness of \(z^\alpha\) we may, without loss of generality, assume that the sequence \(z_n^{\alpha_n}\) converges to some element \(z^{(0)}\).

We shall show that \(z^{(0)}=\bar z^{(0)}\), which will contradict the supposition. Estimate

\[ \begin{aligned} \|Az_n^{\alpha_n}-A\bar z^{(0)}\| &\leq \|Az_n^{\alpha_n}-\widetilde A z_n^{\alpha_n}\| +\|\widetilde A z_n^{\alpha_n}-\widehat u_{\widetilde A}\| +\|\widehat u_{\widetilde A}-A\bar z^{(0)}\| \\ &\leq \delta_n\|z_n^{\alpha_n}\| +\sqrt{M^{\alpha_n}[z_n^{\alpha_n},\widehat u_{\widetilde A},\widetilde A]} +\delta_n(2+\|z^{(0)}\|) \\ &\leq \delta_n\{2(1+\|\bar z^{(0)}\|)+\varepsilon_1(\delta_n)\} +\sqrt{C^2\delta_n^2+\alpha_0(\delta_n)\Omega[\bar z^{(0)}]} =\eta_n\underset{(n\to\infty)}{\longrightarrow}0. \end{aligned} \]

Thus, for \(z^{(0)}=\lim_{n\to\infty} z_n^{\alpha_n}\) we obtain

\[ Az^{(0)}=A\bar z^{(0)},\qquad \|z^{(0)}\|\leq\|\bar z^{(0)}\|, \]

whence it follows that \(z^{(0)}=\bar z^{(0)}\), since these conditions determine a unique element, which proves the theorem.

Remark 1. If the matrix \(A\) is ill-conditioned and in its \(\delta\)-neighborhood there is a singular matrix \(\widetilde A\), where \(\delta\) is the accuracy with which \(A\) is specified, then we are in the conditions of the problem under consideration. Without regularization we may obtain strongly differing solutions, and the use of regularization will give an approximation to the normal solution of the equation \(Az=\bar u\).

Remark 2. Define the generalized normal solution \(z^{(0)}\) by the conditions

\[ A\bar z^{(0)}=\bar u,\qquad \Omega(\bar z^{(0)}-z_0)\leq \Omega(z-z_0) \quad \text{for all } z:\ L(z)=\bar u, \]

where \(z_0\) is an arbitrary fixed element, and \(\Omega\) is a positive definite quadratic form.

Regularization with the functional $\Omega[z] = \|z - z_0\|^2$ proceeds exactly as above and defines the generalized normal solution $\bar z^{(0)}$.

Remark 3. The investigation carried out is not connected with the finite-dimensionality of the spaces $z$ and $u$ and is repeated verbatim for arbitrary continuous linear operators $A[z]$, if $U$ is a Hilbert space and $Z$ is a normed space in which the Hilbert space $\bar Z$ is $s$-compactly embedded (${}^2$). This gives a regularization method for solving linear inhomogeneous equations on the spectrum (${}^1$).

Received
20 IV 1964

REFERENCES CITED

$^{1}$ A. N. Tikhonov, DAN, 151, No. 3 (1963); 153, No. 1 (1963).
$^{2}$ A. N. Tikhonov, DAN, 161, No. 5 (1965).