Full Text
UDC 519.3.35
MATHEMATICS
V. F. KROTOV
SUFFICIENT OPTIMALITY CONDITIONS FOR DISCRETE CONTROLLED SYSTEMS
(Presented by Academician L. S. Pontryagin, March 9, 1966)
1. Let there be sets \(Y\) and \(U\) of arbitrary nature, with elements \(y\) and \(u\), respectively, and an integer sequence \(A=(0,1,2,\ldots,N)\). To each \(i\in A\) there is assigned a subset \(V(i)\) of the direct product \(Y\times U\). We shall consider the set \(D\) of pairs \(y(i), u(i)\) of functions of the integer argument \(i\), defined on \(A\), such that
\[ y(i),\,u(i)\in V(i),\qquad i=0,1,\ldots,N; \tag{1} \]
\[ y(i+1)=f[i,y(i),u(i)],\qquad i=0,1,\ldots,N-1, \tag{2} \]
where the function (operator) \(f[i,y,u]\) is defined on the direct product \(A\times Y\times U\) and maps the latter into the set \(Y\). It is assumed that the set \(D\) is nonempty. The element \(y\) will be called the state of the system, or phase state, and \(u\) the control. The former differs from the latter in that it enters the constraint equation (2) for different values of \(i\).
Define on \(D\) the functional \(I=\sum_{i=0}^{N} f^{0}[i,y(i),u(i)]\), where \(f^{0}(i,y,u)\) is a functional defined on \(A\times Y\times U\). Suppose that the functional \(I\) on \(D\) is bounded below, i.e. \(\inf I=d>-\infty\). It is required to find a sequence \(\{\bar y_s(i), \bar u_s(i)\}\subset D\) minimizing the functional \(I\) on \(D\), i.e. such that \(I[\bar y_s(i),\bar u_s(i)]\to d\) as \(s\to\infty\). In particular, if there exists an element \((\bar y(i),\bar u(i))\in D\) satisfying the equality \(I[\bar y(i),\bar u(i)]=d\) and called an absolute minimum, then the problem reduces to finding the latter.
2. Let us prescribe on the direct product \(A\times Y\) an arbitrary functional \(\varphi(i,y)\) and, with its help, construct the following quantities:
\[ R(i,y,u)=\varphi[i+1,f(i,y,u)]-\varphi(i,y)-f^{0}(i,y,u); \tag{3} \]
\[ \mu(i)=\sup_{y,u\in V(i)} R(i,y,u); \tag{4} \]
\[ \Phi_{0}(y,u)=-\varphi[1,f(0,y,u)]+f^{0}(0,y,u); \]
\[ \Phi_{1}(y,u)=\varphi(N,y)+f^{0}(N,y,u); \tag{5} \]
\[ m_{0}=\inf_{y,u\in V(0)} \Phi_{0}(y,u),\qquad m_{1}=\inf_{y,u\in V(N)} \Phi_{1}(y,u). \tag{6} \]
Theorem. Let there be a sequence \(\{\bar y_s(i),\bar u_s(i)\}\subset D\). In order that this sequence minimize the functional \(I\) on \(D\), it is sufficient (and, if \(f^{0}(i,y,u)\) is bounded on \(V(i)\) for all \(i\in A\), also necessary) that there exist such a functional \(\varphi(i,y)\) that: 1) on \(\{1,2,\ldots,N-1\}\) the function \(\mu(i)\) defined by (4) is determined; 2) for all \(i=1,2,\ldots,N-1\)
\[ R[i,\bar y_s(i),\bar u_s(i)]\to\mu(i),\qquad s\to\infty; \tag{7} \]
3)
\[ \Phi_{0}[\bar y_s(0),\bar u_s(0)]\to m_{0},\qquad \Phi_{1}[\bar y_s(N),\bar u_s(N)]\to m_{1},\qquad s\to\infty. \tag{8} \]
Remark 1. If the sequence appearing in the theorem has the form \(\bar y_s(i)=\bar y(i)\), \(\bar u_s(i)=\bar u(i)\) for all \(s\), then in (7), (8) the convergence requirement is replaced by the equality sign, and the pair \((\bar y(i),\bar u(i))\in D\), satisfying the conditions of the theorem, is an absolute minimum.
For the proof of the theorem the following is needed.
Lemma. Let a functional \(I(v)\), \(v\in D\), bounded from below, be given on a set \(D\), and let there be a set \(E\), containing \(D\), on which a functional \(L\) is defined such that \(L(v)=I(v)\) if \(v\in D\), and there exists a sequence \(\{\bar v_s\}\subset D\), \(L(v_s)\to l=\inf_{v\in E} L(v)\). Then: 1) \(I(\bar v_s)\to d=\inf_D I\); 2) any sequence \(\{v_s\}\subset D\), \(I(v_s)\to d\), satisfies the condition \(L(v_s)\to l\).
Let us prove that \(d=l\). Since \(D\subset E\) and \(L=I\) for \(v\in D\), \(l\le d\). But \(l<d\) is impossible, since \(I(\bar v_s)=L(\bar v_s)\to l\). Consequently, \(d=l\), which also means the validity of the lemma.
Proof of the theorem. Introduce into consideration a set \(E\) satisfying all the conditions of the set \(D\), except for the equalities (2), and define on it the functional
\[ L[y(i),u(i)]=\Phi_0[y(0),u(0)]+\Phi_1[y(N),u(N)]- \sum_{i=1}^{N-1} R[i,y(i),u(i)]. \]
It is easy to see that for \(y(i),u(i)\in D:\ L=I\).
Suppose that there exists such a functional \(\varphi(i,y)\) that, on some sequence \(\{y_s(i),u_s(i)\}\subset D\), the conditions of the theorem are fulfilled. Then this sequence is minimizing for the functional \(L\) on the set \(E\), and, by virtue of the lemma, for the functional \(I\) on \(D\). The first part of the theorem—the sufficiency—has been proved. Necessity will be proved below (item 6), after the description of possible constructions of the functional \(\varphi(i,y)\).
- This theorem generalizes to the discrete case the sufficient optimality conditions formulated in \((^{1a, б, в})\) for continuous processes. Analogously to these conditions, it makes it possible to reduce the problem of the minimum of the functional \(I\) on \(D\) to the aggregate of problems of the maximum of the functional \(R(i,y,u)\) on \(V(i)\) for each \(i\) and of the functionals \(\Phi_0(y,u)\) and \(\Phi_1(y,u)\), implementing the connection between them by the corresponding choice of the functional \(\varphi(i,y)\).
Analogously to \((^{1a, б, в})\), the conditions of the theorem leave freedom in the choice of the functional \(\varphi(i,y)\), making it possible, by imposing additional requirements on \(\varphi\), to specify various methods for solving the problem within the proposed formalization, including discrete analogues of the methods considered in \((^{1,2})\). Let us consider some of them.
- Hamilton–Jacobi–Bellman formalism. Denote by \(V_y(i)\) the projection of the set \(V(i)\) onto \(Y\), i.e., the set of such elements \(y\in Y\) that at least one element \(u\) can be matched with \(y\) so that \((y,u)\in V(i)\); \(V_u(i,y)\) is the section of \(V(i)\) for a given \(y\in V_y(i)\). Construct on \(V_y(i)\) the functionals:
\[ P(i,y)=\sup_{u\in V_u(i,y)} R(i,y,u),\qquad i=1,2,\ldots,N-1; \]
\[ F_0(y)=\inf_{u\in V_u(0,y)} \Phi_0(y,u);\qquad F_1(y)=\inf_{u\in V_u(N,y)} \Phi_1(y,u),\qquad i=0,N, \]
and choose \(\varphi(i,y)\) so that: 1) the functional \(P(i,y)\) exists and does not depend on \(y\)
\[ P(i,y)=c(i),\qquad i=1,2,\ldots,N-1, \tag{9} \]
where \(c(i)\) is an arbitrary function; 2) the functional \(F_1(y)\) exists and does not depend on \(y\)
\[ F_1(y)=c_1, \tag{10} \]
where \(c_1\) is an arbitrary number. In practice, the choice of such a \(\varphi\) reduces to solving the Cauchy problem for the functional equation (9) with initial condition (10) in the direction from \(N\) to 0. Namely, from (10) one finds \(\varphi(N,y)=-\inf\limits_{u\in V_u(N,y)} f^0(N,y,u)+c_1\), and then from (9), for \(i=N-1\),
\[ \varphi(N-1,y)=\sup_{u\in V_u(N-1,y)} \{\varphi[N,f(N-1,y,u)]-f^0(N-1,y,u)\} \]
and so on. If the functional \(f^0(i,y,u)\) is bounded on \(V(i)\) for each \(i\in A\), then a solution \(\varphi(i,y)\) of this problem exists.
Denote by \(\{\bar u_s(i,y)\}\) a sequence of values \(u\in V_u(i,y)\) on which \(R[i,y,\bar u_s(i,y)]\to P(i,y)\) for \(i=1,\ldots,N-1\); \(\Phi_0[y,\bar u_s(0,y)]\to F_0(y)\); \(\Phi_1[y,\bar u_s(N,y)]\to F_1(y)\), \(s\to\infty\), and denote by \(\{y_s(i)\}\) a sequence of solutions of the system \(y(i+1)=f[i,y(i),\bar u_s(i,y(i))]\), \(i=0,1,\ldots,N-1\); \(F_0[\bar u_s(0)]\to m_0\).
If \(\bar y_s(i)\in V_y(i)\), \(i=1,2,\ldots,N\), then the sequence \(\{y_s(i),\bar u_s(i)=\bar u_s[i,\bar y_s(i)]\}\) belongs to \(D\) and satisfies all the conditions of the theorem, i.e., is minimizing. This will hold, in particular, if the sets \(V_y(i)\) coincide with \(Y\) for all \(i=1,2,\ldots,N\). Thus, with the given method of choosing \(\varphi(i,y)\), the posed problem is solved completely.
The functional equation (9) coincides with the equation of R. Bellman’s optimality principle (3), if one sets \(c(i)\equiv 0\), and understands \(\varphi(i,y)\) as the “payoff function” taken with the opposite sign.
5. Lagrange formalism. Let \(Y\) and \(U\) be finite-dimensional Euclidean spaces with elements \(y=(y^1,\ldots,y^n)\) and \(u=(u^1,\ldots,u^r)\), respectively; \(V(i)=V_y(i)\times V_u(i)\); the sets \(V_y(0)\) and \(V_y(N)\) are fixed points \(y_0\in Y\) and \(y_N\in Y\), while for \(i=1,2,\ldots,N-1\) \(V_y(i)\) coincides with \(Y\); \(V_u(i)\), \(i=0,1,\ldots,N\), coincides with \(U\); the vector function \(f(i,y,u)\) and the function \(f^0(i,y,u)\) are continuous and differentiable on \(V(i)\), \(i=0,1,\ldots,N\).
The idea of the method is that the function \(\varphi(i,y)\) is sought jointly with an extremal \(\bar y(i), \bar u(i)\in D\). Assuming \(\varphi(i,y)\) continuous and differentiable with respect to \(y\) for each \(i\), and introducing the vector function \(\psi(i)\), the gradient of \(\varphi\) at the extremal points,
\[ \psi(i)=\partial\varphi(i,y)/\partial y\big|_{y=\bar y(i)}, \tag{11} \]
we write the necessary conditions for the maximum of \(R\):
\[ \frac{\partial R}{\partial y}\bigg|_{y=\bar y(i),\,u=\bar u(i)} \equiv -\psi(i)+\frac{\partial}{\partial y}H[i,\psi(i+1),\bar y(i),\bar u(i)]=0, \tag{12} \]
\[ \frac{\partial}{\partial u}R(i,y,u)\bigg|_{y=\bar y(i),\,u=\bar u(i)} \equiv \frac{\partial}{\partial u}H[i,\psi(i+1),\bar y(i),\bar u(i)]=0, \tag{13} \]
where
\[ H[i,\psi,y,u]=\psi\cdot f(i,y,u)-f^0(i,y,u). \tag{14} \]
These equations are the discrete analogue of the Euler–Lagrange equations of the calculus of variations in the form of L. S. Pontryagin \({}^{(4)}\). Together with the boundary condition
\[ \bar y(0)=y_0;\qquad \bar y(N)=y_N; \]
\[ \frac{\partial}{\partial u}\Phi_0[y_0,\bar u(0)] \equiv -\psi(1)\frac{\partial}{\partial u}f(0,y_0,\bar u(0)) +\frac{\partial}{\partial u}f^0[0,y_0,\bar u(0)]=0, \tag{15} \]
\[ \frac{\partial}{\partial u}\Phi_1[y_N,u(N)] \equiv \frac{\partial}{\partial u}f^0[N,y_N,\bar u(N)]=0 \]
they determine the extremal \(\bar y(i),\bar u(i)\in D\) and \(\psi(i)\). In order to solve the problem to the end, i.e., to prove that the pair \(\bar y(i),\bar u(i)\) is the desired absolute minimum, it is necessary to prove the existence of a function \(\varphi(i,y)\) satisfying (7), (8), (11).
Remark 2. Equations (12), (13) are necessary conditions not only for the maximum of \(R(i,y,u)\), but also for the minimum of the functional \(I\) on \(D\) \({}^{(5)}\). Thus—
We also note that in the discrete variant under consideration, unlike the continuous one, the maximum of the Hamiltonian \(H\) with respect to \(u\) (the maximum principle of L. S. Pontryagin) is not a necessary condition for the maximum of \(R(i,y,u)\) and, at the same time, is not a necessary condition for the optimality of (5).
- We now prove the second part of the theorem—the necessity.
It follows from the lemma that in those cases where one can specify an obviously admissible algorithm for constructing a functional \(\varphi(i,y)\) satisfying the conditions of the lemma, conditions 1)—3) of the theorem are necessary optimality conditions. Such an algorithm is, in particular, given by Bellman’s formalism if the sets \(V_y(i)\), \(i=1,2,\ldots,N\), coincide with \(Y\) and the functional \(f^0(i,y,u)\) is bounded on \(V(i)\) for each fixed \(i\in A\).
Consider the auxiliary problem of minimizing the functional
\[ \widetilde I=\sum_{i=0}^{N}\widetilde f^{0}(i,y(i),u(i)) \]
on the set \(\widetilde D\) of pairs \(y(i),u(i)\). Here \(\widetilde f^0[i,y,u]=f^0(i,y,u)\) for \(y\in V_y(i)\), and \(\widetilde f^0=K\) for \(y\notin V_y(i)\); \(K=q+\sup_{y,u\in V(i),\, i\in A} f^0(i,y,u)\); \(q\) is any number satisfying the inequality
\[ q>\sup_E I-\inf_E I =\sum_{i=0}^{N}\left[\sup_{y,u\in V(i)} f^0(i,y,u)-\inf_{y,u\in V(i)} f^0(i,y,u)\right]. \]
The set \(\widetilde D\) differs from \(D\) only in the structure of the set \(\widetilde V(i)\) of admissible pairs \(y,u\in Y\times U\). Namely, \(\widetilde V_y(i)\) coincides with \(Y\) for all \(i\in A\), while \(\widetilde V_u(i,y)\) coincides with \(V_u(i,y)\) for \(y\in V_y(i)\) and coincides with \(U\) for \(y\in Y/V_y(i)\). Obviously, \(D\subset \widetilde D\) and \(\widetilde I=I\) for \(y(i),u(i)\in D\). On any element \(\widetilde D\) not belonging to \(D\), the value of the functional \(\widetilde I\) is greater than on any element of \(D\). Indeed, suppose the condition \(y(i),u(i)\in V(i)\) is violated for \(1\le m\le N+1\) values of \(i\); then
\[ \widetilde I-\sup_D I\ge mq+\inf_E I-\sup_E I>(m-1)q\ge 0. \]
This means that, beginning with some \(s=S\), all elements of a minimizing sequence of the functional \(\widetilde I\) will lie in \(D\): \(\{\bar y_s(i),\bar u_s(i)\}\subset D\) for \(s>S\). And since \(\widetilde I=I\) on \(D\), this sequence is also minimizing for the functional \(I\).
Let the functional \(\varphi(i,y)\) satisfy the conditions of the theorem for this problem. The functionals \(R(i,y,u)\), \(\Phi_1\), and \(\Phi_2\) corresponding to this \(\varphi(i,y)\) coincide on \(V(i)\) with the analogous functionals for the original problem, and, since \(V(i)\subset \widetilde V(i)\), they tend on the sequence \(\{\bar y_s(i),\bar u_s(i)\}\subset D\) to their greatest (least) value on \(V(i)\), i.e. the functional \(\varphi(i,y)\) satisfies conditions 1)—3) of the theorem also for the problem of minimizing \(I\) on \(D\). Since the problem of minimizing \(\widetilde I\) on \(\widetilde D\) belongs to the type singled out above, for which the existence of such a functional \(\varphi(i,y)\) is necessary, it is necessary also for the original problem. The theorem is proved.
Moscow Aviation Institute
named after S. Ordzhonikidze
Received
10 II 1966
CITED LITERATURE
\(^{1}\) V. F. Krotov, Avtomatika i telemekhanika, a) 23, No. 12 (1962); b) 24, No. 5 (1963); c) 25, No. 7 (1964); d) 25, No. 11 (1964).
\(^{2}\) V. I. Gurman, Avtomatika i telemekhanika, a) 26, No. 5 (1965); b) 26, No. 7 (1965); c) Kosmicheskie issledovaniya, 3, issue 3 (1965).
\(^{3}\) R. Bellman, Dynamic Programming, Foreign Literature Publishing House, 1960.
\(^{4}\) L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, E. F. Mishchenko, Mathematical Theory of Optimal Processes, 1961.
\(^{5}\) A. G. Butkovskii, Optimal Processes in Systems with Distributed Parameters, Nauka, 1965.