Yu. M. Berezanskii

ON SMOOTHNESS UP TO THE BOUNDARY OF THE DOMAIN OF THE SPECTRAL FUNCTION OF A SELF-ADJOINT DIFFERENTIAL ELLIPTIC OPERATOR

(Presented by Academician S. L. Sobolev on 10 IV 1963)

The existence of the spectral function, i.e. the kernel of an operator of the type \(dE_\lambda/d\rho(\lambda)\), for the case of the Schrödinger equation was established by A. Ya. Povzner \(\left({}^{1}\right)\), and in the general case of elliptic equations by Gårding \(\left({}^{2}\right)\) and Browder \(\left({}^{3}\right)\) (see also \(\left({}^{4}\right)\)). In this note, for general equations, the behavior of the spectral function near the boundary of the domain is studied. Similar facts have been published only for the Schrödinger equation \(\left({}^{1}\right)\). The methods of the note are based on theorems on the local increase of smoothness of solutions of elliptic equations \(\left({}^{5,6}\right)\) and on properties of the Green functions of such equations \(\left({}^{7}\right)\).

It is convenient for us in § \(1^\circ\) to set forth general questions of expansion in generalized eigenvectors of a self-adjoint operator (for the literature see \(\left({}^{8}\right)\)), and in § \(2^\circ\) to supplement the theory of expansions for Carleman operators proposed by Mautner \(\left({}^{9}\right)\) and developed in one direction or another in a number of works (see \(\left({}^{8}\right)\)).

\(1^\circ\). Let \(H_0\) be a separable Hilbert space with scalar product \((\,\cdot,\cdot\,)_0\); let \(A\) be a self-adjoint operator acting in \(H_0\); and let \(E(\Delta)\) be its resolution of the identity. An expansion in generalized eigenvectors of \(A\) can be constructed as follows. Denote by
\[ H_- \supset H_0 \supset H_+ \]
some rigging of \(H_0\) by positive and negative spaces, and suppose that the embedding \(H_+ \to H_0\) is quasi-nuclear \(\left({}^{10}\right)\). There exists a nonnegative finite measure \(d\rho(\lambda)\) on the axis \((-\infty,\infty)\) (the spectral density) and an operator function \(P(\lambda)\), defined \(\rho\)-almost everywhere for all \(\lambda \in (-\infty,\infty)\), whose values are Hilbert–Schmidt operators acting from \(H_+\) into \(H_-\), such that
\[ E(\Delta)=\int_\Delta P(\lambda)\,d\rho(\lambda). \tag{1} \]

The range \(\mathfrak{R}(P(\lambda))\) of the operator \(P(\lambda)\) consists, in a certain sense, of generalized eigenvectors of the operator \(A\) corresponding to the number \(\lambda\). Moreover, \(\|P(\lambda)\| \le 1\) (\(\|\cdot\|\) is the Hilbert–Schmidt norm), \((P(\lambda)u,u)_0 \ge 0\) (\(u \in H_+\)). Suppose additionally that there exists a linear topological separable space \(D \subseteq H_+\), dense in \(H_+\), contained in the domain of definition \(\mathfrak{D}(A)\), and mapped continuously by the operator \(A\) into \(H_+\). Then every \(\varphi \in \mathfrak{R}(P(\lambda))\) satisfies the relation
\[ (\varphi,(A-\lambda E)u)_0=0 \quad (u\in D). \]
One can choose (in a nonunique way) a system of vectors
\(\varphi_\alpha(\lambda)\in\mathfrak{R}(P(\lambda))\subseteq H_-\)
\((\alpha=1,\ldots,N_\lambda\le\infty)\) such that
\[ (P(\lambda)u,v)_0 = \sum_{\alpha=1}^{N_\lambda} \overline{u_\alpha(\lambda)}\,v_\alpha(\lambda) \quad (u,v\in H_+),\qquad \widetilde{w}(\lambda)=\bigl(w_1(\lambda),\,w_2(\lambda),\ldots\bigr) \tag{2} \]
\[ \bigl(w_\alpha(\lambda)=(\varphi_\alpha(\lambda),w)_0\bigr) \]
is the “Fourier transform” of the vector \(w\in H_+\).

The condition that the embedding \(H_+ \to H_0\) be quasi-nuclear is also necessary for obtaining (1) for arbitrary \(A\). However, for a fixed \(A\) one may impose a less stringent requirement on this embedding: it is sufficient to assume that there exists a continuous bounded function \(\gamma(\lambda)\), different from zero on the spectrum of \(A\), such that \(\|\gamma(A)J\|<\infty\), where \(J\) is an operator mapping \(H_0\) into \(H_+\) \(\left({}^{10}\right)\) and considered in \(H_0\). We note that the results of this and the following items are also valid for generalized resolutions of the identity.

2°. Let \(H_0=L_2(Q,dx)\), where \(Q\) is a locally compact separable space, and \(dx\) is a measure defined on the Borel sets of \(Q\), finite on compact sets and positive on open sets. A self-adjoint operator \(A\) in \(H_0\) will be called Carleman if, for \(\gamma(\lambda)\) of the form indicated above, the operator \(\gamma(A)\) is integral with kernel \(K(x,y)\), and
\[ \int_Q |K(x,y)|^2\,dx<\infty \]
for almost every \(y\in Q\). Choose a function \(p(x)\geqslant 1\) \((x\in Q)\) so that
\[ \int_Q\int_Q |K(x,y)|^2 p^{-1}(y)\,dx\,dy<\infty; \]
then, according to what was said at the end of § 1°, the expansion in generalized eigenfunctions of the operator \(A\) can be constructed along the chain
\[ H_-\supset H_0\supset H_+ \]
of the form
\[ L_2(Q,p^{-1}(x)\,dx)\supset L_2(Q,dx)\supset L_2(Q,p(x)\,dx). \]
Thus, for a Carleman operator every generalized eigenfunction is an ordinary one, but, generally speaking, it does not belong to \(L_2(Q,dx)\). For such an operator the following holds.

Theorem 1. The operator \(P(\lambda)\) is an integral operator
\[ (P(\lambda)u)(x)=\int_Q P(x,y;\lambda)u(y)\,dy \qquad \bigl(u\in L_2(Q,p,dx)\bigr) \tag{3} \]
with a positive definite kernel \(P(x,y;\lambda)\)—the spectral function of \(A\)—such that
\[ \|P(\cdot,\cdot;\lambda)\|_{L_2(Q\times Q,p^{-1}(x)p^{-1}(y)\,dx\,dy)}\leqslant 1. \]
Relation (2) gives the expansion convergent in
\[ L_2(Q\times Q,p^{-1}(x)p^{-1}(y)\,dx\,dy): \]
\[ P(x,y;\lambda)=\sum_{\alpha=1}^{N_\lambda}\varphi_\alpha(x;\lambda)\varphi_\alpha(y;\lambda) \qquad \bigl(\varphi_\alpha(\cdot;\lambda)=\varphi_\alpha(\lambda)\bigr). \tag{4} \]

If \(P(x,y;\lambda)\) is continuous with respect to \((x,y)\in Q\times Q\), then each of the eigenfunctions \(\varphi_\alpha(x;\lambda)\) \((\alpha=1,\ldots,N_\lambda)\) is continuous in \(x\in Q\), and the expansion (4) converges absolutely and uniformly on every compact subset of \(Q\times Q\).

Let \(f(\lambda)\) be bounded on the spectrum of \(A\); then \(f(A)\) is also an integral operator (a relation analogous to (3) holds), and its kernel \(F(x,y)\) is representable in the form of an absolutely convergent, for almost every \((x,y)\) with respect to the measure \(dx\,dy\), integral
\[ F(x,y)=\int_{-\infty}^{\infty} f(\lambda)P(x,y;\lambda)\,d\rho(\lambda) \in L_2\bigl(Q\times Q,p^{-1}(x)p^{-1}(y)\,dx\,dy\bigr). \tag{5} \]

If \(P(x,y;\lambda)\) is continuous with respect to \((x,y)\in Q\times Q\) and \(\|K(\cdot,y)\|_{L_2(Q,dx)}\) is bounded on every compact subset of \(Q\), then the integral (5), in the case
\[ |f(\lambda)|\leqslant C|\gamma(\lambda)|^2, \]
converges absolutely for every \(x,y\) and is bounded on every compact subset of \(Q\times Q\).

This theorem is proved by developing the arguments in \((^4)\). We note that if, in addition, the kernel of the operator \((\gamma(A))^*\gamma(A)\) is continuous on \(Q\times Q\), then the kernel (5) is also continuous. The growth estimate for the integrals of the eigenfunctions from \((^4)\) (Theorem 3.3) is also valid for general Carleman operators in \(L_2(E_n,dx)=L_2(E_n)\) (\(E_n\) is \(n\)-dimensional space, \(dx\) is Lebesgue measure). Other estimates of the behavior at infinity of eigenfunctions of Carleman operators are given in \((^{11,12})\).

3°. Let \(G\subseteq E_n\), generally speaking, be an unbounded domain, and let \(\Gamma\) be its piecewise-smooth boundary. In \(G\) consider an elliptic formally self-adjoint differential expression
\[ \mathscr L=\sum_{|\mu|\leqslant r} a_\mu(x)D^\mu, \]
where
\[ a_\mu\in C^{|\mu|}(G\cup\Gamma). \]
On \(\Gamma\) there are given formally self-adjoint boundary conditions
\[ (\mathrm{gr})=(\mathrm{gr})^+, \]
defined by a subspace \(W_2^r(\mathrm{gr})\) of the Sobolev space \(W_2^r(G)\) (see more details in \((^{13})\)). The operator in \(L_2(G)\)
\[ u\mapsto \mathscr Lu \]
\[ \bigl(u\in W_2^r(\mathrm{gr})\cap W_{2,0}^r(E_n);\; W_{2,0}^r(G)\text{—finite functions in }W_2^r(G)\bigr) \]
is, obviously, Hermitian; let \(A\) be one of its self-adjoint extensions (in \(L_2(G)\) or with values in a broader space), and let \(E(\Delta)\) be the corresponding (generalized) resolution of the identity.

Let \(a_\mu\in C^{2r+n+p}(G)\) (\(p\ge n+1\)); with the aid of Browder’s theorem on the Carleman property of the resolvent kernel of an elliptic operator of high order \({}^{(3)}\) it is easy to prove that \(A\) is a Carleman operator in \(L_2(Q,dx)=L_2(G)\), and \(\gamma(\lambda)=(\lambda^N-z)^{-1}\left(N=\left[\frac{n}{2r}\right]+1,\ \operatorname{Im} z\ne0\right)\) and \(\|K(\cdot,y)\|_{L_2(G)}\) is bounded on every compact subset of \(G\). Thus the results of item \(2^\circ\) are applicable to \(A\). The function \(\rho(x)\) may be taken from \(C^\infty(G)\); generally speaking, it tends to \(\infty\) as \(x\to\Gamma\) and to \(\infty\). Theorem 1, in particular, establishes the existence of a spectral function \(P(x,y;\lambda)\in L_2(G\times G,\rho^{-1}(x)\rho^{-1}(y)\,dx\,dy)\). Since in the sense of the theory of generalized functions inside \(G\times G\)

\[ \mathcal L_x P(x,y;\lambda)=\lambda P(x,y;\lambda),\qquad \overline{\mathcal L}_y P(x,y;\lambda)=\lambda P(x,y;\lambda), \tag{6} \]

it follows, according to \({}^{(14)}\), that \(P(x,y;\lambda)\) will be sufficiently smooth (all derivatives of the form \(D_x^\alpha D_y^\beta P(x,y;\lambda)\) exist and are continuous in \(G\times G\), \(|\alpha|,|\beta|\le r+n+p\)) and (6) are satisfied in the ordinary sense for \(x,y\in G\). Every \(\varphi\in\mathfrak R(P(\lambda))\) also belongs to \(C^{r+n+p}(G)\), and the expansion (4) may be differentiated term by term—one may take the indicated derivatives \(D_x^\alpha D_y^\beta\). The differentiated expansion (4) will converge absolutely and uniformly on every compact subset of \(G\times G\) (the last results are obtained with the aid of \({}^{(15)}\)).

\(4^\circ\). Let us pass to the formulation of the principal results. In what follows the assumptions of item \(3^\circ\), ensuring the existence and smoothness of \(P(x,y;\lambda)\) inside \(G\times G\), are assumed to be fulfilled.

Theorem 2. Let \(r=2m\) and let the expression \(\mathcal L\) be properly elliptic in \(G\cup\Gamma\) \({}^{(16)}\). Suppose that there exists a bounded subdomain \(\overline{G}_0\subseteq G\), adjoining \(\Gamma\) along a piece \(\Gamma_0\) of class \(C^{4m+q}\) \((q\ge n/2)\), such that \(a_\mu\in C^{2m+\max(|\mu|,q)}(G_0\cup\Gamma_0)\). On \(\Gamma_0\) differential expressions are given

\[ B_j=\sum_{|\mu|\le m_j} b_{j\mu}(x)D^\mu \quad \bigl(b_{j\mu}\in C^{2m+q-1}(\Gamma_0);\quad m_j\le 2m-1;\quad j=1,\ldots,m\bigr), \]

which are assumed to be normal and to cover \(\mathcal L\) \({}^{(16)}\); (bc) on \(\Gamma_0\) have the form: \(B_j u|_{\Gamma_0}=0\) \((j=1,\ldots,m)\). In addition, suppose that: a) in some \(\delta\)-strip \(\Gamma_{0\delta}\subset\Gamma_0\) near the boundary \(\dot\Gamma_0\) of the piece of surface \(\Gamma_0\), the expressions \(B_j\) contain only terms with differentiations along the normal to \(\Gamma_0\); b) in some neighborhood in \(G_0\) of the strip \(\Gamma_{0\delta}\), the expression \(\mathcal L\) contains no terms with mixed derivatives in the normal to \(\Gamma_0\) and tangential directions.

Then the spectral function \(P(x,y;\lambda)\), corresponding to the ordinary resolution of the identity, belongs to
\[ L_{2,\mathrm{loc}}\bigl(G_0\times G_0,\ ((G_0\cup\Gamma_0)\times\Gamma_0)\cup(\Gamma_0\times(G_0\cup\Gamma_0))\bigr)^*. \]
For fixed \(y\in G\) \((x\in G)\) it belongs to \(W^{2m+q}_{2,\mathrm{loc}}(G_0,\Gamma_0)\) and on \(\Gamma_0\) satisfies the boundary conditions:

\[ B_{j,x}P(x,y;\lambda)\big|_{x\in\Gamma_0}=0 \quad \bigl(\overline{B}_{j,y}P(x,y;\lambda)\big|_{y\in\Gamma_0}=0\bigr); \quad j=1,\ldots,m. \tag{7} \]

Each eigenfunction \(\varphi(x;\lambda)\in\mathfrak R(P(\lambda))\) also belongs to \(W^{2m+q}_{2,\mathrm{loc}}(G_0,\Gamma_0)\) and satisfies on \(\Gamma_0\) (bc): \(B_{j,x}\varphi(x;\lambda)|_{x\in\Gamma_0}=0\) \((j=1,\ldots,m)\). The expansion (4) converges absolutely and uniformly in any bounded domain in \(G\times G\) which adjoins the boundary \(G\times G\) inside the piece
\[ \bigl((G_0\cup\Gamma_0)\times\Gamma_0\bigr)\cup \bigl(\Gamma_0\times(G_0\cup\Gamma_0)\bigr). \]

Remarks. 1) If \(G\) is the interior or exterior of some closed surface \(\Gamma\) and \(\Gamma_0=\Gamma\), then conditions a) and b), naturally, are dropped. They may also be dropped if the (bc) are zero on \(\Gamma_{0\delta}\): \(B_j|_{\Gamma_{0\delta}}=\partial^{j-1}/\partial\nu^{j-1}\) \((j=1,\ldots,m;\ \nu\) is the normal to \(\Gamma_0\)). 2) If the (bc) are zero everywhere on \(\Gamma_0\) and \(\mathcal L\) is strongly elliptic in \(G_0\cup\Gamma_0\), then the smoothness restrictions can be weakened: it suffices to assume \(\Gamma_0\) of class \(C^{2m+q}\), and \(a_\mu\in C^{|\mu|+q}(G_0\cup\Gamma_0)\).

\[ {}^*\ \text{Let } \mathfrak D \text{ be a bounded domain, } \gamma \text{ a piece on its boundary. Recall } {}^{(5)} \text{ that we write } u\in W^l_{2,\mathrm{loc}}(\mathfrak D,\gamma)\ (l=0,1,\ldots), \]
if \(u\in W^l_2(\mathfrak D')\) for every domain \(\mathfrak D'\) having common boundary with \(\mathfrak D\) only inside the piece \(\gamma\).

Let us outline the proof. First, it is shown, by means of arguments of the type of (7), that \((A-zE)^{-N}\) \((N=[n/2r]+1,\ \operatorname{Im} z\ne0)\) is an integral operator with kernel \(K(x,y)\) such that the vector-function \(K(\cdot,y)\), with values in \(L_2(G)\) for \(y\in G'_0\cup\Gamma'_0\), is weakly continuously differentiable up to order \(Nr-[n/2]-1\) inclusive \((G'_0\) is a subdomain of \(G_0\) adjoining \(\Gamma\) only along the piece \(\Gamma'_0\) lying inside \(\Gamma_0)\). Then we apply the construction of §2, taking \(Q=G\cup\Gamma,\ L_2(Q,dx)=L_2(G),\ \gamma(\lambda)=(\lambda-z)^{-N}\) (this choice of \(\gamma(\lambda)\), different from that in §3, as is easily seen, does not change \(P(x,y;\lambda)\)). Since \(\|K(\cdot,y)\|_{L_2(G)}\) is bounded up to any \(\Gamma'_0\subset\Gamma_0\), the function \(\rho(x)\) in §2 may be taken from \(C^\infty(G\cup\Gamma)\). This implies the relations: \(L_2(G,\rho^{-1}dx)\subset L_{2,\mathrm{loc}}(G_0,\Gamma_0)\) and \(L_2(G\times G,\rho^{-1}(x)\rho^{-1}(y)\,dxdy)\subset L_{2,\mathrm{loc}}(G\times G,((G_0\cup\Gamma_0)\times\Gamma_0)\cup(\Gamma_0\times(G_0\cup\Gamma_0)))\). The smoothness of \(P(x,y;\lambda)\) and the relations (7) now follow from the theorem on increasing the smoothness of solutions \((^5,^6)\) and from the fact that (6) are satisfied up to \(\Gamma_0\) (let us explain that, as the space \(D\) of §1, one may now take the suitably topologized set of functions \(u\in W_2^r(G)\) which vanish in neighborhoods of \(\Gamma\setminus\Gamma_0\) and of \(\infty\), and on \(\Gamma_0\) satisfy the conditions: \(B_j u|_{\Gamma_0}=0,\ j=1,\ldots,m\)).

Theorem 3. Let \(G\) be the interior or the exterior of some closed surface \(\Gamma\). Suppose that the assumptions of Theorem 2 are fulfilled in the domain \(G_0=G\) (conditions a) and b) are omitted), or the remarks 2 thereto. In that case all derivatives of the form \(D_x^\alpha D_y^\beta P(x,y;\lambda)\), where
\[ |\alpha|,\ |\beta|\le r\left\{\min\left(\left[\frac{q}{r}\right],\left[\frac{p-1}{2r}\right]\right)+1\right\}-\left[\frac{n}{2}\right]-1, \]
exist and are continuous in \((G\cup\Gamma)\times(G\cup\Gamma)\). The expansion (4) converges absolutely and uniformly in every bounded part of \(G\times G\), and it may be differentiated term by term without violating this convergence—namely, one may take precisely the derivatives \(D_x^\alpha D_y^\beta\) just indicated.

The proof in the case of bounded \(G\) follows from the representation
\[ P(x,y;\lambda)=|\lambda-z|^{2N}\int_G\int_G \overline{K(\xi,x)}K(\eta,y)P(\xi,\eta;\lambda)\,d\xi\,d\eta \]
\((x,y\in G)\) and from the smoothness of \(K(\cdot,y)\) indicated above. In the case of unbounded \(G\) the proof is considerably more complicated, since the last integral does not exist. It is based on a close representation, the idea of which was suggested by a certain transform of Gårding (see \((^2)\), §2).

Theorem 3 shows that under its conditions in (7) one may take \(y(x)\) on \(\Gamma\), one may differentiate (7) with respect to \(y(x)\), and so on. We note that a similar theorem is also valid for arbitrary \(G\).

In conclusion we observe that, for the study of \(P(x,y;\lambda)\), one may also apply a technique using the expression \(\mathscr{L}_x+\mathscr{L}_y^+\), developed in \((^4,^7)\).

Institute of Mathematics
Academy of Sciences of the Ukrainian SSR

Received
6 IV 1963

CITED LITERATURE

1. A. Ya. Povzner, Matem. sborn., 32 (74), No. 1, 109 (1953).
2. L. Gårding, 12 Congrès des Math. Scand., 1953, Lund, 1954, p. 44; Sborn. per. Matematika, 1, No. 3, 1957, p. 107.
3. F. E. Browder, Proc. Nat. Acad. Sci. U. S. A., 40, No. 6, 454 (1954).
4. Yu. M. Berezanskii, Matem. sborn., 43 (85), No. 1, 75 (1957).
5. Yu. M. Berezanskii, S. G. Krein, Ya. A. Roitberg, DAN, 148, No. 4 (1963).
6. Ya. A. Roitberg, Ukr. matem. zhurn., 15, No. 4 (1963).
7. Yu. M. Berezanskii, Ya. A. Roitberg, Ukr. matem. zhurn., 15, No. 2 (1963).
8. Yu. M. Berezanskii, A. G. Kostyuchenko, Tr. IV Vsesoyuzn. matem. s’ezda, 1963.
9. F. I. Mautner, Proc. Nat. Acad. Sci. U. S. A., 39, No. 1, 49 (1953); UMN, 10, No. 4 (1955).
10. Yu. M. Berezanskii, UMN, 18, No. 1, 63 (1963).
11. Yu. M. Berezanskii, Ukr. matem. zhurn., 11, No. 1, 16 (1959).
12. Yu. M. Berezanskii, Yu. B. Orochko, Ukr. matem. zhurn., 14, No. 2, 180 (1962).
13. Yu. M. Berezanskii, DAN, 131, No. 3, 478 (1960).
14. Ya. A. Roitberg, Dokl. AN URSR, No. 6, 721 (1960).
15. M. G. Krein, Ukr. matem. zhurn., 1, No. 4, 64 (1949).
16. M. Schechter, Comm. Pure and Appl. Math., 12, No. 3, 457 (1959); 12, No. 4, 561 (1959); Sborn. per. Matematika, 4, No. 5, 1960, p. 92; 4, No. 6, 1960, p. 3.