UDC 519.46

MATHEMATICS

V. F. MOLCHANOV

AN ANALOGUE OF THE PLANCHEREL FORMULA FOR HYPERBOLOIDS

(Presented by Academician P. S. Novikov on 23 I 1968)

1. Let \(G\) be a connected group of linear transformations of an \(n\)-dimensional real space preserving the quadratic form
   \([x,x]=-x_1^2-\cdots-x_p^2+x_{p+1}^2+\cdots+x_n^2\) (a pseudo-orthogonal group), and let \(X\) be the hyperboloid \([x,x]=1\). The measure \(dx=|x_1|^{-1}dx_2\cdots dx_n\) on \(X\) is invariant under the natural action \(x\to xg\) of the group \(G\). In the space \(L^2(X)\) with respect to this measure there arises a unitary representation of the group \(G\): \(f(x)\to f(xg)\). For \(q=n-p>1\) the stationary subgroup of a point of the hyperboloid \(X\) is not compact; therefore harmonic analysis of functions on the hyperboloid is connected with considerable difficulties. The case \(p=1\) was considered in the papers \((^1)\) (\(q=3\)), \((^3)\)*, \((^4)\). In the present note the problem is solved of decomposing the unitary representation of the group \(G\) in \(L^2(X)\) into irreducible components for odd \(p>1\) and arbitrary \(q>1\), with the aid of spherical (zonal) functions.

2. Representations of the pseudo-orthogonal group connected with the cone. Let \(X_0\) be the cone \([x,x]=0,\ x\ne0\). The group \(G\) acts transitively on the cone.

Let \(\chi\) denote the pair \((\sigma,\varepsilon)\), where \(\sigma\) is a complex number and \(\varepsilon\) takes the values 0 and 1. Let \(D_\chi\) be the space of infinitely differentiable functions \(\varphi(\xi)\) on the cone \(X_0\), homogeneous of degree \(\sigma\) and parity \(\varepsilon\),

\[ \varphi(t\xi)=|t|^\sigma \operatorname{sign}^{\varepsilon}t\cdot\varphi(\xi), \]

with the usual topology. In \(D_\chi\) acts the representation \(T_\chi\) of the group \(G\):

\[ T_\chi(g)\varphi(\xi)=\varphi(\xi g). \]

Let \(\sigma^*=2-n-\sigma,\ \chi^*=(\sigma^*,\varepsilon),\ \bar\chi^*=(\bar\sigma^*,\varepsilon)\). If \(\sigma\) is not an integer, then \(T_\chi\) is irreducible and equivalent to \(T_{\chi^*}\).

Let \(S\) be the section of the cone by the Euclidean sphere of radius \(\sqrt{2}\) with center at the origin; \(S\) is the direct product of two Euclidean spheres of unit radius, \(S=S^{p-1}\times S^{q-1}\). The space \(D_\chi\) may be realized as the space of infinitely differentiable functions \(\varphi(s)\) on \(S\) of parity \(\varepsilon\): \(\varphi(-s)=(-1)^\varepsilon\varphi(s)\). The Hermitian form

\[ (\varphi,\psi)=\int \varphi(s)\overline{\psi(s)}\,ds \tag{1} \]

(where \(ds\) is the Euclidean measure on \(S\)) on the pair of spaces \(D_\chi,D_{\bar\chi^*}\) is invariant with respect to \(G\).

The operator \(A_\chi\)

\[ A_\chi\varphi(s)= \frac{1}{\Gamma((q-1)/2+\sigma^*)} \int \left\{[s,s']_-^{\sigma^*}+(-1)^\varepsilon [s,s']_+^{\sigma^*}\right\}\varphi(s')\,ds' \]

* In communication \((^3)\) the contribution of the integral points of the supplementary series to the decomposition formula was omitted; cf. formula (2) of the present note, which refers to the case \(p=1\). In this case there exist two spherical functions corresponding to one representation of the continuous series.

maps \(D_{\chi}\) into \(D_{\chi^*}\) and commutes with the group \(G\). (The integral converges for \(\operatorname{Re}\sigma<3-n\); for the remaining \(\sigma\) it is analytically continued as a meromorphic function.)

The representations \(T_\chi\) are unitary in the following cases:

a) the continuous series: \(\sigma=(2-n)/2+i\rho\), \(-\infty<\rho<\infty\), \(\varepsilon=0,1\), scalar product \((\varphi,\psi)\);

b) the supplementary series: \((1-n)/2<\sigma<(3-n)/2\), \(\varepsilon=0,1\) for odd \(n\); \(-n/2<\sigma<(4-n)/2\), \(\varepsilon\equiv n/2-p\) for even \(n\) (here and below the congruence sign denotes congruence modulo 2); scalar product \((A_\chi\varphi,\psi)\).

For integral \(\sigma\) the representations \(T_\chi\) are, generally speaking, reducible. To clarify the structure of the representations \(T_\chi\) for integral \(\sigma\), we use the restriction of \(T_\chi\) to the maximal compact subgroup \(K\) of the group \(G\). \(K\) is isomorphic to the direct product of two orthogonal groups: \(K=SO(p)\times SO(q)\), and acts transitively on \(S\) (each component on its own sphere). In the realization on \(S\), the representation \(T_\chi\) assigns to elements of the group \(K\) rotations: \(T_\chi(k)\varphi(s)=\varphi(sk)\), \(k\in K\). The representation of the group \(K\) by rotations in the space \(C^\infty(S)\) decomposes into a direct sum of representations, each of which is realized in the space of homogeneous polynomials harmonic and of degree \(l\) in the variables \(s_1,\ldots,s_p\) and harmonic and of degree \(m\) in the variables \(s_{p+1},\ldots,s_n\) (see [2], Ch. IX), and is therefore specified by a pair \((l,m)\) of nonnegative integers. We shall call this pair the weight of the representation.

The restriction of \(T_\chi\) to \(K\) contains representations with weights from the set \(\Lambda^\varepsilon\): \(l+m\equiv\varepsilon\). Consider the following four functions, linear with respect to \(l,m\):

\[ \begin{aligned} \beta_1^\sigma(l,m)&=\sigma-l-m, & \beta_3^\sigma(l,m)&=\sigma+p-2+l-m,\\ \beta_2^\sigma(l,m)&=\sigma+q-2-l+m, & \beta_4^\sigma(l,m)&=\sigma+n-4+l+m. \end{aligned} \]

If the line \(\beta_i^\sigma(l,m)=0\) intersects \(\Lambda^\varepsilon\), then the subspace \(D_{\chi,i}\) in \(C^\infty(S)\), containing the spaces of representations with weights from \(\{\beta_i^\sigma(l,m)\ge 0\}\cap\Lambda^\varepsilon\), is invariant with respect to \(G\).

From the representations \(T_\chi\) with integral \(\sigma\) we extract the following two discrete series of unitary representations.

The first discrete series is realized, for integral \(\sigma>(2-n)/2\), \(\varepsilon\equiv p+\sigma\), in the factor spaces \(\widetilde D_\sigma=D_\chi/D_{\chi,3}\); the scalar product is induced by the form \((\varphi,\psi)_\sigma=(A_\chi\varphi,\psi)\).

The second discrete series is realized, for integral \(\sigma>(2-n)/2\), \(\varepsilon\equiv\sigma+q\), in the factor spaces \(D_\chi/D_{\chi,2}\). The representations of the first (respectively, second) discrete series are irreducible for \(q>2\) (respectively, \(p>2\)) and decompose into the direct sum of two irreducible representations for \(q=2\) (respectively, \(p=2\)).

The representation \(T'_\chi\) of the group \(G\), contragredient to \(T_{\chi^*}\), is constructed in the space \(D'_\chi\) of functionals on \(D_{\chi^*}\) (therefore, in the space of generalized functions on \(S\) of parity \(\varepsilon\)) and is given by the formula

\[ (T'_\chi(g)F,\varphi)=(F,T_{\chi^*}(g)\varphi),\qquad \varphi\in D_{\chi^*},\quad F\in D'_\chi . \]

The space \(D_\chi\) is embedded in \(D'_\chi\) with the aid of the Hermitian form (1), and \(T'_\chi\) is an extension of \(T_\chi\). The structure of invariant subspaces is transferred from \(D_\chi\) to \(D'_\chi\). In particular, we denote by \(\widetilde D'_\chi\) the space \(D'_\chi/D'_{\chi,1}\). The operator \(A_\chi\) maps \(\widetilde D'_\chi\) into \(D_{\chi^*}\).

1. Spherical functions. Let \(G_0\) be the stationary subgroup of the point \(x^0=(0,\ldots,0,1)\in X\). We indicate functions from \(D'_\chi\) invariant with respect to \(G_0\). For the continuous series,

\[ \theta_{\rho,\varepsilon}(s)=(s_n)_+^\sigma+(-1)^\varepsilon(s_n)_{-}^{\sigma},\qquad \sigma=(2-n)/2+i\rho . \]

For the first discrete series \((\chi=\chi_l=(l,\varepsilon)\), where \(l\) is an integer \(\ge -[\nu]\), \(\nu=(n-3)/2,\ \varepsilon \equiv p+l)\)

\[ \theta_l(s)=\{(s_n)^\sigma+(-1)^\varepsilon (s_n)^\sigma\}/\Gamma(\sigma+1)\big|_{\sigma=l} \quad \text{for } p\equiv 1, \]

\[ \theta_l(s)=\{(s_n)^\sigma+(-1)^\varepsilon (s_n)^{-\sigma}\}_{\sigma=l} \quad \text{for } p\equiv 0. \]

Denote
\[ \theta_{\rho,\varepsilon}^x=T_\chi'(g^{-1})\theta_{\rho,\varepsilon};\qquad \theta_l^x=T_\chi'(g^{-1})\theta_l, \]
where \(x=x^0=x^0g\). To each finite infinitely differentiable function \(f(x)\) on \(X\) we associate its Fourier components of the continuous series:

\[ F_{\rho,\varepsilon}(s)=\int \theta_{\rho,\varepsilon}^x(s)f(x)\,dx\in D_\chi \]

and of the first discrete series \(\widetilde G_l\), where \(\widetilde G_l\) is an element of the factor space \(\widetilde D_{\chi_l}\) with representative

\[ G_l(s)=\int \theta_l^x(s)f(x)\,dx\in D_{\chi_l}. \]

By the spherical function corresponding to the representation of the continuous series (respectively, of the first discrete series) we mean the generalized function \(\Phi_{\rho,\varepsilon}\) (respectively \(\Psi_l\)) on the hyperboloid \(X\), whose value on \(f(x)\) is equal to \((\theta_{\rho,\varepsilon},F_{\rho,\varepsilon})\) (respectively \((A_{\chi_l}\theta_l,G_l)\)).

For \(p\equiv 0,\ l\ge 0\), the function \(\theta_l\) is a polynomial and belongs to the space \(D_{\chi,1}\); therefore the spherical function of the first discrete series defined in the preceding paragraph vanishes, and hence the expansion formula given below is proved under the assumption that \(p\) is odd.

The main result of the work is the expansion of the delta-function on the hyperboloid in spherical functions for odd \(p>1\)

\[ \delta=\sum_{l=-[\nu]}^\infty a_l\Psi_l+\int_{-\infty}^{\infty}\omega(\rho)\sum_{\varepsilon}\Phi_{\rho,\varepsilon}\,d\rho, \tag{2} \]

where \(\delta\) is the delta-function concentrated at \(x^0\),

\[ a_l=(2l+n-2)\cdot 2^{\,l-n-3}\pi^{-3/2(n-2)}\Gamma(n-2+l)\Gamma(l+n/2)/\Gamma((n+p-1)/2+l), \]

\[ \omega(\rho)=2^{-n-1}\pi^{-n}\rho\,\operatorname{sh}\pi\rho\cdot \left|\Gamma((n-2)/2+i\rho)\right|^2. \]

From (2) follows the inversion formula

\[ f(x)=\sum_{l=-[\nu]}^\infty a_l\,\overline{(A_{\chi_l}\theta_l^x,G_l)} +\int_{-\infty}^{\infty}\omega(\rho)\sum_{\varepsilon} \overline{(\theta_{\rho,\varepsilon}^x,F_{\rho,\varepsilon})}\,d\rho, \]

where the integral and the series converge absolutely and uniformly with respect to \(x\) from any compact subset of \(X\), and the analogue of the Plancherel formula for odd \(p>1\):

\[ \int |f(x)|^2\,dx = \sum_{l=-[\nu]}^\infty a_l\,(G_l,G_l)_l +\int_{-\infty}^{\infty}\omega(\rho)\sum_{\varepsilon} (F_{\rho,\varepsilon},F_{\rho,\varepsilon})\,d\rho. \]

4. Explicit expressions for the spherical functions

The spherical functions for \(|x_n|\ne 1\) are analytic functions of \(x_n\) of parity \(\varepsilon\) and are expressed in terms of Legendre functions as follows. Continuous series:

\[ \Phi_{\rho,\varepsilon}(x)= \]

\[ = \begin{cases} c_\sigma (x_n^2-1)^{-\nu/2} \left\{ \left(\sin(p/2+\nu+\sigma)\pi+\sin(p/2-\nu-\varepsilon)\pi\right)P_{\nu+\sigma}^{\nu}(x_n) \right.\\ \left. \qquad\qquad -\dfrac{2}{\pi}\sin\dfrac{p\pi}{2}\sin(2\nu+\sigma)\pi\cdot Q_{\nu+\sigma}^{\nu}(x_n) \right\}, & x_n>1,\\[1.2em] c_\sigma (1-x_n^2)^{-\nu/2}\sin\dfrac{p\pi}{2} \left\{P_{\nu+\sigma}^{\nu}(-x_n)+(-1)^\varepsilon P_{\nu+\sigma}^{\nu}(x_n)\right\}, & |x_n|<1, \end{cases} \]

where

\[ c_\sigma=2^{\nu+1}\pi^\nu\Gamma(\sigma+1)\Gamma(-2\nu-\sigma),\qquad \nu=(n-3)/2. \]

The first discrete series for \(p=1,\ q=1\):

\[ \Psi_l(x)= \begin{cases} 0, & |x_n|>1,\\[4pt] b_l\,\dfrac{\pi}{2}\,(-1)^{(q+1)/2}(1-x_n^2)^{-\nu/2}P_{l+\nu}^{\nu}(x_n), & |x_n|<1; \end{cases} \]

for \(p=1,\ q=0\):

\[ \Psi_l(x)= \begin{cases} b_l(-1)^{(q-2)/2}(x_n^2-1)^{-\nu/2}e^{-i\pi\nu}Q_{l+\nu}^{\nu}(x_n), & x_n>1,\\[4pt] b_l(-1)^{(q-2)/2}(1-x_n^2)^{-\nu/2}Q_{l+\nu}^{\nu}(x_n), & |x_n|<1, \end{cases} \]

where

\[ b_l=2^{-\nu-1}\pi^{-\nu-2}(2l+n-2)/a_l. \]

Received
21 I 1968

REFERENCES

\(^{1}\) I. M. Gel'fand, M. I. Graev, Trans. Moscow Math. Soc., 11, 243 (1962).
\(^{2}\) N. Ya. Vilenkin, Special Functions and the Theory of Group Representations, “Nauka,” 1965.
\(^{3}\) V. F. Molchanov, Dokl. Akad. Nauk SSSR, 171, No. 4, 794 (1966).
\(^{4}\) T. Shintani, Proc. Japan Acad., 43, No. 1, 1 (1967).