A. A. KIRILLOV

UNITARY REPRESENTATIONS OF NILPOTENT LIE GROUPS

(Presented by Academician I. G. Petrovskii on 26 XII 1960)

In the theory of unitary representations of Lie groups the following problems are usually considered: the description of the set of irreducible representations, the explicit construction of representations, the computation of characters, the decomposition into irreducible components of representations realized in functions on homogeneous manifolds, and the decomposition of irreducible representations of a group into irreducible representations of a subgroup. In the present note these problems are solved for connected nilpotent Lie groups. Partial results for this case were obtained by Dixmier ((^1)) and by the author ((^2)).

Let (\mathfrak G) be a connected simply connected Lie group, (G) its Lie algebra, and (G') the space of real linear functionals on (G). In the space (G) acts the adjoint representation (\rho) of the group (\mathfrak G). Denote by (\rho') the dual representation of (\mathfrak G) in the space (G').

We shall call a subalgebra (G_0 \subset G) subordinate to a functional (f \in G') if ((f,[g_1,g_2])=0) for all (g_1,g_2 \in G_0). It is easy to see that if the subalgebra (G_0) is subordinate to the functional (f), then the formula

[
\exp g \to e^{i(f,g)}
\tag{*}
]

defines a one-dimensional unitary representation of the corresponding subgroup. ((\exp) is the canonical mapping of the algebra onto the group.)

Theorem 1. Let (f) be an arbitrary functional on (G); (G_0) a subalgebra of maximal dimension subordinate to (f).

A. The representation (T) of the group (\mathfrak G), induced by the one-dimensional representation ((*)) of the subgroup (\mathfrak G_0=\exp G_0), is irreducible and does not depend on the choice of (G_0) (more precisely, the representations corresponding to the same functional (f) and to different subalgebras (G_0) are equivalent).

B. The representations (T_1) and (T_2), corresponding to functionals (f_1) and (f_2), are equivalent if and only if (f_1) and (f_2) belong to the same orbit in (G') with respect to (\rho').

C. Every irreducible unitary representation of (\mathfrak G) corresponds to some functional (f \in G').

This theorem can be proved by induction on the dimension of the group, similarly to the way in which the main theorem was proved in ((^{2a})).

Consider the set (X(G')) of orbits in (G') with respect to (\rho'). Theorem 1 establishes a one-to-one correspondence between the set (X(G')) and the set (\widehat{\mathfrak G}) of all irreducible unitary representations of (\mathfrak G), considered up to equivalence. Both of these sets may be regarded as topological spaces. The topology in (X(G')) is defined as the strongest one in which the natural mapping (G' \to X(G')) is continuous. The topology in (\widehat{\mathfrak G}) may be specified by the condition: the representations (T^{(n)}) converge to the representation (T) if, for every vector (x) from the space of the representation (T), there exist vectors (x_n) from the spaces of the representations (T^{(n)}) such that (f_n(g)=(T_g^{(n)}x_n,x_n)) converge to (f(g)=(T_gx,x)) uniformly on every compact set in (\mathfrak G) ((^3)). There exist other definitions leading to the same topology in (\widehat{\mathfrak G}) ((^{4,5})).

Theorem 2. The mapping (X(G')) into (\widehat{\mathfrak G}), defined by Theorem 1, is continuous.

As was proved in (1ш, 2б), for every basic (i.e., infinitely differentiable and rapidly decreasing) function (\varphi) on (\mathfrak G) and any irreducible unitary representation (T) of the group (\mathfrak G), the operator
[
T_\varphi=\int \varphi(g)T_g\,dg
]
has a trace. The generalized function (\chi) on (\mathfrak G), defined by the formula ((\chi,\varphi)=\operatorname{sp} T_\varphi), is called the character of the representation (T). For the computation of characters it will be more convenient for us, instead of a function (\varphi) on (\mathfrak G), to consider the function (\psi(g)=\varphi(\exp g)) on the algebra (G). Obviously, every basic function (\varphi) on (\mathfrak G) corresponds to a basic function (\psi) on (G), and conversely. In the space of functions on (G) one can define the Fourier transform, which takes a basic function (\psi(g)) into the basic function
[
\widetilde\psi(f)=\int e^{i(f,g)}\psi(g)\,dg
]
on (G'). To the Fourier transform of basic functions there corresponds, as usual, the Fourier transform of generalized functions.

Theorem 3. If (\chi) is the character of an irreducible unitary representation of the group (\mathfrak G), corresponding by virtue of Theorem 1 to a certain orbit (\Omega) in (G'), then its Fourier transform (\widetilde\chi) coincides with the (\delta)-function concentrated on (\Omega). (More precisely, on (\Omega) there exists a measure (d\omega), invariant with respect to (\rho'), such that for any basic function (h) on (G')
[
(\widetilde\chi,h)=\int_\Delta h(\omega)\,d\omega.)
]

The explicit construction of representations given in Theorem 1 makes it possible to clarify the relation between representations of the group and representations of a subgroup of codimension 1. With the help of induction on the codimension of the subgroup, the results obtained can be transferred to arbitrary connected closed subgroups. This leads to the solution of the problems mentioned in the introduction. We shall formulate only the final results. Let (\mathfrak G_0) be a subgroup of (\mathfrak G). Then there is defined a natural mapping
[
p:G'\to G_0'.
]

Theorem 4. An irreducible representation of the group (\mathfrak G), corresponding to an orbit (\Omega\subset G'), decomposes into a direct integral (or direct sum) of irreducible representations of the subgroup (\mathfrak G_0), corresponding to those orbits in (G_0') into which (p(\Omega)) splits.

Theorem 5. Let an irreducible representation (U) of the subgroup (\mathfrak G_0) correspond to an orbit (\Omega_0\subset G_0'). Then the representation of (\mathfrak G) induced by the representation (U) decomposes into a direct integral (direct sum) of irreducible representations of (\mathfrak G), corresponding to those orbits (\Omega) in (G') for which (p(\Omega)) contains (\Omega_0).

In particular, the representation realized in functions on (\mathfrak G/\mathfrak G_0) decomposes into representations corresponding to those orbits (\Omega) for which (p(\Omega)) contains the zero functional.

Theorems 1–5 are transferred to non-simply connected groups in the following way. If (\mathfrak G) is not simply connected, then under the canonical mapping (\exp:G\to\mathfrak G) an integral lattice (R\subset G) goes into the identity of the group (\mathfrak G). Denote by (\widetilde G) the abelian group (G/R) and by (\widetilde G') the group dual to it. Then Theorems 1–6 are valid for non-simply connected groups if one replaces (G,G_0,G',G_0') by (\widetilde G,\widetilde G_0,\widetilde G',\widetilde G_0'), respectively.

Let us also note that Theorems 1, 2, 4, 5 carry over to certain solvable groups, in particular to triangular matrix groups.

The author expresses his sincere gratitude to I. M. Gelfand for his constant attention and valuable advice.

Moscow State University
named after M. V. Lomonosov

Received
20 XII 1960

REFERENCES

1. J. Dixmier, a) Bull. Soc. Math. France, 85, 325 (1957); b) Canad. J. Math. 10, 321 (1958); c) Canad. J. Math., 11, 321 (1959); d) Am. J. Math., 81, 160 (1959); e) Bull. Soc. Math. France, 87, 65 (1959); f) Canad. J. Math., 12, No. 2 (1960).
2. A. A. Kirillov, a) DAN, 128, No. 5 (1959); b) 130, No. 5 (1960).
3. R. Godement, Trans. Am. Math. Soc., 63, 1 (1948).
4. J. M. G. Fell, Illinois J. Math., 4, No. 2 (1960).
5. D. P. Zhelobenko, DAN, 126, No. 2 (1959).