Full Text
Reports of the Academy of Sciences of the USSR
1963. Volume 149, No. 6
PHYSICS
D. S. LEBEDEV, L. B. LEVITIN
THE MAXIMUM AMOUNT OF INFORMATION TRANSMISSIBLE BY AN ELECTROMAGNETIC FIELD
(Presented by Academician A. N. Kolmogorov on 22 XI 1962)
The limitations imposed by the statistical character of physical processes and by the quantum nature of matter make it fundamentally impossible to transmit an arbitrarily large amount of information by means of a limited amount of energy. It is of interest to ask what is the minimum energy required, under given conditions, for the transmission of a unit amount of information, or what is the maximum throughput for a given average signal power. Below we briefly set forth results concerning the electromagnetic field as a carrier of information.
The electromagnetic communication channel was first studied from the physical point of view in the works of Gabor \((^{1,2})\), who approximately estimated the minimum energy required for the transmission of one binary unit of information. L. Brillouin \((^3)\) showed in the general case that the minimum energy required for the transmission of one natural unit (nit) of information is not less than \(kT\). Some considerations were expressed in the article by G. I. Rukman and Kh. M. Khaplanov \((^4)\). Stern \((^{5,6})\), using Gabor’s method of time-frequency cells, established the properties of a photon source of maximum entropy (in the absence of thermal noise) and obtained a number of other interesting results. Gordon \((^7)\), applying the frequency representation of one-dimensional signal and noise, derived an expression for the throughput capacity of a narrow-band channel and considered information losses in certain amplifying and receiving devices.
We shall consider an information-transmission channel formed by a transmitter emitting an electromagnetic signal and a receiver absorbing the signal with a given average power \(P\) and being in equilibrium with thermal radiation of temperature \(T\) (which is equivalent to the addition of additive noise). The signal is assumed to be one-dimensional, i.e. consisting of photons with the same state of polarization and the same directions of the wave vectors. It is completely described by occupation numbers over energy levels. We shall assume that the transmitter can unambiguously set the microstate (occupation numbers) of the signal. The collection of various microstates forms an ensemble of signals. The receiver—an ideal analyzer, carrying out an exact spectral decomposition of the signal (in accordance with the Heisenberg relation, over infinite time)—is a set of oscillators with negligibly small natural line width. Thus we disregard additional restrictions connected with the physical nature of the transmitter and receiver.
Let us first suppose that the signal has a discrete energy spectrum \(h\nu_i = hi/\tau\) \((i = 0, 1, 2, \ldots)\), where \(\tau\) is the period of the signal. The receiver—a set of oscillators with natural frequencies \(\nu_i\)—in the absence of a signal at temperature \(T\), in accordance with Planck’s formula, has energy
\[ E_1 = \sum_{i=0}^{\infty} \frac{h\nu_i}{\exp(h\nu_i/kT) - 1}. \tag{1} \]
The zero-point energy of the oscillators has been omitted, since it does not affect the value of the entropy.
The entropy of such a system is equal to
\[ H_1=\int_0^{E_1}\frac{1}{kT'}\,dE =\int_0^T \frac{1}{kT'}\frac{dE}{dT'}\,dT', \]
where \(E(T')\) is the energy of the system as a function of temperature, \(E(T)=E_1\). (Here and below the entropy is measured in dimensionless units—nats, differing from the quantity adopted in thermodynamics by the factor \(1/k\).)
The signal transfers to the receiver the energy \(P\tau\) over the period. In this process the entropy of the receiver remains the same, since the signal has zero entropy (its microstate is completely determined). According to L. Brillouin’s negentropic principle \({}^{(3)}\), the maximum amount of information \(I\) that can be transmitted by the signal to the receiver is equal to the “entropy deficit,” i.e., to the difference between the maximum entropy that a system of oscillators with energy
\[ E_2=E_1+P\tau, \tag{2} \]
could have, and the actual entropy of the system. But the maximum entropy \(H_2\) would be attained at thermodynamic equilibrium corresponding to some temperature \(T_{\mathrm{eff}}\):
\[ H_2=\int_0^{T_{\mathrm{eff}}}\frac{1}{kT'}\frac{dE}{dT'}\,dT', \]
and, consequently,
\[ I=H_2-H_1=\int_T^{T_{\mathrm{eff}}}\frac{1}{kT'}\frac{dE}{dT'}\,dT'. \]
Hence the channel capacity \(C\), i.e., the maximum amount of information carried by the signal per unit time, is
\[ C=\frac{1}{\tau}\int_T^{T_{\mathrm{eff}}}\frac{1}{kT'}\frac{dE}{dT'}\,dT' =\int_T^{T_{\mathrm{eff}}}\frac{1}{kT'}\frac{dP(T')}{dT'}\,dT', \tag{3} \]
where
\[ P(T')=E(T')/\tau. \tag{4} \]
Passing to a signal with a continuous spectrum and putting \(1/\tau \to d\nu\), we obtain from (1):
\[ P(T')=\int_0^\infty \frac{h\nu}{\exp(h\nu/kT')-1}\,d\nu =\frac{\pi^2(kT')^2}{6h}. \tag{5} \]
Substituting (5) into (3), we find:
\[ C=\frac{\pi^2 k}{3h}\,(T_{\mathrm{eff}}-T). \tag{6} \]
From (2) and (4) it follows that \(P(T_{\mathrm{eff}})=P(T)+P\), whence, using (5), one can find \(T_{\mathrm{eff}}\): \(T_{\mathrm{eff}}=T\sqrt{1+6hP/\pi^2(kT)^2}\). Eliminating \(T_{\mathrm{eff}}\) from (6), we obtain an expression for the capacity of an electromagnetic channel at mean signal power \(P\) and thermal-noise temperature \(T\):
\[ C=\frac{\pi^2 kT}{3h}\left(\sqrt{1+\frac{6hP}{\pi^2(kT)^2}}-1\right). \tag{7} \]
The minimum energy \(E_{\min}=P/C\) necessary for the transmission of one nit at rate \(C\) is
\[ E_{\min}=kT+\frac{3}{2\pi^2}hC. \tag{8} \]
For a small ratio of signal power to noise power, \(6hP/\pi^2(kT)^2 \ll 1\), expression (7) gives
\[ C_0=P/kT, \tag{9} \]
which is Shannon’s formula \({}^{(8)}\) for a broadband signal with additive Gaussian noise.
For infinitely slow transmission of information, when \(hC/kT \to 0\), formula (8) becomes Brillouin’s formula
\[ E_0=kT. \tag{10} \]
Thus, for weak signals (or, what is the same thing, for low rates of information transmission), formulas (7) and (8) give the classical limiting values (independent of \(\hbar\)).
In the opposite limiting case, when \(6hP/\pi^2(kT)^2 \gg 1\), the capacity is limited only by quantum effects,
\[ C=\pi\sqrt{2P/3h}, \tag{11} \]
which (to within a factor \(\sqrt{2}\)) coincides with the result following from Stern’s work \({}^{(5)}\).
Let us now consider a narrowband channel, when all the signal power is concentrated in a band \(\Delta\nu\), narrow in comparison with the central frequency \(\nu\): \(\Delta\nu/\nu \ll 1\). In this case one may write
\[ P(T')=\Delta\nu\,\varepsilon(\nu,T'), \tag{12} \]
where
\[ \varepsilon(\nu,T')=\frac{h\nu}{\exp(h\nu/kT')-1} \tag{13} \]
is the energy of an oscillator of frequency \(\nu\) at temperature \(T'\). Analogously to (2), \(\varepsilon_2=\varepsilon_1+P_{\mathrm{sp}}\), where \(P_{\mathrm{sp}}=P/\Delta\nu\) is the spectral intensity of the signal, \(\varepsilon_1=\varepsilon(\nu,T)\).
We shall calculate the capacity according to formula (3), substituting (12) and passing to integration over \(\varepsilon\):
\[ C=\Delta\nu\int_{\varepsilon_1}^{\varepsilon_2}\frac{1}{kT'(\varepsilon)}\,d\varepsilon, \]
where \(T'(\varepsilon)\) is determined from (13). As a result we obtain
\[ \begin{aligned} C(\nu)=\Delta\nu\Bigg\{& \ln\left[1+\frac{P_{\mathrm{sp}}}{h\nu}\left(1-\exp(-h\nu/kT)\right)\right] \\ &+\left[\frac{P_{\mathrm{sp}}}{h\nu}+\frac{1}{\exp(h\nu/kT)-1}\right] \ln\left[1+\frac{h\nu\left(\exp(h\nu/kT)-1\right)} {P_{\mathrm{sp}}\left(\exp(h\nu/kT-1)\right)+h\nu}\right] -\frac{h\nu/kT}{\exp(h\nu/kT)-1} \Bigg\}. \end{aligned} \tag{14} \]
Formula (14) essentially coincides with expression (5) from Gordon’s paper \({}^{(7)}\), obtained by another method. For fixed \(\Delta\nu\) and \(P_{\mathrm{sp}}\), \(C(\nu)\) is a monotonically decreasing function of \(\nu\).
In the classical limit \((h\nu/kT \ll 1)\) we obtain the well-known Shannon formula \({}^{(8)}\)
\[ C=\Delta\nu\ln(1+P_{\mathrm{sp}}/kT) \tag{15} \]
(to within terms of order \((h\nu/kT)^2\)).
The quantum limiting case \((h\nu/kT \gg 1,\; P_{\mathrm{sp}}[\exp(h\nu/kT)-1]\,h/\nu \gg 1)\) gives
\[ C(\nu)=\Delta\nu\{\ln(1+P_{\mathrm{sp}}/h\nu)+(P_{\mathrm{sp}}/h\nu)\ln(1+h\nu/P_{\mathrm{sp}})\}. \tag{16} \]
We have obtained expressions for the capacity of photon channels, proceeding from thermodynamic considerations and using Brillouin’s principle. The same results can also be obtained by the usual methods of information theory. Let the occupation numbers at frequency \(\nu_i=i/\tau\) be denoted by \(m_i\) for the signal, \(n_i\) for the noise, and \(l_i\) for the received signal (the sum of signal and noise), \(l_i=m_i+n_i\) \((l_i,m_i,n_i=0,1,2,\ldots)\), and let the corresponding probability distributions be \(q(m_i)\), \(p(n_i)\), and \(s(l_i)\). The signal is subject to a constraint of the form
\[ \frac{1}{\tau}\sum_{i=0}^{\infty}\sum_{m_i=0}^{\infty} m_i q(m_i)h\nu_i=P . \]
The photon distribution of thermal noise \(p(n_i)\) is known—it is the Gibbs distribution:
\[ p(n_i)=[1-\exp(-h\nu_i/kT)]\exp(-n_i h\nu_i/kT). \]
To determine the capacity, one must find such a distribution of the occupation numbers of the received signal \(s(l_i)\) (as a function of \(l_i\) and the frequency \(\nu_i\)) for which the amount of information in the numbers \(l_i\), relative to the occupation numbers of the signal \(m_i\), is maximal. Solving the variational problem, one can show that, as was to be expected from thermodynamic considerations, the maximum amount of information is also attained for the Gibbs distribution:
\[ s(l_i)=[1-\exp(-h\nu_i/kT_{\mathrm{eff}})]\exp(-l_i h\nu_i/kT_{\mathrm{eff}}). \]
Knowing \(p(n_i)\) and \(s(l_i)\), it is easy to find the distribution of the signal occupation numbers \(q(m_i)\), which is no longer Gibbsian. Thus the signal differs from the noise in its statistical character.
Passing to the limit as \(\tau\to\infty\) and determining \(T_{\mathrm{eff}}\) from the condition of additivity of the signal and noise powers \(P_2=P_1+P\), where
\[ P_2=\int_{0}^{\infty}\sum_{l=0}^{\infty} l s(l)h\nu\,d\nu \]
is the power of the received signal,
\[ P_1=\int_{0}^{\infty}\sum_{n=0}^{\infty} n p(n)h\nu\,d\nu \]
is the noise power, we obtain for the capacity the previous expression (7).
Institute for Problems of Information Transmission
Academy of Sciences of the USSR
Received
22 XI 1962
CITED LITERATURE
- D. Gabor, Phil. Mag., 41, 7, 1161 (1950).
- D. Gabor, Arch. Elektr. Übertragung, 1, 95 (1953).
- L. Brillouin, Science and Information Theory, Moscow, 1960.
- G. I. Rukman, Kh. M. Khaplanov, Problems of Radio Electronics, ser. 1, issue 6, 3 (1959).
- T. E. Stern, Trans. IRE IT-6, 4, 435 (1960).
- T. E. Stern, IRE International Convention Record, pt. 4, 182 (1960).
- D. P. Gordon, Proc. IRE, 50, 9, 1898 (1962).
- K. Shannon, in: Information Theory and Its Applications, Moscow, 1959.