LECTURES ON COMMUNICATION THEORY - DSpace@MIT: …

1 20A-118 LECTURES ON COMMUNICATION THEORYD. GABORTECHNICAL REPORT NO. 238 APRIL 3, 1952 RESEARCH LABORATORY OF ELECTRONICSMASSACHUSETTS INSTITUTE OF TECHNOLOGYCAMBRIDGE, MASSACHUSETTS ___i __1 IIYPIIYUYL -LI_- 1 IIIDI--C- 1 _ I11I 1 - I ---- 111 MASSACHUSETTS INSTITUTE OF TECHNOLOGYRESEARCH LABORATORY OF ELECTRONICST echnical Report No. 238 April 3, 1952 LECTURES on COMMUNICATION TheoryD. Gaborof the Imperial College of Science and Technology, LondonThis report presents a series of LECTURES that weregiven under the sponsorship of the ResearchLaboratory of Electronics during the Fall Term,1951, at Massachusetts Institute of TechnologyAbstractThese LECTURES on selected chapters of COMMUNICATION THEORY are comple-mentary to the well-known works of American authors on the statistical theoryof COMMUNICATION , which is not discussed here at any length.

2 About one-thirdof the LECTURES have as their subject the THEORY of signal analysis or represen-tation, which precedes the statistical THEORY , both logically and mathematical THEORY is followed by a physical THEORY of signals, in whichthe fundamental limitations of signal transmission and recognition are discussedin the light of classical and of quantum physics. It is shown that the viewpointsof COMMUNICATION THEORY represent a useful approach to modern physics, of appre-ciable heuristic power, showing up the insufficiencies of the classical final part of the LECTURES is a report on the present state of speech analysisand speech compression, with suggestions for further __IIL II1__ _ _ ,7 -- LECTURES ON COMMUNICATION THEORYI.

3 What is Information? COMMUNICATION THEORY owes its origin to a few theoretically interested engineers whowanted to understand the nature of the goods sold in COMMUNICATION systems. The general answerhas, of course, been known for a long time. COMMUNICATION systems sell information capacity, aspower systems sell energy. The way from this general idea to the quantitative definition of theconcept of information was a long one, and we are not by any means at its first step in dispersing the cloud of vagueness which hangs around the concept of in-formation is the realization that information, if it is to be communicable at all, must be of a discretenature.

4 It must be expressible by the letters of the alphabet, adding to these, if necessary, mathe-matical symbols. In general, we must make use of an agreed language, in which the process ofreducing our chaotic sensations to a finite number of elements has led to some sort of this vocabulary came to exist, how it is enriched from day to day by new concepts which crys-tallize in the form of a word, and how it is learned by children- these are dramatically interestingquestions, but outside the range of COMMUNICATION we have a vocabulary, COMMUNICATION becomes a process of selection.

5 A selectioncan always be carried out by simple binary selections, by a series of yeses or noes. For instance,if we want a letter in the 32-letter alphabet, we first answer the question "is it or is it not in theupper half?" By five such questions and answers we have fixed a letter. Writing 1 for a "yes"and 0 for a "no", the letter can be expressed by a symbol such as 01001, where the first digit isthe answer to the first question, and so on. This symbol also expresses the order number of theletter (in this example, the number 9) in a binary the same method we can also communicate quantities.

6 Physical quantities can be meas-ured only with finite accuracy, and if we take as the unit of our measure the smallest interval whichwe can assert with about 50 percent confidence that the quantity in question is inside it, we canwrite the result ..010011 +1. Alternatively we can also use a "decimal" (really "binary") measured number must break off somewhere. It is true that there are numbers, such as 2, or7t, which do not break off, but in these cases the instruction to obtain them can be communicatedin a finite number of words or other symbols. Otherwise, they could never have been suggests immediately that the number of digits, that is, the number of yeses and noesby which a certain statement can be described, should be taken as a measure of the step was essentially taken by Hartley in 1928 (1).

7 It may be noted that if n such independentbinary selections are carried out, the result is equivalent to one selection of N = n = log2N, the informative value of a selection from N possibilities appears as the logarithmof N to the base 2. This is unity for one binary selection. The unit is called one "bit" or "binit",short for "binary digit".The word "possibilities" suggests an extension of this definition. The N selections areall possible, but are they also equally probable? Of course, the answer is that in general they arenot. It may be remembered that we are conversing in a certain language.

8 Whether this languageis as full of cliches as the lyrics of the music halls, or as full of surprises as the report on a racingday, there will always be certain features which we can predict with more or less certainty fromwhat has gone before. But what we knew before, we evidently cannot count as extension of the information concept to the case in which we have certain expectationsregarding the message has been made by N. Wiener and C. E. Shannon. They chose, for good reasons,-1-_ unexpectedness of a message as a measure of its informative value. Consider, for simplicity,a set of possible events which we think may happen, and to which we assign expectationvalues Pi.

9 In general, the Pi present a thorny problem (for instance, if they represent the probabil-ity of the horse i winning a race). They become simple only in the so-called "ergodic" case, in iwhich the next event is picked out of a homogeneous statistical series. In this case we take thepast experience as a guide and identify the probabilities Pi with the frequency of the occurrenceof the event i in the past, in similar that we know the probabilities Pi of the events i in such an ergodic series. Usingthe language of COMMUNICATION THEORY , let us consider these events as "symbols" delivered by anergodic "source".

10 By Shannon's definition, the expectation value of the information isNH =- E pilog2pi bits per symbol (1)an expression which is also called "the entropy of the source". This definition, in order to beacceptable, must satisfy certain postulates. The first is that in the case of equal probabilities itmust go over into Hartley's definition; that is, for Pi = 1/N we must have S = log2N, which is easilyverified. Likewise one can verify also that if the events in question are composite (consisting oftwo or more independent events), the S are additive. Shannon also shows that in whatever way theevents are broken down into component events, with their respective probabilities, the result is thesame.