Transcription of INTRODUCTION TO INFORMATION THEORY
{{id}} {{{paragraph}}}
Example: bachelor of science
INTRODUCTION TO INFORMATION THEORY
1 INTRODUCTION TO INFORMATION THEORY {ch:intro_info}This chapter introduces some of the basic concepts of INFORMATION THEORY , as wellas the definitions and notations of probabilities that will be used throughoutthe book. The notion of entropy, which is fundamental to the whole topic ofthis book, is introduced here. We also present the main questions of informationtheory, data compression and error correction, and state Shannon s Random variablesThe main object of this book will be the behavior of large sets ofdiscreterandom variables. A discrete random variableXis completely defined1bythe set of values it can take,X, which we assume to be a finite set, and itsprobability distribution{pX(x)}x X. The valuepX(x) is the probability thatthe random variableXtakes the valuex. The probability distributionpX:X [0,1] must satisfy the normalization conditionXx XpX(x) = 1.
measure is roughly speaking the logarithm of the number of typical values that the variable can take, as the following examples show. Example 1.3 A fair coin has two values with equal probability. Its entropy is 1 bit. Example 1.4 Imagine throwing Mfair coins: the number of all possible out-comes is 2M. The entropy equals Mbits.
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: