INTRODUCTION TO INFORMATION THEORY

1 INTRODUCTION TO INFORMATION THEORY {ch:intro_info}This chapter introduces some of the basic concepts of INFORMATION THEORY , as wellas the definitions and notations of probabilities that will be used throughoutthe book. The notion of entropy, which is fundamental to the whole topic ofthis book, is introduced here. We also present the main questions of informationtheory, data compression and error correction, and state Shannon s Random variablesThe main object of this book will be the behavior of large sets ofdiscreterandom variables. A discrete random variableXis completely defined1bythe set of values it can take,X, which we assume to be a finite set, and itsprobability distribution{pX(x)}x X. The valuepX(x) is the probability thatthe random variableXtakes the valuex. The probability distributionpX:X [0,1] must satisfy the normalization conditionXx XpX(x) = 1.

where the second form uses the indicator function I(s) of a logical statement s,which is defined to be equal to 1 if the statement sis true, and equal to 0 if the statement is false. The expectation value of a real valued function f(x) is given by the integral on X: Ef(X) = Z f(x) dpX(x) . (1.4)

Loading..

Tags:

  Second, The second

Information

Related search queries