Transcription of 2.4.8 Kullback-Leibler Divergence
{{id}} {{{paragraph}}}
Example: tourism industry
2.4.8 Kullback-Leibler Divergence
Kullback-Leibler DivergenceTo measure the difference between two probability distributions over the samevariablex, a measure, called theKullback- leibler Divergence , or simply, theKLdivergence, has been popularly used in the data mining literature. The conceptwas originated in probability theory and information KL Divergence , which is closely related torelative entropy,informa-tion Divergence , andinformation for discrimination, is a non-symmetric mea-sure of the difference between two probability distributionsp(x) andq(x).Specifically, the Kullback-Leibler (KL) Divergence ofq(x) fromp(x), denotedDKL(p(x), q(x)), is a measure of the information lost whenq(x) is used to ap-proximatep(x).Letp(x) andq(x) are two probability distributions of a discrete randomvariablex. That is, bothp(x) andq(x) sum up to 1, andp(x)>0 andq(x)>0for (p(x), q(x)) is defined in Equation ( ).DKL(p(x)||q(x)) = x Xp(x) lnp(x)q(x)( )The KL Divergence measures the expected number of extra bits required tocode samples fromp(x) when using a code based onq(x), rather than using acode based onp(x).
from an observed frequency distribution, as illustrate in the following example. Example 2.24. Computing the KL Divergence by Smoothing. Sup-pose there are two sample distributions P and Q as follows: P: (a: 3/5,b:
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: