Transcription of 2.4.8 Kullback-Leibler Divergence
{{id}} {{{paragraph}}}
Example: bankruptcy
2.4.8 Kullback-Leibler Divergence
Kullback-Leibler DivergenceTo measure the difference between two probability distributions over the samevariablex, a measure, called theKullback- leibler Divergence , or simply, theKLdivergence, has been popularly used in the data mining literature. The conceptwas originated in probability theory and information KL Divergence , which is closely related torelative entropy,informa-tion Divergence , andinformation for discrimination, is a non-symmetric mea-sure of the difference between two probability distributionsp(x) andq(x).Specifically, the Kullback-Leibler (KL) Divergence ofq(x) fromp(x), denotedDKL(p(x), q(x)), is a measure of the information lost whenq(x) is used to ap-proximatep(x).Letp(x) andq(x) are two probability distributions of a discrete randomvariablex. That is, bothp(x) andq(x) sum up to 1, andp(x)>0 andq(x)>0for (p(x), q(x)) is defined in Equation ( ).DKL(p(x)||q(x)) = x Xp(x) lnp(x)q(x)( )The KL Divergence measures the expected number of extra bits required tocode samples fromp(x) when using a code based onq(x), rather than using acode based onp(x).
The continuous version of the KL divergence is DKL(p(x)||q(x)) = ∫ ∞ −∞ p(x)ln p(x) q(x) dx (2.2) Although the KL divergence measures the “distance” between two distri-butions, it is not a distance measure. This is because that the KL divergence is not a metric measure. It is not symmetric: the KL from p(x) to q(x) is
Loading..
Tags:
Information
Domain:
Source:
Link to this page:
Please notify us if you found a problem with this document: