# Kullback-Leibler divergence of mixtures/Differential Entropy of mixtures

**Guaranteed bounds on the Kullback-Leibler divergence of univariate
mixtures using piecewise log-sum-exp inequalities** (arxiv 1606.05850)

by Frank Nielsen and Ke Sun

Information-theoretic measures such as the entropy, cross-entropy and the KullbackLeibler
divergence between two mixture models is a core primitive in many signal processing
tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a
closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic
integration, approximated, or bounded using various techniques. We present a fast and
generic method that builds algorithmically closed-form lower and upper bounds on the entropy,
the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the
versatile method by reporting on our experiments for approximating the Kullback-Leibler
divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures,
and Gamma mixtures.

## Python source code

Download the Python code for reproducible research:
klmm.tar.gz

## Differential Entropy

For the Shannon entropy, the shell command produces the following pdf results:

entropy_gmm1.pdf
entropy_gmm2.pdf
entropy_gmm3.pdf
entropy_gmm4.pdf
entropy_gmm5.pdf
entropy_gmm6.pdf

## Kullback-Leibler divergence of mixtures

For the Kullback-Leibler divergence, the shell command produces the following pdf results:

kl_gmm1_gmm2.pdf
kl_rmm1_rmm2.pdf
kl_emm1_emm2.pdf
kl_gamm1_gamm2.pdf

Combinatorial descriptions of the lower and upper envelopes are written in text files *.envelope.

For example, for the GMM1 data set file `gmm1`

(positive weights alpha are renormalized to one by the software) :
# alpha mu sigma
0.5 -5 1
1 -2 0.5
2 5 0.3
2 10 0.5
0.5 15 0.4
3 25 0.5
1 30 2

We get the combinatorial support intervals followed by the indices of the component for the upper and lower envelopes as below
(file `gmm1.envelope`

):
# start end upper lower
-inf -40.000000 0 6
-40.000000 -25.020076 1 6
-25.020076 -24.968606 1 5
-24.968606 -3.219053 1 4
-3.219053 1.219053 2 4
1.219053 2.615495 1 4
2.615495 6.890300 6 4
6.890300 11.489828 3 4
11.489828 12.824402 3 2
12.824402 15.525749 5 2
15.525749 17.548493 5 6
17.548493 22.905116 0 6
22.905116 26.428217 4 6
26.428217 inf 0 6

Last updated, August 2016, v2 arxiv.