Basic Shannon measures

The information on this page is drawn from [3].

Entropy

The entropy measures how much information is in a random variable \(X\).

\[\H[X] = - \sum_{x \in X} p(x) \log_2 p(x)\]
entropy(dist, rvs=None, rv_names=None)[source]

Returns the entropy H[X] over the random variables in rvs.

If the distribution represents linear probabilities, then the entropy is calculated with units of ‘bits’ (base-2).

Parameters :
  • dist (Distribution or float) – The distribution from which the entropy is calculated. If a float, then we calculate the binary entropy.
  • rvs (list, None) – The indexes of the random variable used to calculate the entropy. If None, then the entropy is calculated over all random variables. This should remain None for ScalarDistributions.
  • rv_names (bool) – If True, then the elements of rvs are treated as random variable names. If False, then the elements of rvs are treated as random variable indexes. If None, then the value True is used if the distribution has specified names for its random variables.
Returns:

H (float) – The entropy of the distribution.

Conditional Entropy

The conditional entropy is the amount of information in variable \(X\) beyond that which is in variable \(Y\).

\[\H[X|Y] = \sum_{x \in X, y \in Y} p(x, y) \log_2 p(x|y)\]
conditional_entropy(dist, rvs_X, rvs_Y, rv_names=None)[source]

Returns the conditional entropy of H[X|Y].

If the distribution represents linear probabilities, then the entropy is calculated with units of ‘bits’ (base-2).

Parameters :
  • dist (Distribution) – The distribution from which the conditional entropy is calculated.
  • rvs_X (list, None) – The indexes of the random variables defining X.
  • rvs_Y (list, None) – The indexes of the random variables defining Y.
  • rv_names (bool) – If True, then the elements of rvs_X and rvs_Y are treated as random variable names. If False, then their elements are treated as random variable indexes. If None, then the value True is used if the distribution has specified names for its random variables.
Returns:

H_XgY (float) – The conditional entropy H[X|Y].

Mutual Information

The mutual information is the amount of information shared by \(X\) and \(Y\).

\[\begin{split}\I[X:Y] &= \H[X,Y] - \H[X|Y] - \H[Y|X] \\ &= \H[X] + \H[Y] - \H[X,Y] \\ &= \sum_{x \in X, y \in Y} p(x, y) \log_2 \frac{p(x, y)}{p(x)p(y)}\end{split}\]
mutual_information(dist, rvs_X, rvs_Y, rv_names=None)[source]

Returns the mutual information I[X:Y].

If the distribution represents linear probabilities, then the entropy is calculated with units of ‘bits’ (base-2).

Parameters :
  • dist (Distribution) – The distribution from which the mutual information is calculated.
  • rvs_X (list, None) – The indexes of the random variables defining X.
  • rvs_Y (list, None) – The indexes of the random variables defining Y.
  • rv_names (bool) – If True, then the elements of rvs_X and rvs_Y are treated as random variable names. If False, then their elements are treated as random variable indexes. If None, then the value True is used if the distribution has specified names for its random variables.
Returns:

I (float) – The mutual information I[X:Y].

Read the Docs v: dev
Versions
latest
dev
Downloads
PDF
HTML
Epub
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.