Basic Shannon measures
The information on this page is drawn from [3].
Entropy
The entropy measures how much information is in a random variable \(X\).
\[\H[X] = - \sum_{x \in X} p(x) \log_2 p(x)\]
-
entropy(dist, rvs=None, rv_names=None)[source]
Returns the entropy H[X] over the random variables in rvs.
If the distribution represents linear probabilities, then the entropy
is calculated with units of ‘bits’ (base-2).
Parameters : |
- dist (Distribution or float) –
The distribution from which the entropy is calculated. If a float,
then we calculate the binary entropy.
- rvs (list, None) –
The indexes of the random variable used to calculate the entropy.
If None, then the entropy is calculated over all random variables.
This should remain None for ScalarDistributions.
- rv_names (bool) –
If True, then the elements of rvs are treated as random variable
names. If False, then the elements of rvs are treated as random
variable indexes. If None, then the value True is used if the
distribution has specified names for its random variables.
|
Returns: | H (float) –
The entropy of the distribution.
|
Conditional Entropy
The conditional entropy is the amount of information in variable \(X\)
beyond that which is in variable \(Y\).
\[\H[X|Y] = \sum_{x \in X, y \in Y} p(x, y) \log_2 p(x|y)\]
-
conditional_entropy(dist, rvs_X, rvs_Y, rv_names=None)[source]
Returns the conditional entropy of H[X|Y].
If the distribution represents linear probabilities, then the entropy
is calculated with units of ‘bits’ (base-2).
Parameters : |
- dist (Distribution) –
The distribution from which the conditional entropy is calculated.
- rvs_X (list, None) –
The indexes of the random variables defining X.
- rvs_Y (list, None) –
The indexes of the random variables defining Y.
- rv_names (bool) –
If True, then the elements of rvs_X and rvs_Y are treated as
random variable names. If False, then their elements are treated as
random variable indexes. If None, then the value True is used if
the distribution has specified names for its random variables.
|
Returns: | H_XgY (float) –
The conditional entropy H[X|Y].
|