Construct · 1 fields · 2 citations

mutual information: a standalone construct

One name. One construct. A landmark on the map.

● anchor · a standalone construct Nearest construct by meaninginformationcompare the meanings
How to read this

Every construct on the map is compared with every other by the meaning of its definitions, and the closest one is surfaced here. The nearest construct to mutual information is information.

Hold this as a prompt, not a verdict. Two constructs sitting close in meaning is a reason to read them side by side and decide for yourself whether they are one idea under two names. It is not proof that they are. Compare the meanings in the Collider.

mutualinformation1 CONSTRUCTMutual informationSHANNON · SHARED INFO

A single standalone construct; nearby constructs are located against it in the Collider. Tap a field for its definition, unit, and measures.

Information theory

The information shared between two random variables

Mutual information was introduced by Claude Shannon in his 1948 paper A Mathematical Theory of Communication, which founded information theory and defined the amount of information that one random variable carries about another. It measures how much knowing the value of one variable reduces uncertainty about a second variable: it is zero exactly when the two variables are statistically independent, and it grows as their outcomes become more predictive of each other. Cover and Thomas, in the standard reference Elements of Information Theory, give the modern formulation, writing the mutual information of variables X and Y as the entropy of X minus the entropy of X once Y is known, equivalently the expected logarithm of the ratio of their joint probability to the product of their separate probabilities. What makes it a landmark is that it captures any statistical dependence, linear or nonlinear, rather than only the straight-line association a correlation coefficient records, it is symmetric so that X reveals as much about Y as Y reveals about X, and it is expressed in fixed information units such as bits or nats.[1][2]

Unit
A non-negative quantity of information, measured in bits (log base 2), nats (natural log), or shannons
Recovery means
Not applicable
Measured by
Computed from the joint and marginal probability distributions as the sum over outcomes of p(x,y) times the log of p(x,y) divided by p(x)p(y), equivalently H(X) + H(Y) minus H(X,Y); estimated from data with histogram, kernel, or k-nearest-neighbor estimators

Next

Run it through the Collider, or take the citations with you.

See how mutual information sits against neighbouring constructs in the Collider, or export every reference on this page in one click.

Open in ColliderExport BibTeXReport this entry

Nearby constructs

information4 fields · 5 citationsjingle · one word, four constructsentropy6 fields · 12 citationsjingle · one name, six constructscapacity5 fields · 8 citationsjingle · one word, four limits

References

  1. Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  2. Cover, T. M., & Thomas, J. A. (2006). Elements of information theory (2nd ed.). Wiley-Interscience. https://doi.org/10.1002/047174882X

Cited on this page?

Confirm your construct, or correct us.

If your work is cited behind one of these entries, you know better than we do whether we captured the meaning right. Confirm it, suggest an edit, or dispute it, and your name goes on the entry as an author who has checked it, linked to your ORCID.

Claim your construct