Mutual information (MI) is a fundamental quantity in information theory that measures the amount of information one random variable contains about another. Unlike linear correlation, MI captures all types of statistical dependencies, including nonlinear relationships, making it particularly valuable for analyzing neural coding and stimulus-response relationships.
= H(X) − H(X|Y)
= ΣΣ p(x,y) · log₂[p(x,y) / (p(x)·p(y))]
Properties and Interpretation
MI is always non-negative and equals zero if and only if X and Y are statistically independent. It is symmetric: I(X;Y) = I(Y;X). Normalizing MI by dividing by the geometric mean of the marginal entropies gives a value between 0 and 1, facilitating comparison across different variable pairs.
Applications in Psychology and Neuroscience
In neuroscience, MI quantifies how much information neural responses carry about stimuli, providing a measure of neural coding efficiency. In psychophysics, MI between stimulus categories and responses provides a criterion-free measure of information transmission that complements SDT measures. In language research, MI between adjacent words (pointwise MI) is used to identify collocations and measure word associations.