Equivalently, about variance?
I realize it measures the spread of a distribution, but many other metrics could do the same (e.g., the average absolute deviation). What is its deeper significance? Does it have
- a particular geometric interpretation (in the sense, e.g., that the mean is the balancing point of a distribution)?
- any other intuitive interpretation that differentiates it from other possible measures of spread?
What’s so special about it that makes it act as a normalizing factor in all sorts of situations (for example, convert covariance to correlation)?
There’s a very nice geometric interpretation.
Random variables of finite mean form a vector space. Covariance is a useful inner product on that space. Oh, wait, that’s not quite right: constant variables are orthogonal to themselves in this product, so it’s only positive semi-definite. So, let me be more precise – on the quotient space formed by the equivalence relation “is a linear transformation of”, covariance is a true inner product. (If quotient spaces are an unfamiliar concept, just focus on the vector space of zero-mean, finite-variance variables; it gets you the same outcome in this context.)
Right, let’s carry on. In the norm this inner product induces, standard deviation is a variable’s length, while the correlation coefficient between two variables (their covariance divided by the product of their standard deviations) is the cosine of the “angle” between them. That the correlation coefficient is in $[-1,\,1]$ is then a restatement of the vector space’s Cauchy-Schwarz inequality.