Building Intuition for the F-Statistic Numerator
The General F-Test Formula
The general form of the F-statistic with constraint matrix \(C\) is:
\[F = \frac{(C\hat{\beta} - d)^T [C(X^TX)^{-1}C^T]^{-1} (C\hat{\beta} - d) / q}{\text{MSE}}\]
For all our purposes, \(d = 0\) (we’re testing if parameters equal zero), so we can drop it!
\[F = \frac{(C\hat{\beta})^T [C(X^TX)^{-1}C^T]^{-1} (C\hat{\beta}) / q}{\text{MSE}}\]
- Denominator (MSE): This is always the same - our estimate of \(\sigma^2\)
- The division by \(q\): This just adjusts for the number of restrictions we’re testing
- What we need intuition for: \((C\hat{\beta})^T [C(X^TX)^{-1}C^T]^{-1} (C\hat{\beta})\)
Let’s focus JUST on this “numerator of the numerator” and build some intuition!
Understanding Three Pieces
\(C\hat{\beta}\) - How far are we from 0?
This is a vector. For example, if we’re testing \(\beta_1 = \beta_2 = 0\), then:
\[C\hat{\beta} = \begin{bmatrix} \hat{\beta}_1 \\ \hat{\beta}_2 \end{bmatrix}\]
This vector tells us: “How far from zero are the parameters we’re testing?”
\(C(X^TX)^{-1}C^T\) - The Variance-Covariance Matrix
This is a \(q \times q\) matrix that captures the joint uncertainty. The \(C\) matrices “extract” the relevant parts of the full variance-covariance matrix:
\(\text{Var}(C\hat{\beta}) = \sigma^2 \cdot C(X^TX)^{-1}C^T\)
For our example where \(C\) selects \(\beta_1\) and \(\beta_2\), this multiplication gives us:
\[= \sigma^2 \begin{bmatrix} \text{Var}(\hat{\beta}_1) & \text{Cov}(\hat{\beta}_1, \hat{\beta}_2) \\ \text{Cov}(\hat{\beta}_1, \hat{\beta}_2) & \text{Var}(\hat{\beta}_2) \end{bmatrix}\]
- Diagonal = how uncertain we are about each parameter
- Off-diagonal = whether parameters move together
The \(C\) matrices do the work of pulling out just the rows and columns we care about from the full \((X^TX)^{-1}\) matrix.
The Quadratic Form - A Standardized Distance
\[(C\hat{\beta})^T [C(X^TX)^{-1}C^T]^{-1} (C\hat{\beta})\]
This combines everything into a single number that answers:
“How far from zero are we, accounting for uncertainty and correlation?”
It’s the squared distance, which is like a correlation-adjusted measure of how extreme our estimates are. It is actually known as a Malahanobis distance.
Why Not Just Use \(||\hat{\beta}||\)?
If we just computed \(\sqrt{\hat{\beta}_1^2 + \hat{\beta}_2^2}\) (without the middle, \([C(X^TX)^{-1}C^T]^{-1}\)), we’d be treating all coefficients equally. But:
- \(\hat{\beta}_1\) might be estimated very precisely (small variance)
- \(\hat{\beta}_2\) might be estimated poorly (large variance)
- They might be correlated!
The quadratic form weights by precision giving more importance to coefficients we estimated well.
The Bottom Line
The numerator \((C\hat{\beta})^T [C(X^TX)^{-1}C^T]^{-1} (C\hat{\beta})\) is just:
“How far are our estimates from zero, measured in a way that accounts for their precision and correlation?”
For the overall F-test (testing all slopes = 0), this exactly equals \(SS_{reg}\), the explained sum of squares.