Final

Problem 1

Consider \(y = X\beta + \varepsilon\) with \(E[\varepsilon]=0\), \(\operatorname{Var}(\varepsilon)=\sigma^2I\).

(a) Suppose someone proposes a different estimator: \(\tilde{\beta} = (X^{T}X)^{-1}X^{T}y + c\) where \(c\) is any fixed vector. This represents adding a “correction term” to OLS. Under what condition on \(c\) is this estimator unbiased? Now compute \(\operatorname{Var}(\tilde{\beta})\) and show it’s always at least as large as \(\operatorname{Var}(\hat{\beta}_{\text{OLS}})\). What does this reveal about why we project onto \(\operatorname{col}(X)\)?

(b) You fit two nested models with design matrices \(X_1 \subset X_2\) (meaning \(X_1\) columns are a subset of \(X_2\) columns). Your colleague claims: “The residuals from model 2 must be strictly smaller in squared length than residuals from model 1, because we have more parameters to fit with.” Is this always true? Prove it or give a counterexample. When does equality occur?

(c) In the full model \(y = X\beta + \varepsilon\), we get fitted values \(\hat{y}\) and residuals \(e\). Your colleague asks: “Why can’t we improve the fit by using \(\hat{y} + 0.1e\) instead? We’d be closer to \(y\).” Explain precisely why this doesn’t work - what goes wrong with the properties we care about?

Problem 2

Consider \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \varepsilon\).

(a) You run two regressions:

Regression A: \(y = \alpha_0 + \alpha_1 x_1 + \varepsilon\)
Regression B: \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \varepsilon\)

Your colleague notices \(\hat{\alpha}_1 \neq \hat{\beta}_1\) and says: “The relationship between \(y\) and \(x_1\) changed when we added \(x_2\). The model must be wrong.” Use the Frisch-Waugh-Lovell theorem to explain exactly what \(\hat{\alpha}_1\) measures versus what \(\hat{\beta}_1\) measures. When are they equal?

(b) For the model \(y = \beta_0 + \beta_1 x + \beta_2 x^2 + \varepsilon\) (i.e., \(x_1 = x\) and \(x_2=x^2\)), someone interprets \(\hat{\beta}_1\) as “the linear effect of \(x\).” Use FWL to show that \(\hat{\beta}_1\) does NOT equal the slope from simple regression of \(y\) on \(x\). What does \(\hat{\beta}_1\) actually measure?

(c) Suppose \(x_1\) and \(x_2\) are uncorrelated in your sample (and both regressions include an intercept). Does this mean \(\hat{\beta}_1\) from the multiple regression equals \(\hat{\alpha}_1\) from the simple regression? Prove your answer.

Problem 3

Consider \(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \varepsilon\) with \(\varepsilon \sim N(0,\sigma^2 I)\).

(a) You test \(H_0: \beta_2 = 0\) and get \(p = 0.30\) (fail to reject). You test \(H_0: \beta_3 = 0\) and get \(p = 0.25\) (fail to reject). Then you test \(H_0: \beta_2 = \beta_3 = 0\) and get \(p = 0.01\) (reject). Your colleague says this is impossible: “How can both coefficients be jointly significant when neither is individually significant?” Explain how this can happen using the geometry of \(F\)-tests.

(b) Describe the relationship between \(x_2\) and \(x_3\) that would produce the pattern in part (a). Why does high collinearity between \(x_2\) and \(x_3\) lead to this phenomenon? (Optional: if you wish, construct a concrete numerical example with \(n=20\) observations demonstrating this pattern.)

(c) In the model from part (a), you fit the full model and get \(SSE_{\text{full}} = 100\). You test dropping \(x_3\) and get \(SSE = 120\). Then you test dropping \(x_2\) and get \(SSE = 115\). Your colleague predicts: “If we drop both \(x_2\) and \(x_3\), we’ll get \(SSE = 135\).” Is this correct? What’s the actual range of possible values for \(SSE\) when dropping both?

Problem 4

True model: \(y_i = \beta_0 + \beta_1 x_i + \varepsilon_i\) where \(\operatorname{Var}(\varepsilon_i) = \sigma^2 w_i\).

(a) You suspect heteroskedasticity and decide to use weights. But you get the direction wrong: instead of using weight matrix \(W = \text{diag}(1/w_1, \ldots, 1/w_n)\), you use \(W = \text{diag}(w_1, \ldots, w_n)\). So your estimator is:

\[\tilde{\beta} = (X^{T}WX)^{-1}X^{T}Wy\]

instead of the correct \(\hat{\beta}_{\text{WLS}} = (X^{T}W^{-1}X)^{-1}X^{T}W^{-1}y\). Is \(\tilde{\beta}\) still unbiased? Compare \(\operatorname{Var}(\tilde{\beta})\) to \(\operatorname{Var}(\hat{\beta}_{\text{OLS}})\) — can using the wrong weights make things worse than doing nothing?

(b) Suppose the true variances follow \(w_i = (\beta_0 + \beta_1 x_i)^2\) (variance proportional to the squared mean). You don’t know \(\beta\), so you:

Step 1: Run OLS to get \(\hat{\beta}_0, \hat{\beta}_1\)
Step 2: Use weights \(\hat{w}_i = (\hat{\beta}_0 + \hat{\beta}_1 x_i)^2\)
Step 3: Run WLS with these estimated weights

Is this procedure valid for inference? Under what conditions would \(E[\tilde{\beta}] \approx \beta\) hold in large samples? What new issues arise from estimating the weights rather than knowing them?

(c) Your colleague suggests: “Let’s use robust standard errors instead of WLS. Then we don’t need to know the weights.” Compare these two approaches:

Approach 1: Use WLS with correct weights \(1/w_i\)
Approach 2: Use OLS with robust standard errors

Which estimator has smaller variance? Under what conditions is inference valid for each approach? Give at least two reasons why you might prefer Approach 2 even though it’s less efficient.