Confidence Intervals and Coverage

Problem 1

Given the following data:

\[\mathbf{y} = \begin{bmatrix}12\\15\\18\\22\\25\\28\end{bmatrix}\]

\[\mathbf{X} = \begin{bmatrix}1&2\\1&4\\1&6\\1&8\\1&10\\1&12\end{bmatrix}\]

a) Fit the linear regression model \(y = \beta_0 + \beta_1 x + \varepsilon\) and construct a 95% confidence interval for \(\beta_1\) using the standard OLS approach.

b) Interpret the confidence interval in context.

c) Calculate a 90% confidence interval for \(\beta_1\). How does it compare to the 95% interval?

Problem 2

Using the same data from Problem 1:

a) Implement a bootstrap to construct a 95% confidence interval for \(\beta_1\).

b) Compare your bootstrap confidence interval to the parametric confidence interval from Problem 1. Are they similar? Why or why not?

c) Create a histogram of your bootstrap distribution of \(\hat{\beta}_1\) and overlay the theoretical normal distribution.

Problem 3

In this problem, you’ll verify that under correct model assumptions, 95% confidence intervals actually contain the true parameter 95% of the time.

Simulation setup:

  • True model: \(y = 2 + 1.5x + \varepsilon\) where \(\varepsilon \sim N(0, 1)\)
  • Sample size: n = 30
  • Generate x from \(N(0, 1)\)
  • Number of simulations: 5000

a) Write a simulation function that:

  1. Generates data from the true model
  2. Fits OLS regression
  3. Constructs a 95% confidence interval for \(\beta_1\)
  4. Checks whether the true value (1.5) is contained in the interval
  5. Returns TRUE or FALSE.

b) Run the simulation 5000 times and calculate the coverage probability (proportion of intervals containing the true value).

c) What coverage probability did you obtain? Is it close to 95%? Explain why this makes sense given that all assumptions are satisfied.

Problem 4

Now investigate what happens when errors are not normally distributed.

Instead of \(\varepsilon \sim N(0,1)\), let’s try a skewed distribution. Use this for your epsilon generation to make it lognormally distributed:

epsilon <- exp(rnorm(n)) - exp(0.5)

and then complete the same simulation as above, calculating the coverage probability for the interval for \(\beta_1\).

a) What coverage probability did you obtain?

b) Create a histogram of your simulated \(\varepsilon\) and describe their distribution. How does this violate OLS assumptions?

c) Despite the non-normal errors, does the CI coverage seem reasonably close to nominal? Why might this be the case?

Problem 5

Now let’s investigate what happens with severe non-constant variance. Instead of generating your \(\varepsilon\sim N(0,1)\) let’s have it depend on \(x\) like this:

sigma <- exp(x / 2)  
epsilon <- rnorm(n, 0, sigma)

Such that the standard deviation of your errors increases expondentially with \(x\) and then complete the same simulation as above, calculating the coverage probability for the interval for \(\beta_1\).

a) What coverage probability did you obtain? How does it compare to the nominal 95% level?

b) Create a scatterplot showing one simulated dataset with x on the horizontal axis and y on the vertical axis. Add the true regression line. Describe how the variability changes across the range of x.

c) Explain why heteroskedasticity affects confidence interval coverage. What aspect of the standard error calculation is violated?

d) What could you do to obtain valid confidence intervals in the presence of heteroskedasticity?

Problem 6

Let’s now investigate what happens when the model is misspecified due to an omitted variable.

Simulation setup:

  • True model: \(y = 2 + 1.5x_1 + 2x_2+\varepsilon\) where \(\varepsilon \sim N(0, 1)\)
  • Sample size: n = 30
  • Generate \(x_1\) from \(N(0, 1)\)
  • Generate \(x_2\) from \(N(0, 1)\)
  • Number of simulations: 5000

Write a simulation function that fits a misspecified model, omitting \(x_2\) (i.e., the model should be lm(y ~ x1)) and then complete the same simulation as above, calculating the coverage probability for the interval for \(\beta_1\).

a) What coverage probability did you obtain? How does it compare to 95%?

b) In the simulation code, the true effect of \(x_1\) on y is 1.5. When we fit the model omitting \(x_2\), what happens to our estimate of \(\beta_1\)? Is it biased?

c) Calculate the average value of \(\hat{\beta}_1\) across all simulations and compare it to the true value of 1.5. This is the bias in the coefficient estimate.

Now fit this model:
- True model: \(y = 2 + 1.5x_1 + 2x_2+\varepsilon\) where \(\varepsilon \sim N(0, 1)\)
- Sample size: n = 30
- Generate \(x_2\) from \(N(0, 1)\)
- Generate \(x_1 = x_2 + \varepsilon_x\) where \(\varepsilon_x\sim N(0, 1)\)
- Number of simulations: 5000

Write a new simulation function using the above specifications that fits a misspecified model, omitting \(x_2\) (i.e., the model should be lm(y ~ x1)) and then complete the same simulation as above, calculating the coverage probability for the interval for \(\beta_1\).

d) Explain how omitted variable bias could affect confidence interval coverage. Even though the confidence intervals are correctly constructed for the model we fit, why might they not have the correct coverage for the true parameter?

Bonus

Try to come up with simulation scenarios that “break” 95% coverage. What assumptions did you make in these scenarios?