Diagnostics

Dr. Lucy D’Agostino McGowan

We Assume

Linearity
Constant variance
Normality of errors
Independence
Randomness
Errors have mean 0

We Assume

Linearity
Constant variance
Normality of errors
Independence
Randomness
Errors have mean 0

Residuals vs. Errors

Errors \(\varepsilon_i\) are not observable

We can instead examine residuals \(\hat{\varepsilon}_i\)
\[\hat{\varepsilon}_i = y_i - \hat{y}_i\]

Checking Constant Variance

The Assumption

Homoscedasticity: \(\text{Var}(\varepsilon_i) = \sigma^2\) for all \(i\)

What we want: Constant spread in residuals across all fitted values

Residuals vs. Fitted Plot

The diagnostic:
Plot \(\hat{\varepsilon}_i\) vs. \(\hat{y}_i\)

What to look for:
- Random scatter around zero
- Constant spread (no funnel shape)
- No patterns or trends

Example: Good vs. Bad

Creating the Plot in R

# Fit model
model <- lm(y ~ x1 + x2)

library(ggplot2)
ggplot(data.frame(fitted = fitted(model), 
                  resid = residuals(model)), 
       aes(x = fitted, y = resid)) +
  geom_point() +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(x = "Fitted Values", y = "Residuals") +
  theme_minimal()

Checking Normality

The Assumption

Normality: \(\varepsilon_i \sim N(0, \sigma^2)\)

Tool: Q-Q Plot (Quantile-Quantile Plot)

Good news: CLT helps a lot!
- Coverage probabilities are robust to non-normality
- With moderate sample sizes, normality violations are often okay
- We can generally ignore minor deviations

Q-Q Plot

How it works:
- Plot sample quantiles vs. theoretical normal quantiles
- If residuals are normal, points fall on a straight line

Example Q-Q Plots

Q-Q Plot in R

ggplot(data.frame(resid = residuals(model)), 
       aes(sample = resid)) +
  stat_qq() +
  stat_qq_line() +
  labs(x = "Theoretical Quantiles", y = "Sample Quantiles") +
  theme_minimal()

When Errors Are Not Normal

The reality: Coverage is usually fine!

Why? Central Limit Theorem kicks in pretty well

Outliers and Influential Points

Three Types of Problematic Points

1. Outliers: Points that don’t fit the model well
- Large residuals
- May or may not affect model fit

2. High Leverage Points: Extreme in predictor space
- Far from center of \(X\) values
- Have potential to influence fit

3. Influential Points: Actually change the model fit
- Combine large residual with high leverage
- Removing them substantially changes \(\hat{\boldsymbol{\beta}}\)

Leverage

Definition: Diagonal elements of the hat matrix
\[h_i = H_{ii} \text{ where } \mathbf{H} = \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\]

Properties:

\(\sum_{i=1}^n h_i = p\) (where \(p\) = number of parameters)
Average leverage: \(\bar{h} = p/n\) (Check points with \(h_i > 2p/n\))

Why Leverage Matters

Variance of residuals:
\[\text{Var}(\hat{\varepsilon}_i) = \sigma^2(1 - h_i)\]

Large \(h_i\) makes residual small!

Standardized Residuals

Problem: Raw residuals have different variances

Solution: Standardize by their standard deviation
\[r_i = \frac{\hat{\varepsilon}_i}{\hat{\sigma}\sqrt{1-h_i}}\]

Standardized Residuals

Approximately follows standard normal (if assumptions hold)
Values beyond ±2 or ±3 suggest outliers

Studentized Residuals

Even better: Use leave-one-out estimate of \(\sigma\)
\[t_i = \frac{\hat{\varepsilon}_i}{\hat{\sigma}_{(i)}\sqrt{1-h_i}}\] where \(\hat{\sigma}_{(i)}\) is computed without observation \(i\)

Why? Doesn’t let outlier affect its own diagnostic

Distribution: Follows \(t_{n-p-1}\) distribution exactly

Computing Standardized Residuals in R

set.seed(1)
x <- 1:20
y <- 2 + 3 * x + rnorm(20)
y[20] <- 80  
model <- lm(y ~ x)

# Standardized residuals
std_resid <- rstandard(model)

# Studentized residuals  
stud_resid <- rstudent(model)

# Compare
tail(data.frame(
  obs = 1:20,
  standardized = round(std_resid, 2),
  studentized = round(stud_resid, 2)
))

Computing Standardized Residuals in R

   obs standardized studentized
15  15        -0.31       -0.30
16  16        -0.70       -0.69
17  17        -0.78       -0.77
18  18        -0.60       -0.58
19  19        -0.72       -0.71
20  20         4.12       16.69

The Danger of Automatic Exclusion

Outliers can be:

Errors (typos, measurement problems)
The most interesting part of your data!
Evidence that your model is wrong
A real phenomenon you need to understand

Never automatically exclude outliers
Investigate them carefully first!

Influential Points: Cook’s Distance

What is Cook’s Distance?

Measures: How much the fitted values change if we remove observation \(i\)

Formula:
\[D_i = \frac{(\hat{y}-\hat{y}_{(i)})^T(\hat{y}-\hat{y}_{(i)})}{p\hat{\sigma}^2}\] where \(\hat{y}_{(i)}\) are the fitted values when observation \(i\) is removed

Alternative form:
\[D_i = \frac{r_i^2}{p} \cdot \frac{h_i}{1-h_i}\] Combines residual size and leverage!

Cook’s Distance in R

# Using previous model with outlier
cooks_d <- cooks.distance(model)

# Plot
ggplot(data.frame(obs = 1:20, cooks = cooks_d), 
       aes(x = obs, y = cooks)) +
  geom_segment(aes(xend = obs, yend = 0), color = "#d5008f") +
  geom_point(color = "#d5008f", size = 2) +
  geom_hline(yintercept = 4/20, linetype = "dashed") +
  labs(title = "Cook's Distance", x = "Observation", y = "Cook's Distance") +
  theme_minimal()

Cook’s Distance in R

Checking Linearity

Partial Regression Plots

Purpose: Isolate the effect of \(x_j\) on \(y\) after accounting for other predictors

How it works:
1. Regress \(y\) on all predictors except \(x_j\) → get residuals \(e_y\)
2. Regress \(x_j\) on all other predictors → get residuals \(e_x\)
3. Plot \(e_y\) vs. \(e_x\)
4. Should see linear relationship if model is correct

Why Partial Regression Plots Help

Benefits:

Shows relationship between \(y\) and \(x_j\) controlling for other variables
Helps detect non-linearity for individual predictors
Can reveal need for transformations
Shows influence of individual points on that coefficient

The slope of the partial regression plot equals \(\hat{\beta}_j\)

Partial Regression Plot Example

Partial Regression Plots in R

model_full <- lm(y ~ x1 + x2)
e_y <- residuals(lm(y ~ x2))
e_x1 <- residuals(lm(x1 ~ x2))

ggplot(data.frame(e_x1, e_y), aes(x = e_x1, y = e_y)) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE, formula = y ~ x) +
  labs(x = "x1 residuals", y = "y residuals") +
  theme_minimal()

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean	The error distribution is centered at zero	by default	–
Constant Variance

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean	The error distribution is centered at zero	by default	–
Constant Variance	The variability in the errors is the same for all values of the predictor variable	Residuals vs.fits plot	fit a better model (try taking the log or square root of the outcome)
Independence

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean	The error distribution is centered at zero	by default	–
Constant Variance	The variability in the errors is the same for all values of the predictor variable	Residuals vs.fits plot	fit a better model (try taking the log or square root of the outcome)
Independence	The errors are assumed to be independent from one another	👀 data generation	Find better data or fit a fancier model
Random

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean	The error distribution is centered at zero	by default	–
Constant Variance	The variability in the errors is the same for all values of the predictor variable	Residuals vs.fits plot	fit a better model (try taking the log or square root of the outcome)
Independence	The errors are assumed to be independent from one another	👀 data generation	Find better data or fit a fancier model
Random	The data are obtained using a random process	👀 data generation	Find better data
Normality

Conditions for OLS

Assumption	What it means	How do you check?	How do you fix?
Linearity	The relationship between the outcome and explanatory variable or predictor is linear holding all other variables constant	Residuals vs. fits plot Partial regression plots	fit a better model (transformations, polynomial terms, more / different variables, etc.)
Zero Mean	The error distribution is centered at zero	by default	–
Constant Variance	The variability in the errors is the same for all values of the predictor variable	Residuals vs.fits plot	fit a better model (try taking the log or square root of the outcome)
Independence	The errors are assumed to be independent from one another	👀 data generation	Find better data or fit a fancier model
Random	The data are obtained using a random process	👀 data generation	Find better data
Normality	The random errors follow a normal distribution	QQ-plot	Get more data