set.seed(1)
n <- 100
x <- runif(n, 0, 10)
y <- 2 + 3*x - 0.3*x^2 + rnorm(n)
# Linear model (wrong)
model_linear <- lm(y ~ x)
# Quadratic model (correct)
model_quad <- lm(y ~ x + I(x^2))
# Create grid for predictions
x_grid <- seq(0, 10, length.out = 100)
pred_linear <- predict(model_linear,
newdata = data.frame(x = x_grid))
pred_quad <- predict(model_quad,
newdata = data.frame(x = x_grid))
ggplot() +
geom_point(aes(x = x, y = y), alpha = 0.5, color = "#d5008f") +
geom_line(aes(x = x_grid, y = pred_linear, color = "Linear"),
linewidth = 1) +
geom_line(aes(x = x_grid, y = pred_quad, color = "Quadratic"),
linewidth = 1) +
scale_color_manual(values = c("Linear" = "blue", "Quadratic" = "#d5008f")) +
labs(title = "Linear vs. Quadratic Fit",
x = "x", y = "y", color = "Model") +
theme_minimal() +
theme(legend.position = "top")Transformations
Dr. Lucy D’Agostino McGowan
Why transform?
Transformations are powerful tools in regression analysis that help us:
- Stabilize variance (transforming the response)
- Model non-linear relationships (transforming predictors)
- Meet model assumptions (normality, linearity, homoscedasticity)
Transforming the Response Variable
Why Transform the Response?
Non-constant variance (heteroscedasticity)
Solution: Transform \(y\) to stabilize variance
Example: Non-Constant Variance
Common Transformations
Log transformation: \(\log(y)\)
- Use when variance increases with mean
- Multiplicative errors become additive
- Can’t use with zero or negative values
Common Transformations
Square root transformation: \(\sqrt{y}\)
- Use for count data
- Stabilizes variance for Poisson-like data
- Works with zeros
Applying a Log Transformation
Back-Transformation
\[\log(y) = \hat\beta_0 + \hat\beta_1 x_1...\hat\beta_px_p \]
\[\hat{y}=e^{\hat\beta_0}e^{\hat\beta_1x_1}... e^{\hat\beta_px_p}\]
Handling Zeros in Log Transformation
Problem: \(\log(0)\) is undefined
Common solution: Add a small constant
\[\log(y + c)\]
where \(c\) is a small positive number (e.g., 0.5 or 1)
Transforming Predictor Variables
Why Transform Predictors?
Problem: Relationship between \(y\) and \(x\) is non-linear
Solution: Include functions of \(x\) in the model
- Polynomials: \(x, x^2, x^3, \ldots\)
- Splines: piecewise polynomials
- Other: \(\log(x)\), \(\sqrt{x}\), etc.
Polynomial Regression
Model: \[y = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_p x^p + \varepsilon\]
Interpretation changes!
- Can no longer say “one-unit increase in \(x\)”
- Effect of \(x\) depends on current value of \(x\)
Polynomial Example
Polynomial Example
Interpreting Polynomial Coefficients
IQR-Based Interpretation
When you don’t have a specific change in mind:
Use the interquartile range (IQR) of \(x\)
Predicted \(y\) at Q1 (3.23): 8.49
Predicted \(y\) at Q3 (7.67): 7.419
Change for IQR increase in \(x\): -1.071
Why IQR? Represents a “typical” change in \(x\)
IQR-change
Restricted Cubic Splines (Natural Splines)
Better than polynomials for flexible modeling
Key idea:
- Piecewise cubic polynomials
- Joined smoothly at “knots”
- Linear beyond boundary knots
- Doesn’t oscillate wildly at boundaries
Why Splines?
Problems with high-degree polynomials:
- Unstable at boundaries
- All coefficients change when you add data
- Global influence: one point affects fit everywhere
Advantages of splines:
- Local influence: only nearby regions affected
- Stable, smooth fits
- More degrees of freedom where you need them
Splines: Visual Intuition
Using Splines in R
The splines package provides ns() function
library(splines)
# Fit model with natural spline
# df = degrees of freedom (flexibility)
model_spline <- lm(y ~ ns(x, df = 4))
# Look at coefficients
coef(model_spline) (Intercept) ns(x, df = 4)1 ns(x, df = 4)2 ns(x, df = 4)3 ns(x, df = 4)4
0.6434110 -2.6657928 1.4944870 -0.1684378 -1.1756994 Interpretation: Still no “one-unit” change!
Must evaluate predicted values at specific \(x\) values
Choosing Degrees of Freedom
df controls flexibility:
- Smaller df = smoother, less flexible
- Larger df = more flexible, risk overfitting
model_df3 <- lm(y ~ ns(x, df = 3))
model_df6 <- lm(y ~ ns(x, df = 6))
model_df10 <- lm(y ~ ns(x, df = 10))
x_grid <- seq(min(x), max(x), length.out = 200)
pred_df3 <- predict(model_df3, newdata = data.frame(x = x_grid))
pred_df6 <- predict(model_df6, newdata = data.frame(x = x_grid))
pred_df10 <- predict(model_df10, newdata = data.frame(x = x_grid))
ggplot() +
geom_point(aes(x = x, y = y), alpha = 0.3) +
geom_line(aes(x = x_grid, y = pred_df3, color = "df = 3"), linewidth = 1) +
geom_line(aes(x = x_grid, y = pred_df6, color = "df = 6"), linewidth = 1) +
geom_line(aes(x = x_grid, y = pred_df10, color = "df = 10"), linewidth = 1) +
scale_color_manual(values = c("df = 3" = "blue",
"df = 6" = "#d5008f",
"df = 10" = "orange")) +
labs(title = "Effect of Degrees of Freedom",
x = "x", y = "y", color = "Model") +
theme_minimal() +
theme(legend.position = "top")Choosing Degrees of Freedom
Creating Spline Basis “By Hand”
What does ns() actually do?
It creates a design matrix with basis functions
# Create spline basis matrix
x_example <- c(1, 2, 3, 4, 5)
basis_matrix <- ns(x_example, df = 3)
# This is what gets multiplied by coefficients
basis_matrix 1 2 3
[1,] 0.00000000 0.0000000 0.0000000
[2,] -0.08921458 0.4785813 -0.3190542
[3,] 0.32047336 0.4760799 -0.2965533
[4,] 0.50226181 0.3213396 0.1060861
[5,] -0.14285714 0.4285714 0.7142857
attr(,"degree")
[1] 3
attr(,"knots")
[1] 2.333333 3.666667
attr(,"Boundary.knots")
[1] 1 5
attr(,"intercept")
[1] FALSE
attr(,"class")
[1] "ns" "basis" "matrix"Each column is a basis function evaluated at the \(x\) values
Design Matrix for Splines
# Generate some x values
x_vals <- seq(0, 10, length.out = 100)
# Create design matrix manually
X <- cbind(1, ns(x_vals, df = 4))
# Look at structure
dim(X)[1] 100 5 1 2 3 4
[1,] 1 0.000000e+00 0.00000000 0.00000000 0.00000000
[2,] 1 1.099317e-05 -0.01023872 0.03071616 -0.02047744
[3,] 1 8.794540e-05 -0.02044399 0.06133197 -0.04088798Design matrix has:
- First column: intercept (all 1s)
- Remaining columns: spline basis functions
Fitting with Design Matrix “By Hand”
# Create response and design matrix
set.seed(1)
n <- 50
x <- runif(n, 0, 10)
y <- sin(x) + rnorm(n, sd = 0.3)
# Design matrix: intercept + spline basis
X <- cbind(1, ns(x, df = 4))
# Solve normal equations manually
beta_hat <- solve(t(X) %*% X) %*% t(X) %*% y
# Compare to lm()
model <- lm(y ~ ns(x, df = 4))
data.frame(
manual = round(as.vector(beta_hat), 4),
lm_function = round(coef(model), 4)
) manual lm_function
(Intercept) 0.4543 0.4543
ns(x, df = 4)1 -2.3171 -2.3171
ns(x, df = 4)2 1.6965 1.6965
ns(x, df = 4)3 0.2911 0.2911
ns(x, df = 4)4 -1.3703 -1.3703Visualizing Spline Basis Functions
The predicted curve is a weighted sum of these basis functions!
Natural Splines: The Big Picture
Natural splines are piecewise cubic polynomials joined smoothly at knots
Step 1: Choose Knot Locations
Knots are the points where polynomial pieces connect
For df = 4, we typically use 3 interior knots
Placement: Usually at quantiles of the data
- 1st knot at 25th percentile
- 2nd knot at 50th percentile
- 3rd knot at 75th percentile
Knot Placement Example
Step 2: Between Knots, Use Cubic Polynomials
Between each pair of knots, the function is a cubic polynomial:
\[f_i(x) = a_i + b_i x + c_i x^2 + d_i x^3\]
With 3 interior knots, we have 4 regions (4 different cubics)
Visualizing Piecewise Cubics
Problem: Piecewise Cubics Can Be Jumpy
Without constraints, polynomials don’t connect smoothly!
Step 3: Enforce Smoothness at Knots
At each knot \(\xi_k\), we require three conditions:
Condition 1: Function Values Match
\[f_{\text{left}}(\xi_k) = f_{\text{right}}(\xi_k)\]
The curve doesn’t “jump” at the knot
Condition 2: First Derivatives Match
\[f'_{\text{left}}(\xi_k) = f'_{\text{right}}(\xi_k)\]
The slope is continuous at the knot
No sharp corners or kinks!
Condition 3: Second Derivatives Match
\[f''_{\text{left}}(\xi_k) = f''_{\text{right}}(\xi_k)\]
The curvature is continuous at the knot
The curve bends smoothly through the knot
After Smoothness Constraints
Step 4: Natural Boundary Conditions
Problem: Cubic polynomials can oscillate wildly at the boundaries
Solution: Force the function to be linear beyond the outer knots
Natural Boundary Condition
Beyond the boundary knots, set second derivative to zero:
\[f''(x) = 0 \text{ for } x < \xi_1 \text{ or } x > \xi_k\]
Since \(f''(x) = 0\) for a cubic means linear, the tails are straight lines
Why “Natural”?
Putting It All Together
- Choose knots at quantiles
- Fit cubics between knots
- Enforce smoothness at knots (match function, slope, curvature)
- Go linear at boundaries
Result: Smooth, flexible curve that doesn’t go crazy at the edges!
From Constraints to Basis Functions
All these constraints determine a basis
Each basis function satisfies all the smoothness and boundary conditions
Our model is a linear combination of these basis functions:
\[f(x) = \beta_0 + \beta_1 b_1(x) + \beta_2 b_2(x) + \cdots + \beta_k b_k(x)\]
The Basis Functions
