Automating model selection — stepwise, ridge, lasso, elastic net, in Python
{‘One idea’: ‘a price for large coefficients’}
Ridge — shrink every coefficient, with an example
Lasso — remove the weak ones, with an example and by hand
Elastic net — keep correlated predictors together, with an example
The three side by side, and choosing \(\lambda\) by cross-validation
What you may report, and key takeaways
Appendices — the geometry and simulations behind the slides
What session 5 is for
The mathematics
Say how stepwise (AIC), ridge, the lasso and the elastic net each automate model selection
Say in plain words what ridge, the lasso and the elastic net each do to the coefficients
Say what the elastic net adds when predictors are correlated
Apply soft-thresholding by hand to a small example
The Python
Standardise before penalising, with StandardScaler inside a pipeline
Fit Ridge, Lasso, ElasticNet, knowing that scikit-learn calls \(\lambda\)alpha
Choose \(\lambda\) with RidgeCV, LassoCV, ElasticNetCV
Read a coefficient path and a cross-validation curve
The international business
Choose among 56 candidate determinants of bilateral FDI
Say why a variable the lasso dropped is not thereby irrelevant
Report a penalised model honestly, including what selection did
Your Python so far
Everything you need to type is on one sheet — how to start Python in VS Codium on your platform, how to make a file and run it, the DataFrame basics, and every name from sessions 1 to 5, each with a line showing it in use.
We want to explain R&D intensity (R&D as % of GDP) across European countries. We have 29 candidate predictors but only 37 usable rows — almost as many predictors as countries. Which ones go into the model?
Choosing by hand is slow and subjective. Instead, we can automate model selection:
Method
How it chooses
Where you met it
Stepwise (the old style)
adds or drops one predictor at a time, keeps the change that lowers AIC most, and stops when nothing improves
AIC: session 03
Ridge
keeps every predictor, but shrinks all coefficients toward zero
today
Lasso
shrinks, and sets the weak predictors to exactly zero — so it selects
today
Elastic net
a mix of the two: selects like the lasso, but keeps correlated predictors together
today — our default
Stepwise is a loop: try each predictor, compare AIC, keep the best, repeat. Nothing more.
Ridge, lasso and elastic net are OLS with one extra term: a price for large coefficients, set by a dial \(\lambda\).
In Python, each is one line, and cross-validation chooses \(\lambda\) for you.
The computer screens dozens of candidates in a second — the screening you would otherwise do by eye, and by prejudice.
On our data, the penalised models cut the prediction error on new rows by more than half compared with OLS. Nothing was added; the coefficients were only restrained.
Stepwise makes all-or-nothing choices, one at a time, so resampling the rows often changes its model. The penalties shrink smoothly, which makes them more stable. That is why the elastic net is the modern default.
Magic, but not truth: what any of these methods keeps is a set of predictors that predicts well on this sample — not “the variables that matter”.
from sklearn.linear_model import RidgeCVridge = make_pipeline(StandardScaler(), # same scale for all RidgeCV(alphas=np.logspace(-3, 3, 60)) # try 60 values of λ ).fit(Xc, yc)print(f"λ = {ridge[-1].alpha_:.2f}")print(f"ridge kept {np.sum(ridge[-1].coef_ !=0)} of {len(XCOLS)} — it never drops a predictor")
λ = 29.82
ridge kept 29 of 29 — it never drops a predictor
from sklearn.linear_model import LassoCVlasso = make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0, max_iter=50_000) # λ by 5-fold CV ).fit(Xc, yc)kept = [c for c, b inzip(XCOLS, lasso[-1].coef_) if b !=0]print(f"λ = {lasso[-1].alpha_:.4f}")print(f"lasso kept {len(kept)} of {len(XCOLS)}:", kept)
from sklearn.linear_model import ElasticNetCVenet = make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=0.5, # half lasso, half ridge cv=5, random_state=0, max_iter=50_000) ).fit(Xc, yc)kept = [c for c, b inzip(XCOLS, enet[-1].coef_) if b !=0]print(f"λ = {enet[-1].alpha_:.4f}")print(f"elastic net kept {len(kept)} of {len(XCOLS)}:", kept)
λ = 0.0635
elastic net kept 10 of 29: ['cloud_use', 'ict_specialists', 'digital_intensity', 'ai_use_any', 'ai_text_mining', 'barrier_cost', 'barrier_skills', 'barrier_legal', 'aux_04', 'aux_10']
In scikit-learn, \(\lambda\) is called alpha, and the elastic net’s mix is l1_ratio (1 = lasso, 0 = ridge). Here the mix is fixed at 0.5; pass a list, such as l1_ratio=[0.1, 0.5, 0.9, 1], and cross-validation chooses it too. The next sections show what each penalty does, on an example where we know the right answer.
OLS chooses the coefficients that fit the data as well as possible — nothing else counts. With few rows and many predictors, “as well as possible” means fitting the noise too: big coefficients, wrong signs, useless variables kept.
The three methods today add one thing: a price for large coefficients.
\[
\text{choose the } \beta\text{s to minimise} \quad
\underbrace{\sum_i (y_i - \hat y_i)^2}_{\text{how badly we fit}}
\;+\;
\lambda \times \underbrace{\text{size of the coefficients}}_{\text{the penalty}}
\]
\(\lambda\) (“lambda”) is the dial. \(\lambda = 0\): plain OLS. \(\lambda\) very large: every coefficient is pushed to 0.
A coefficient can stay large only if it earns it, by improving the fit by more than it costs.
The three methods differ only in how they measure the size of the coefficients:
Method
Size of the coefficients
In one phrase
Ridge
\(\sum_j \beta_j^2\) — the squares
shrinks every coefficient, removes none
Lasso
\(\sum_j \lvert\beta_j\rvert\) — the absolute values
shrinks, and sets the weak ones to exactly 0
Elastic net
a mix of the two
sets weak ones to 0, but keeps “twin” predictors together
1. Standardise the predictors first. The penalty compares coefficients with one another. A predictor measured in euros and one measured in thousands of euros would be punished differently for the same effect. StandardScaler() puts every predictor on the same scale (mean 0, standard deviation 1).
2. In Python, \(\lambda\) is called alpha. scikit-learn writes Ridge(alpha=10) for \(\lambda = 10\). The elastic net’s mix between lasso and ridge is l1_ratio: 1 is pure lasso, 0 is pure ridge.
We want to explain a firm’s export sales from seven candidate predictors. The data are simulated, so — unlike in real research — we know the right answer and can check each method against it.
Predictor
What it is
True effect
productivity
output per worker
3
size
number of employees
2
rd_spending
R&D spending
1.5
patents
patents filed — moves almost exactly with R&D (correlation 0.95)
1.5
ceo_birth_month
pure noise
0
hq_floor
pure noise
0
website_colour
pure noise
0
Two problems are built in, and both are common in real IB data: three useless predictors, and two “twins” (R&D and patents) that measure nearly the same thing. Only 40 firms.
# Simulate 40 firms. We choose the true effects, so we can grade each method.rng = np.random.default_rng(17)n =40productivity = rng.normal(size=n)size = rng.normal(size=n)rd_spending = rng.normal(size=n)patents =0.95* rd_spending + np.sqrt(1-0.95**2) * rng.normal(size=n) # the twinuseless = rng.normal(size=(n, 3))firms = pd.DataFrame({"productivity": productivity, "size": size,"rd_spending": rd_spending, "patents": patents,"ceo_birth_month": useless[:, 0], "hq_floor": useless[:, 1],"website_colour": useless[:, 2],})firms["exports"] = (10+3* productivity +2* size +1.5* rd_spending+1.5* patents + rng.normal(0, 3, n))FEATURES = [c for c in firms.columns if c !="exports"]TRUTH = pd.Series([3, 2, 1.5, 1.5, 0, 0, 0], index=FEATURES)
OLS says R&D lowers exports (−1.07) and triples the weight of patents (4.41): it cannot tell the twins apart. And the useless predictors get non-zero effects.
Ridge charges each coefficient its square. Large coefficients are expensive, small ones almost free.
So every coefficient is pulled toward zero — but none reaches exactly zero. Ridge shrinks; it never removes a predictor.
With twins, ridge shares the credit between them instead of giving it all to one, or giving one a wild positive value and the other a wild negative one.
Think of it as a tax on big coefficients: everyone pays, so everyone gets smaller, but nobody is driven out.
Use it when: many predictors are correlated and you want to keep them all, with stable estimates.
R&D and patents now share the effect (1.05 and 1.91), both positive. The useless predictors are smaller but not zero (hq_floor is −0.003, not 0): ridge kept all seven.
R&D intensity across European countries, the twelve largest OLS coefficients. OLS swings wildly in both directions; ridge keeps the same signs, mostly, at a fraction of the size.
mod = make_pipeline( StandardScaler(), # 1. same scale for every predictor RidgeCV(alphas=np.logspace(-3, 3, 60)) # 2. try 60 values of λ, keep the best).fit(Xtr, ytr)print(f"chosen lambda = {mod[-1].alpha_:.3f}")
The useless predictors are now exactly 0. But the lasso also dropped rd_spending and gave its share to patents (2.43) — a reader would conclude “R&D does not matter”, which the data cannot support.
In the simplest case, the lasso’s rule is: take what the data say about a predictor, subtract \(\lambda\); if the result is below zero, the coefficient is zero.
Example, with \(\lambda = 0.20\) and four predictors whose “evidence” (ρ) is:
what the data say, \(\rho_j\)
\(\lvert\rho_j\rvert - \lambda\)
lasso coefficient
\(0.42\)
\(0.22\)
\(\mathbf{0.22}\)
\(-0.18\)
\(-0.02\) → below zero
\(\mathbf{0}\)
\(0.09\)
\(-0.11\) → below zero
\(\mathbf{0}\)
\(0.31\)
\(0.11\)
\(\mathbf{0.11}\)
The sign is kept: a negative \(\rho\) that survives gives a negative coefficient. This is called soft-thresholding, \(\hat\beta_j = \mathrm{sign}(\rho_j)\,(|\rho_j| - \lambda)_+\). You can do it with a pen — doing it once makes the rule concrete. Ridge, by contrast, divides: \(\rho_j / (1 + \lambda)\), which is never zero.
lcv = LassoCV(cv=5, random_state=0, max_iter=50_000).fit(Ztr, ytr) # tries many λ, keeps the bestkept = [(c, b) for c, b inzip(XCOLS, lcv.coef_) ifabs(b) >1e-8] # 1e-8: floating-point dustprint(f"lambda = {lcv.alpha_:.4f} kept {len(kept)} of {P}")
lambda = 0.0520 kept 6 of 29
With more columns than rows, the lasso can keep at most \(n\) variables — a ceiling, not a finding
Among highly correlated predictors it keeps one and zeroes the rest, almost arbitrarily — as with R&D and patents
It finds the “true” variables only under strong conditions that real data rarely meet
So do not present a lasso selection as “the variables that matter”. It is a set of columns that predicts well, at one \(\lambda\), on one sample.
A mix of the two penalties. The mixing weight \(r\) is l1_ratio in Python: \(r = 1\) is the lasso, \(r = 0\) is ridge.
From the lasso part, it keeps the ability to set useless predictors to exactly zero.
From the ridge part, it keeps the habit of sharing credit among twins — so correlated predictors enter or leave together.
The lasso picks one member of a family; the elastic net keeps the family.
Use it when: you want a short model, but your predictors come in correlated groups (several measures of AI adoption, of openness, of institutions…). In applied work this is the usual case, which is why the practice uses it.
scikit-learn writes the ridge part with an extra factor ½. It changes nothing in what you see.
The useless predictors are 0, as with the lasso; the twins are both kept (0.88 and 1.32), as with ridge. The price: every surviving coefficient is shrunk a little toward zero.
Black is the truth. OLS (red) misses most, including a wrong sign on R&D. Ridge (green) keeps everything; the lasso (blue) removes the noise and one twin; the elastic net (grey) removes the noise and keeps both twins.
Five of our columns all measure how much AI a country’s firms use: twins, five times over.
The lasso keeps one AI measure; the elastic net keeps three of the five. Say “AI adoption matters”, not “text mining, specifically”.
6 · Side by side
The three methods, compared
Ridge
Lasso
Elastic net
Penalty
\(\lambda \sum \beta_j^2\)
\(\lambda \sum \lvert\beta_j\rvert\)
a mix of both
Shrinks coefficients toward 0
yes
yes
yes
Sets some exactly to 0
no
yes
yes
Twin (correlated) predictors
shares the credit
keeps one, drops the others
keeps them together
Python
Ridge(alpha=…)
Lasso(alpha=…)
ElasticNet(alpha=…, l1_ratio=…)
With \(\lambda\) chosen for you
RidgeCV
LassoCV
ElasticNetCV
Use it when
you want to keep every predictor, stably
you want a short list
you want a short list and predictors come in groups
All three: standardise first, and let cross-validation choose \(\lambda\).
Let the data set the dial
In real work you do not choose \(\lambda\) by hand. Cross-validation does it: hide a fifth of the rows, fit on the rest, score on the hidden fifth; repeat for each fifth and each \(\lambda\); keep the best.
LassoCV(cv=5).fit(Z, y).alpha_ # also RidgeCV, ElasticNetCV
The practice also asks for the simplest λ: the largest one whose error is indistinguishable from the best.
Same columns, same rows. OLS barely beats predicting the mean; the penalised fits cut the error by more than half. Nothing was added. Something was restrained.
def test_rmse(model):"""Fit on the training split, score on the untouched 30%.""" model.fit(Xtr, ytr) # the model never sees Xte herereturn np.sqrt(mean_squared_error(yte, model.predict(Xte)))ALPHAS = np.logspace(-3, 3, 60) # a log grid: the order of magnitude mattersL1S = [0.1, 0.5, 0.7, 0.9, 0.95, 1.0] # elastic-net mixing weightsscores = {"predict the mean": np.sqrt(mean_squared_error(yte, np.full_like(yte, ytr.mean()))),"OLS": test_rmse(make_pipeline(StandardScaler(), LinearRegression())),"ridge (CV)": test_rmse(make_pipeline(StandardScaler(), RidgeCV(alphas=ALPHAS))),"lasso (CV)": test_rmse(make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0))),"elastic net": test_rmse(make_pipeline(StandardScaler(), ElasticNetCV(l1_ratio=L1S, cv=5, random_state=0))),}
make_pipeline(StandardScaler(), …): the scaler is refitted inside each fold, so no test information leaks into the standardisation.
Everything in this section can be watched happening. The TensorFlow Playground with the hidden layers removed is not a neural network at all — it is a penalised linear classifier, which is exactly our object.
Only two are needed — \(X_1^2 + X_2^2\) is the radius
So the question is the one this whole session asks: which does the penalty keep?
No penalty — all seven connections stay, at some weight. Nothing is discarded because nothing was asked to be.
L2, rate 0.03 — the lines thin. All seven survive, the useless ones merely small. This is ridge: shrinkage without selection.
L1, rate 0.03 — connections disappear from the diagram. What remains is \(X_1^2\) and \(X_2^2\). This is the lasso, performing variable selection in front of you.
Fitted here as a penalised logistic regression on those same seven features, so the picture is checkable rather than decorative: L1 keeps 2 of 7 — the two squares, everything else exactly zero — while L2 keeps 7 of 7, its largest weights on the same two.
The playground offers L1 or L2. It does not offer the mixture — which is the gap this section exists to fill.
Now imagine that demonstration with \(X_1^2\)duplicated — two columns measuring one thing, as our five AI-adoption columns do. L1 would keep one twin and zero the other, arbitrarily, and the diagram would show it happening. That arbitrariness is the grouping problem, and l1_ratio\(< 1\) is the answer to it.
Four minutes here makes the coefficient path concrete: you have now seen the thing that path is a plot of.
What you may report
Standard errors and \(p\)-values computed after selection are not valid: the same data chose the variables and then “tested” them.
You may report the selected model and its out-of-sample error
You may not report summary() from an OLS refit on the selected columns as if it were an ordinary regression
Honest inference after selection needs the tools of Session 10
Why: Appendix D shows pure noise passing a naive test after selection.
Model selection can be automated: stepwise (AIC) the old way; ridge, lasso, elastic net the modern way
All three methods add a price for large coefficients; \(\lambda\) (alpha in Python) sets the price
Ridge shrinks every coefficient and removes none
Lasso sets weak predictors to exactly zero — and keeps only one of a group of twins
Elastic net sets weak predictors to zero and keeps twins together
Standardise first, let cross-validation set \(\lambda\), and do not report \(p\)-values after selection
Penalising unstandardised predictors
Standardising before the CV split, so the folds see test information
Reporting the lasso’s selected set as “the variables that matter”
Reporting post-lasso \(p\)-values with no caveat
Choosing \(\lambda\) on the test set, then reporting the test error
Now, the practice
Ninety minutes, in your groups. You will fit a full elastic-net path on a wide panel, choose \((\alpha, \lambda)\) honestly, and report what survives — and what that does and does not license you to claim.
Required: the next session’s article, its Dataverse package, and the self-check. That is the pre-session deck.
Optional: the QMIB Lab for this session — a knowledge check on the hour you have just sat through: warin.ca/qmib-labs
Never a prerequisite, never marked for correctness. A completed report counts toward participation because attempting it is the engagement being measured.
Appendices
About the appendices
The geometry and the simulations behind the main slides, kept for reading after the session. Nothing here is new material; it is the why behind what the main slides show.
When two predictors move together, the data can tell you their sum but not how to split it — so OLS splits it differently every sample (the long diagonal cloud). Ridge refuses the extreme splits: slightly off-centre, far less spread.
One line per predictor. Blue: the 14 named indicators; grey: the 15 unvetted series. Ridge shrinks, it never selects.
The lasso charges \(\lambda \sum_j |\beta_j|\) instead of \(\lambda \sum_j \beta_j^2\). Why does that produce coefficients that are exactly zero?
Ridge’s push weakens as a coefficient gets small, so it never quite gets there. The lasso pushes just as hard at 0.01 as at 1 — so a coefficient that is not earning its keep is pushed all the way to zero, and stays there.
The ellipses are equal-fit contours around OLS; the shaded shape is what the budget allows. The estimate is where they first touch. The diamond has corners on the axes, so the touch point usually has \(\beta_2 = 0\). The disc has no corners: both stay nonzero.
In words: if the evidence for a variable is weaker than \(\lambda\), its coefficient is zero; otherwise, shrink it toward zero by \(\lambda\). This is soft-thresholding, \(\hat\beta_j = \mathrm{sign}(\rho_j)\,(|\rho_j| - \lambda)_+\). Its derivation is in chapter 5.
Blue: the 14 named indicators. Grey: the 15 unvetted auxiliary series. Read left to right: as the penalty rises, ridge squeezes every line toward zero; the lasso switches them off one at a time.
Five columns measuring, in substance, one thing: how much AI a country uses. The lasso must pick one — and your paper ends up saying “text mining, specifically, and none of its neighbours”, a claim the data cannot support.
Only the five AI columns are drawn. Ridge keeps all five and shrinks them together; the lasso lets use_any carry the whole group; the elastic net — a mix of both penalties — still produces zeros, but lets more members of the family in. The mixing weight is a second dial; the next slide measures what it buys.
The difference between a model that says “AI adoption matters” and one that says “image recognition, specifically”. The first survives a referee.
Twenty-nine columns of pure noise, 37 rows. Pick the one most correlated with \(y\) — the first step of a lasso path — then test it as if it had been chosen in advance.
