Before any model, know your variable. How large is a typical value? How far apart are the values? Is the distribution symmetric, or does it have a long tail?

Example: GDP per capita across 30 European countries, 2009–2023 — one row per country-year.

• A regression in session 3 will use this variable. If it is strongly skewed, a few very rich countries can drive the result.
• The description decides later choices: report the mean or the median? Model GDP or its logarithm? Look for outliers?
• It also catches errors early: a negative GDP or a value ten times too large shows up here, not in a regression table.

Rule of thumb: the distribution is substantially skewed if \(|g_1| > 2\sqrt{6/n}\).

• Centre: the mean (€30,373) is above the median (€25,733). A few very rich countries pull the mean up; the median does not move.
• Shape: skewness \(g_1 =\) 0.88 is far above the threshold \(2\sqrt{6/n} =\) 0.23: the distribution is substantially right-skewed.
• Spread: for a skewed variable, the IQR (€22,215) describes the typical range better than the standard deviation, which the tail inflates.
• Consequence: describe a “typical” country with the median and IQR, and consider modelling \(\log(\text{GDP})\) later.

“GDP per capita has a median of €25,733 (IQR €18,289–€40,505, n = 437 country-years). The distribution is right-skewed (\(g_1 =\) 0.88), so we report medians.”

Do more productive European countries have higher income per head — and can we trust the answer?

The model says GDP per capita (\(y\)) is a straight line in productivity (\(x\)), plus an error:

\[y_i = \beta_0 + \beta_1 x_i + \varepsilon_i\]

• \(\beta_1\), the slope, is what we want: how much \(y\) changes, on average, when \(x\) is one unit higher.
• \(\varepsilon_i\) is everything about country-year \(i\) that productivity does not explain.

Ordinary least squares picks the line that makes the squared vertical distances as small as possible:

\[\min_{\beta_0,\,\beta_1} \sum_i \big(y_i - \beta_0 - \beta_1 x_i\big)^2\]

Squaring makes every miss count, above or below the line, and punishes big misses most. Then four checks, because fitting is the easy part:

• The slope: one more point of productivity goes with €1,111 more GDP per capita, on average. Say the units: euros per index point.
• The intercept (€-86,118) is the prediction at productivity 0, far outside the data (72 to 129). Do not interpret it.
• Adequacy: \(R^2 =\) 0.63 — productivity accounts for 63% of the variation in GDP per capita. It says nothing about whether a straight line is the right shape.
• Validity: any curve or funnel in the residual plot is structure the line missed — a reason to try a squared term or a log (Part 2).
• Robustness: if one spike towers over the others, one country-year is carrying the result. Find out which, and report the fit with and without it.

“Across 428 country-years, a one-point higher productivity index is associated with €1,111 higher GDP per capita (SE 42; \(R^2 =\) 0.63).”

Which loans default? default is 1 if the borrower did not repay, 0 otherwise.

• A straight line through 0s and 1s (the linear probability model) can predict probabilities below 0 or above 1: for a very good credit score it predicts a negative chance of default.
• We need a model whose predictions always stay between 0 and 1. The logistic function does that: it bends the line into an S-shape.

The model is linear on the log-odds scale:

\[\ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1\,\text{fico}, \qquad p = P(\text{default} = 1)\]

• \(\frac{p}{1-p}\) is the odds: a probability of 0.2 is odds of 0.25, “one default for four repayments”.
• It is not fitted by least squares but by maximum likelihood: Python picks the \(\beta\)s that make the defaults and repayments we actually observed as probable as possible.
• The price: the coefficients come out as log-odds, which nobody reads directly. So we convert them, in two steps.

Python prints \(b_1 =\) -0.0119. Read it in two passes.

First the sign. It is negative: a higher FICO score goes with a lower chance of default. The sign is the one thing you can read straight from a log-odds coefficient — positive raises the chance, negative lowers it.

Then the size. “One more FICO point lowers the log-odds of default by 0.0119.” That sentence is correct, but nobody thinks in log-odds: a reader cannot tell whether 0.0119 is a lot or a little.

• Why does the model use log-odds at all? Because on that scale the relationship is a straight line: every extra point changes the log-odds by the same amount, wherever you start.
• So the log-odds are the model’s working scale. To talk to a reader, we convert them — first to odds, then to probabilities.

What are odds? The chance that something happens divided by the chance that it does not: \(\text{odds} = p / (1 - p)\).

The odds ratio \(e^{b_1} =\) 0.988 is a multiplier: each extra FICO point multiplies the odds by 0.988, i.e. lowers them by 1.2%, holding everything else constant (ceteris paribus).

• Check it: odds at 700 = 0.2028; at 701 = 0.2028 × 0.988 = 0.2004.
• For a bigger step, multiply the coefficient first: 100 points → \(e^{100 \times b_1} =\) 0.305, odds about 70% lower. From 700 to 800 the odds fall from 0.203 to 0.062.
• The rule: if \(e^b > 1\) the odds rise by \((e^b - 1) \times 100\%\); if \(e^b < 1\) they fall by \((1 - e^b) \times 100\%\).

Odds are still not what most readers want. For a probability, put the whole linear predictor through the logistic function: \(p = e^{b_0 + b_1 x} / (1 + e^{b_0 + b_1 x})\). That is what predict() does.

• Every 50-point step multiplies the odds by the same factor, yet the probability falls by less and less. The S-curve is steepest where defaults are common and flattens where they are already rare.
• So a logistic model has no single “effect on the probability”: it depends on where the borrower starts. Report probabilities at a few meaningful values — and only inside the scores you observed (612 to 827).

“Each additional FICO point is associated with 1.2% lower odds of default (odds ratio 0.988). The predicted probability of default falls from 0.27 at a score of 650 to 0.06 at 800.”

How likely is a student to apply to graduate school? The answer is unlikely < somewhat likely < very likely: three categories with a natural order.

• A binary logit would throw information away (collapse to two categories); a multinomial logit would ignore the order.
• The ordinal logit models the cumulative log-odds — the chance of being at or below each cut:

\[\ln\left(\frac{P(Y \le j)}{P(Y > j)}\right) = c_j - (\beta_1\,\text{pared} + \beta_2\,\text{public} + \beta_3\,\text{gpa}), \qquad j = 1, 2\]

• Two cutpoints \(c_1 < c_2\) act as intercepts, one per cut. There is one \(\beta\) per predictor, the same at both cuts: the proportional-odds assumption.
• Because of the minus sign, a positive \(\beta\) pushes students up the scale.

What does “a higher category” mean? There are two cuts: somewhat or very likely versus unlikely, and very likely versus the rest. An odds ratio says how the odds of being above a cut change — and, by proportional odds, it is the same at both cuts.

pared (a parent went to graduate school): \(b =\) 1.048, positive, so it pushes students up the scale. \(e^{b} =\) 2.85: their odds of being above any cut are 2.85 times those of other students — 185% higher.

Check it with the predicted probabilities from the Results tab, at the first cut:

and 1.997 / 0.700 = 2.85 — the odds ratio.

gpa: \(e^{b} =\) 1.85 per full GPA point. A full point is a big move (GPA’s standard deviation is about 0.4); for a 0.4-point rise the odds ratio is \(e^{0.4\,b} =\) 1.28.

public: \(p =\) 0.84. The data cannot tell its effect from zero: report “no detectable effect”, not “6% lower odds”.

The Results table, row by row:

• Each row is a full distribution over the three answers, so it sums to one. Raising one category means lowering another.
• Moving pared from 0 to 1 takes \(P(\text{unlikely})\) from 0.59 down to 0.33, and \(P(\text{very likely})\) from 0.08 up to 0.20. One coefficient shifts the whole distribution up the scale.
• The middle category can go either way: it gains students from below and loses some to above. Here it moves from 0.33 to 0.47.
• This is why you show probabilities: an odds ratio of 2.85 does not tell a reader that the share of “very likely” students more than doubles.

• The sign convention. statsmodels writes \(c_j - x\beta\), so a positive \(\beta\) means higher categories. Some software writes \(c_j + x\beta\); check before you read a sign.
• Proportional odds is an assumption, not a result. Fit a separate binary logit at each cut: if the slopes differ a lot, one number is the wrong summary (session 4, Appendix C).
• The cutpoints are not effects: they only place the thresholds between categories. Do not interpret them as you would a coefficient.

Which high-school programme does a student follow — academic, general or vocational? Three categories with no order: vocational is not “more” than general.

• Pick a baseline category (academic, the most common). The model writes one log-odds equation per other category, each against the baseline:

\[\ln\frac{P(\text{general})}{P(\text{academic})} = b_{10} + b_{11}\,\text{middle} + b_{12}\,\text{high} + b_{13}\,\text{write}\]

\[\ln\frac{P(\text{vocation})}{P(\text{academic})} = b_{20} + b_{21}\,\text{middle} + b_{22}\,\text{high} + b_{23}\,\text{write}\]

• Unlike the ordinal model, every predictor gets its own coefficient in each equation. The equations are estimated together by maximum likelihood, so the three probabilities sum to one.

Everything is relative to the baseline (academic). A relative risk ratio (RRR) multiplies the ratio \(P(\text{category}) / P(\text{academic})\) — how many students are in that programme for every academic one.

write, general vs academic: RRR = 0.944. Each extra writing point multiplies the general-to-academic ratio by 0.944: 5.6% lower. Over 10 points, multiply the coefficient by 10: \(e^{10b} =\) 0.56 — the ratio is almost halved.

write, vocation vs academic: RRR = 0.893 per point, \(e^{10b} =\) 0.32 over 10 points. Writing pulls students away from vocation faster than away from general.

high SES, general vs academic: RRR = 0.313 (\(p =\) 0.024). Compared with a low-SES student, a high-SES student’s general-to-academic ratio is about a third.

middle SES: \(p =\) 0.23 and 0.54 — no detectable difference from low SES in either equation.

From the Results table, middle-SES students:

• Academic rises steadily and vocation falls steadily: easy to read.
• General first rises, then falls — even though its coefficient on write is negative. Why? Vocation loses students faster than general does, and some of the weakest writers who leave vocation land in general.
• The lesson: a coefficient tells you how a category moves against the baseline, not how its own probability moves. Only the probabilities show that.

• Forgetting the baseline. Every coefficient is “relative to academic”. Choose another baseline and every number changes — so always name it.
• Reading a negative coefficient as “less likely”. As the General row shows, a category’s probability can rise while its coefficient is negative.
• Comparing two non-baseline categories directly. General vs vocation is not in the output; it is the difference of their coefficients, \(b_{1k} - b_{2k}\).
• Small categories give unstable estimates: check how many students each programme has before trusting a coefficient.

Which of 29 candidate predictors explain R&D intensity, with only 37 countries?

• With almost as many predictors as rows, OLS fits the noise: large coefficients, wrong signs, useless variables kept. It looks perfect on the data it was fitted to and fails on new data.
• Choosing the predictors by hand is slow and subjective. So we automate model selection.

The three penalised methods all minimise

\[\sum_i (y_i - \hat y_i)^2 + \lambda \times \text{size of the coefficients}\]

\(\lambda\) (alpha in Python) sets the price: 0 gives OLS, a huge value pushes everything to 0. Cross-validation chooses it, and the predictors must be standardised first.

• OLS with every predictor has the largest error on rows it never saw (0.483): it learned the noise of the training rows.
• The penalised models cut that error by more than half. Nothing was added — the coefficients were only restrained.
• Ridge keeps all 29 predictors; the lasso keeps 6 and the elastic net 8. When predictors come in correlated groups, the elastic net keeps the group, which is why it is our default.

“We selected predictors with an elastic net (mix 0.5, λ chosen by 5-fold cross-validation on standardised predictors). It keeps 8 of 29 candidates and reduces the test error from 0.483 (OLS) to 0.198.”

A panel follows the same entities (countries, firms) over several periods. Here: government debt and economic-freedom indices for countries over the years.

• Countries differ in many ways we do not measure — institutions, history, culture. If those also affect debt, a plain regression mixes them into the slopes.
• Fixed effects (FE): give each country its own intercept. Only changes within a country over time identify the slopes, so anything constant about a country is controlled for.
• Random effects (RE): treat the country differences as random noise. More efficient, but only valid if those differences are unrelated to the predictors.

\[\text{debt}_{it} = \alpha_i + \beta_1\,\text{busfreedom}_{it} + \beta_2\,\text{finfreedom}_{it} + \beta_3\,\text{invfreedom}_{it} + \varepsilon_{it}\]

\(\alpha_i\) is the country’s own intercept; \(i\) is the country, \(t\) the year.

If FE and RE give similar coefficients, RE is fine (and more precise). If they differ by more than sampling noise, RE is inconsistent: use FE.

from scipy import stats

diff = fixed.params - random_.params
vdiff = fixed.cov - random_.cov
H = float(diff.values @ np.linalg.pinv(vdiff.values) @ diff.values)
p_value = stats.chi2.sf(H, len(diff))
print(f"Hausman H = {H:.2f} on {len(diff)} df,  p = {p_value:.4f}")

Hausman H = 27.59 on 3 df,  p = 0.0000

• Small \(p\) (below 0.05): the two disagree → use fixed effects.
• Large \(p\): no evidence they disagree → random effects is acceptable, and more precise.
• A fixed-effects slope answers a within question: when a country’s business freedom rises, what happens to its debt? It cannot say why some countries have more debt than others.

Does bonus pay fall with age at the same rate for men and women? A plain regression bonus ~ age + female assumes yes. An interaction lets the slope differ:

\[\text{bonus} = \beta_0 + \beta_1\,\text{age} + \beta_2\,\text{female} + \beta_3\,(\text{age} \times \text{female})\]

• For men (female = 0): slope in age \(= \beta_1\)
• For women (female = 1): slope in age \(= \beta_1 + \beta_3\)
• \(\beta_3\) is the difference between the two slopes. If \(\beta_3 = 0\), the effect of age is the same for both.

Does each extra square foot add the same amount to a house’s price, or does the effect accelerate? Add the predictor squared:

\[\text{price} = \beta_0 + \beta_1\,\text{area} + \beta_2\,\text{area}^2\]

• \(\beta_2 > 0\): the effect accelerates (a U-shape, upward curve). \(\beta_2 < 0\): it levels off (an inverted U).
• The effect of one more square foot is no longer one number: it is \(\beta_1 + 2\beta_2 \times \text{area}\), so it depends on the size of the house.
• It is still a linear regression: linear in the coefficients. Squares, logs, interactions are all allowed.