Session 8: Practice

Does flexibility buy you anything on your own data?

Thierry Warin, PhD

HEC Montréal

Outline · The workshop

  1. Ninety minutes
  2. The script: one file, four parts
  3. Reproduce — 20 min
  4. Apply to your own angle — 35 min
  5. Break one assumption — 15 min
  6. Write it down — 20 min
  7. The usual mistakes
  8. Two-minute report
  9. After class, if you want it

What session 8 is for

The mathematics

  • Explain how KNN classifies, by the vote of the \(K\) nearest neighbours
  • Find nearest neighbours by hand from a Euclidean distance
  • Decompose test error into squared bias, variance and irreducible error
  • Say why cross-validation estimates out-of-sample error, and what classical in-sample fit cannot tell you

The Python

  • Standardise, split, and choose \(k\) by cross-validation
  • Measure a classifier with AUC rather than accuracy alone
  • Fit the simple alternative too, and let it win if it does

The international business

  • Classify firms or countries when you have no theory of the boundary
  • Say why a flexible method that scores 0.51 has told you something
  • Justify complexity against a simpler model that was actually fitted

Your Python so far

Everything you need to type is on one sheet — how to start Python in VS Codium on your platform, how to make a file and run it, the DataFrame basics, and every name from sessions 1 to 8, each with a line showing it in use.

Python Cheatsheet · Session 08 (PDF) · in the repository at 08-…/PYTHON-CHEATSHEET.pdf

One sheet per package, functions explained with pictures: pandas · NumPy · matplotlib · scikit-learn

Print it, or keep it open beside the editor. Nothing on it is examinable; all of it is assumed.

Practice

Does flexibility buy you anything on your own data?

Ninety minutes

00–20 Reproduce one result from the paper you read
20–55 Apply the method, to your own angle
55–70 Break one assumption, deliberately
70–90 Write, and report 250 words, then two minutes to the room

The presenter is drawn at random when your group is called. All three of you prepare.

The script: one file, four parts

Work in your own session-08 folder (the pre-session’s last slide set it up). Download the starter into it:

session08_practice.py — reads both datasets from the course site, and is cut into the four parts of this workshop.

  • Change one line before you run it: MY_ANGLE = "angle_c_country" → your group’s angle.
  • Run one part at a time: select its lines, Shift + Enter, read the output.
  • CERTIFICATE_VERIFY_FAILED on a Mac: open /Applications/Python\ 3*/Install\ Certificates.command, once.

Needs only pip install pandas numpy matplotlib scikit-learn. Nothing to hand in: this is practice, and the script and your note stay in your folder.

1 · Reproduce — 20 min

Bluwstein, Buckmann, Joseph, Kapadia & Şimşek (2023): can credit growth and the yield curve predict a banking crisis — and does a flexible model beat a logit? Part 1 of the script:

import pandas as pd

SITE = "https://warint.github.io/quantitative-methods/data/"
jst = pd.read_csv(SITE + "session-08/jst_crises.csv")   # 18 economies, 1870-2020

# outcome: a crisis starts within two years · train 1870-1979, test 1980-2020
# -> logit, KNN (K by cross-validation) and a random forest, scored by test AUC
# -> which predictors matter: permutation importance

A failed reproduction that you diagnose teaches as much as one that works. One you do not attempt teaches nothing. Does your ranking of predictors match the paper’s? Does the flexible model win, as it does there?

“It did not work” is not a diagnosis. Where did it stop, and what did you check?

2 · Apply to your own angle — 35 min

Part 2 of the script — after you set MY_ANGLE:

core = pd.read_csv(SITE + "core.csv")            # shared: falling_behind_next is the outcome
mine = pd.read_csv(SITE + MY_ANGLE + ".csv")     # your angle
mine = mine.groupby(["geo", "time"]).mean(numeric_only=True).reset_index()  # one row per country-year
df   = core.merge(mine, on=["geo", "time"], how="inner")

Then the same comparison — logit against KNN, \(K\) by cross-validation, AUC on the last two years. Replace the default predictor list with the ones your question needs.

A number without units is not a finding — and an AUC without its benchmark (0.5, and the logit) is not one either.

3 · Break one assumption — 15 min

Every method in this course rests on something.

Find the assumption this one leans on hardest. Break it deliberately. Record what happened to your answer. Part 3 of the script has three ready to run: (a) no standardising, (b) fewer predictors, (c) a random split instead of a time split.

This is the part that matters most. A group that shows how its result falls apart understands it; a group that only shows it holding does not yet know.

4 · Write it down — 20 min

250 words, in a file in your own folder — it is what you report from:

  • What you found, in units
  • Which assumption you broke, and what it did
  • What this result does not license you to claim

And the standing rule: one LLM output you identified as wrong, and how you established that.

The usual mistakes

  • Running KNN on unstandardised predictors
  • Choosing \(k\) on the test set
  • Reporting accuracy with no benchmark
  • Treating a low training error as evidence of anything

And everywhere in this course: an association described in causal language.

Two-minute report

One number with its units and its uncertainty
One assumption the one you broke, and what happened
One limit what you cannot claim from this

Listen for traceable evidence, not confidence of delivery.

After class, if you want it

The QMIB Lab has a knowledge check on what you have just done: warin.ca/qmib-labs

Optional, never a prerequisite for the next lecture, and never marked for correctness. A completed report counts because attempting it is the engagement being measured — it is one of the three routes to the participation mark.

What is required before the next session: the article, its Dataverse package, and the self-check. That is the pre-session deck, not this.