Session 2: Exploratory Data Analysis - Fundamentals

What session 2 is for

Your Python so far

Everything you need to type is on one sheet — how to start Python in VS Codium on your platform, how to make a file and run it, the DataFrame basics, and every name from sessions 1 to 2, each with a line showing it in use.

Python Cheatsheet · Session 02 (PDF) · in the repository at 02-…/PYTHON-CHEATSHEET.pdf

One sheet per package, functions explained with pictures: pandas · NumPy · SciPy · statsmodels

Class Organization

Introduction

• Statistics is a language, constructed by mathematicians.

• The tools (formulas and processes) are constructs. They do not represent the truth, just the tools to get closer to the truth. They are the foundations that we use in our applications as a data analyst.

So, let us spend some time on the foundations first.

Introduction (cont.)

The magic in the data

Web app

Introduction - Differences between statistics and statistical learning?

The goal is to understand the models, intuitions, and strengths and weaknesses of the various approaches

Introduction - Differences between statistics and statistical learning? (cont.)

graph TB
A[Types of data] --> B[Quantitative]; B --> D{Numerical}
A --> C[Qualitative]; C --> E{Categorical}

D --> F(Continuous)
D --> G(Discrete)

E --> H(Ordinal)
E --> I(Nominal)

Introduction - Differences between statistics and statistical learning? (cont.)

Introduction - Differences between statistics and statistical learning? (cont.)

• Outcomes associated with random experiments (probabilistic) called random variables:

\[ X, Y, Z \]

• Observed values:

\[ x, y, z \]

Let \(X\) be a r.v. taking values in the sample space \(S=\left\{x_{1},\, x_{2},\,\ldots,x_{k}\right\}\)

\[ p(x_{i})=\Pr(X=x_{i}), \quad i=1,2,\ldots,k \]

\[ \sum_{i=1}^{k}p(x_{i}) = p(x_{1})+p(x_{2})+\cdots+p(x_{k}), = 1 \]

Introduction - Differences between statistics and statistical learning? (cont.)

Measuring the center

In what follows, we will look into the following concepts:

Basics:

• The sample mean
• The sample median

Extensions:

• The trimmed mean
• The sample quantile
• The sample percentile

The basics: the sample mean

The basics: the sample mean (cont.)

Wait!

Greek letters are for the RV and latin letters are for the sample?

The basics: the sample mean

# In the VS Codium terminal, with .venv active:  python
import statsmodels.api as sm

# 21 observations from an ammonia-oxidation plant — ships with statsmodels,
# so there is nothing to download.
mydata = sm.datasets.stackloss.load_pandas().data
stack_loss = mydata["STACKLOSS"]

stack_loss.mean()

np.float64(17.523809523809526)

The basics: the sample median

How to find it

• sort the data into an increasing sequence of \(n\) numbers

• \(\tilde{x}\) lies in position \((n + 1)/2\)

• Good: resistant to extreme values, easy to describe

• Bad: not as mathematically tractable, need to sort the data to calculate

stack_loss.median()

np.float64(15.0)

Extensions

from scipy import stats

stats.trim_mean(stack_loss, 0.05)

np.float64(16.789473684210527)

stack_loss.quantile(0.75)

np.float64(19.0)

stack_loss.quantile([0, 0.25, 0.80])

0.00     7.0
0.25    11.0
0.80    20.0
Name: STACKLOSS, dtype: float64

stats.norm.cdf(1800, loc=1500, scale=300)

np.float64(0.8413447460685429)

Measures of spread

In what follows, we will look into the following concepts:

• The sample variance

• The sample standard deviation

• The sample interquartile range (IQR)

Measures of spread

print(stack_loss.var(ddof=1))
print(stack_loss.std(ddof=1))

103.46190476190478
10.171622523565489

[1] 10.17162

stack_loss.quantile(0.75) - stack_loss.quantile(0.25)

np.float64(8.0)

stack_loss.quantile([0.25, 0.75])

0.25    11.0
0.75    19.0
Name: STACKLOSS, dtype: float64

Measuring the shape

In what follows, we will look into the following concepts:

• Skewness: degree of “symmetry”

• Kurtosis: degree of “thickness”

Measuring the shape

Measures of shape: sample skewness

The sample skewness \(g_{1}\):

\[ g_{1} = \frac{1}{n}\frac{\sum_{i = 1}^{n}(x_{i} - \overline{x})^{3}}{s^{3}} \]

Things to notice:

• \(-\infty < g_{1} < \infty\)

• sign of \(g_{1}\) indicates direction of skewness \(\pm)\)

Measures of shape

import numpy as np

def skewness(x):
    """g1, exactly as defined on the previous slide."""
    x = np.asarray(x, float)
    xbar, s = x.mean(), x.std(ddof=1)
    return ((x - xbar) ** 3).mean() / s ** 3

skewness(stack_loss)

np.float64(1.1564007167787318)

# The same test on course data: GDP per capita across 30 European countries
import pandas as pd
core = pd.read_parquet("data/spine/core.parquet")
gdp = core["gdp_pc_eur"].dropna()

skewness(gdp)

np.float64(0.8805789352785794)

2 * np.sqrt(6 / len(gdp))

np.float64(0.2343499700593521)

def excess_kurtosis(x):
    """g2, with the -3 that puts the normal distribution at zero."""
    x = np.asarray(x, float)
    xbar, s = x.mean(), x.std(ddof=1)
    return ((x - xbar) ** 4).mean() / s ** 4 - 3

excess_kurtosis(stack_loss)

np.float64(0.13435238060300536)

# Renewable share: symmetric, but far from normal in the tails
ener = pd.read_parquet("data/spine/angle_a_country.parquet")
renew = ener["share_renew"].dropna()

excess_kurtosis(renew)

np.float64(-0.6560298273186116)

4 * np.sqrt(6 / len(renew))

np.float64(0.47866161858603057)

Conclusion

• We have seen the basic concepts of exploratory data analysis
• We have seen the basic concepts of measuring the center, spread and shape of a distribution
• Every number today described one variable at a time
• Session 03 puts two variables in the same sentence: it fits the smallest possible model, and then asks whether that model can be trusted

Before the next session