Session 13 · One More Thing: Principal Component and Factor Analyses

• Labels

• training / testing

• supervised / unsupervised (or self-supervised) / reinforcement learning

• System 1 (fast, intuitive) / System 2 (slow, analytical) (D. Kahneman)

• A set of principles that can be applied to regression or classification

What is the difference between regression and classification models?

• Regression predicts a number: a price, a volume, a share.

• Classification predicts a category: default or not, beer or wine. A classifier draws a boundary between the classes.

Multivariate Statistics is a group of statistical methods that focus on studying multiple variables together while focusing on the variation that those variables have in common.

Definition: Multivariate Statistics deals with the treatment of data sets with a large number of dimensions.

Its goals are therefore different from supervised modeling, but also different from segmentation and clustering models.

There are many models in the family of Multivariate Statistics.

• Modern datasets often have a very large number of variables. This makes it difficult to inspect each of the variables individually, due to the practical fact that the human mind cannot easily digest data on such a large scale.

• When a dataset contains a large number of variables, there is often a serious amount of overlap between those variables.

• The components that are found by PCA are ordered from the highest information content to the lowest information content.

PCA is a statistical method that allows you to “regroup” your variables into a smaller number of variables, called components.

This regrouping is done based on variation that is common to multiple variables.

Interesting book: https://shainarace.github.io/LinearAlgebra/pca.html

The goal of PCA is to regroup variables in such a way that the first (newly created) component contains a maximum of variation.

• The second component contains the second-largest amount of variation, etc.

The last component logically contains the smallest amount of variation.

Thanks to this ordering of components, it is made possible to retain only a few of the newly created components, while still retaining a maximum amount of variation.

We can then use the components rather than the original variables for data exploration.

The mathematical definition of the PCA problem is to find a linear combination of the original variables with maximum variance.

This means that we are going to create a (new) component. Let’s call it \(z\). This z is going to be computed based on our original variables \((X_1, X_2, …)\) multiplied by a weight for each of our variables \((u_1, u_2, …)\).

This can be written as \(z = Xu\).

[We apply it to cross-section data. For time-series cross-section data, we can use the superPCA package. Not exam material!]

The mathematical goal is to find the values for \(u\) that will maximize the variance of \(z\), with a constraint of unit length on \(u\).

This problem is mathematically called a constrained optimization using Lagrange Multiplier, but in practice, we use computers to do the whole PCA operation at once.

This can also be described as applying matrix decomposition to the correlation matrix of the original variables.

PCA is efficient in finding the components that maximize variance.

This is great if we are interested in reducing the number of variables while keeping a maximum of variance.

Sometimes, however, we are not purely interested in maximizing variance: we might want to give the most useful interpretations to our newly defined dimensions.

And this is not always easiest with the solution found by a PCA.

We can then apply Factor Analysis: an alternative to PCA that has a little bit more flexibility.

• PCA only works with numeric data

• Categorical data must be encoded as numeric data (e.g. "one-hot encoding" = creating dummies from categories, pd.get_dummies)

• Numeric data must be scaled (otherwise your PCA will be misleading)

Just like PCA, Factor Analysis is also a model that allows reducing information in a larger number of variables into a smaller number of variables. In Factor Analysis we call those latent variables.

Definition: Factor Analysis tries to find latent variables that make sense to us. We can rotate the solution until we find latent variables that have a clear interpretation and “make sense”.

Factor Analysis is based on a model called the common factor model. It starts from the principle that there a certain number of factors in a data set, and that each of the measured variables captures a part of one or more of those factors.

We could imagine having students who are overall good in languages, but bad in technical subjects. In this case, we could try to find a latent variable for language ability and a second latent variable for technical ability.

We now have latent variables that measure the general ability of a student for Language and Technical subjects. But it would still be possible that some students are great at languages overall, but that they are just bad at German.

This is why the Common Factor Model has specific factors: they measure the impact of one specific measured variable on the measured variable.

We could describe it as “Ability for learning German while taking into account the general ability for learning languages”.

Where the PCA model is more of a pragmatic approach, in Factor Analysis we are hypothesizing that latent variables exist.

In a case with two latent variables, we can compute our original variables \(X\) by attributing one part of its variation to our first common latent variable (let’s call it \(k_1\)), part to the second common latent variable (\(k_2\)), and part to a specific factor (specific to this variable; called \(d\)).

In a case with 4 original variables, the Factor Analysis model would be as follows:

$$X_1 = c_{11} * k_1 + c_{12} * k_2 + d_1$$

$$X_2 = c_{21} * k_1 + c_{22} * k_2 + d_2$$

$$X_3 = c_{31} * k_1 + c_{32} * k_2 + d_3$$

$$X_4 = c_{41} * k_1 + c_{42} * k_2 + d_4$$

\(c\) are each coefficient of the coefficient matrix, which are the values that we need to estimate.

• PCA allows us to find the components that contain the maximum amount of information in fewer variables. This makes it a great tool for dimension reduction.

• PCA on the other hand is used in cases where we want to retain the largest amount of variation in the smallest number of variables possible.

PCA is much used in data preparation for Machine Learning tasks, where we want to help the Machine Learning model by already “summarising” the data in an easier to digest form.

• Then as a consequence, the mathematics behind the two methods are, while close to each other, not exactly the same. Although both methods use decomposition, they differ in the details.

• This also causes the fact that Factor Analysis has an additional possibility for rotation of the final solution, while PCA does not. Different Applications

• As Factor Analysis is more flexible for interpretation, due to the possibility of rotation of the solution, it is very valuable in studies for marketing and psychology.

PCA’s advantage is that it allows for dimension reduction while still keeping a maximum amount of information in a data set. This is often used to simplify exploratory analyses or to prepare data for Machine Learning pipelines.