Machine Learning

A Gentle Introduction

Sakeef M. Karim
New York University

CONSORTIUM ON ANALYTICS FOR DATA-DRIVEN DECISION-MAKING

What is Machine Learning?

Defining Machine Learning

A broad definition
- A branch of artificial intelligence (AI) that involves using algorithms and statistical models to enable computer systems to automatically learn from data (via ChatGPT).
A more concrete definition
- A class of flexible algorithmic and statistical techniques for prediction and dimension reduction (see Grimmer, Roberts, and Stewart 2021).
An even more concrete definition
- A way to learn from data and estimate complex functions that discover representations of some input ( $X$ ), or link the input to an output ( $Y$ ) in order to make predictions on new data (see Molina and Garip 2019).

Once deployed, SML algorithms learn the complex patterns linking $X$ —a set of features (or independent variables)—to a target variable (or outcome), $Y$ .
- The goal of SML is to optimize predictions—i.e., to find functions or algorithms that offer substantial predictive power when confronted with new or unseen data.
- Examples of SML algorithms include logistic regressions, random forests, ridge regressions and neural networks.
A quick note on terminology
- If a target variable is quantitative, we are dealing with a regression problem.
- If a target variable is qualitative, we are dealing with a classification problem.

Unsupervised Machine Learning (UML)

High-Level Overview
An Illustration

UML techniques search for a representation of the inputs (or features) that is more useful than $X$ itself (Molina and Garip 2019).
- Put another way, UML algorithms search for hidden structure in high dimensional space.
In UML, there is no observed $Y$ variable—or target—to supervise the estimation process. Instead, we only have a vector of inputs to work with.
The goal in UML is to develop a lower-dimensional representation of complex data by inductively learning from the interrelationships among inputs.
- This can be achieved by reducing a vector of features to a smaller set of scales (e.g., via principal component analysis) or partitioning the sample into a small number of unobserved groups (e.g., via k-means clustering).

Quantity of Interest	Primary Goals	Key Strengths	Key Limitations
Generative (i.e., Classical Statistics)
	Inferring relationships between X and Y	Interpretability; emphasis on uncertainty around estimates; explanatory power	Bounded by statistical assumptions, inattention to variance across samples
Predictive (i.e., Machine Learning)
	Generating accurate predictions of Y	Predictive power; potential to simplify high dimensional data; relatively unconstrained by statistical assumptions	Inattention to explanatory processes, opaque links between X and Y

Bias-Variance Tradeoff

Image can be retrieved here.

Bias emerges when we build SML algorithms that fail to sufficiently map the patterns—or pick up the empirical signal–linking $X$ and $Y$ . Think: underfitting.
Variance arises when our algorithms not only pick up the signal linking $X$ and $Y$ , but some of the noise in our data as well. Think: overfitting.
When adopting an SML framework, researchers try to strike the optimal balance between bias and variance.

$k$ -Fold Cross-Validation

High-Level Overview
An Illustration

Unlike conventional approaches to sample partition, $k$ or $v$ -fold cross-validation allows us to learn from all our data.
$k$ -fold cross-validation proceeds as follows:
- We randomly divide our overall sample into $k$ subsets or folds.
- We train our algorithm on $k - 1$ folds, holding just one group out for model assessment.
- We repeat this process $k$ times—every fold is held out once and used to fit the model $k - 1$ times.
- We then pool or average the evaluation metrics (e.g., predictive accuracy) for all the held-out runs.
Stratified $k$ -fold cross-validation ensures that the distribution of class labels (or for numeric targets, the mean) is relatively constant across folds.

Hyperparameter Optimization

In SML settings, we automatically learn the parameters (e.g., coefficients) of our algorithms during estimation.
Hyperparameters, on the other hand, are chosen by the analyst, guide the entire learning process, and can powerfully shape our algorithm’s predictive performance.
- Examples of hyperparameters include the $k$ in nearest neighbours algorithms, the $α$ penalty term in ridge regressions, or the number of hidden layers in a neural network.
How can analysts settle on the right hyperparameter value(s) for their algorithm?
- Test different values via trial and error.
- Use automated procedures like GridSearchCV from scikit-learn.

$k$ -Nearest Neighbours

Brief Overview of KNN

$k$ -nearest neighbours (KNNs) are simple, non-parametric algorithms that predict values of $Y$ based on the distance between rows (or observations’ inputs).
The estimation of KNNs proceeds as follows:
- The analyst defines the distance metric (e.g., Euclidean, Manhattan) they will use to determine how similar any two observations are based on their vector of features.
- The analyst defines a value for the hyperparameter $k$ —that is, the number of nearest neighbours to find in the training subsample.
- When fed a new data point, KNNs find the $k$ nearest neighbours in the training data, and:
  - Assign the new observation to the class that most of its $k$ nearest neighbours belong to (for classification problems.)
  - Generate a prediction of $Y$ by taking the average of the target variable for the new observation’s $k$ nearest neighbours (for regression problems.)

KNN in Python

Initial Steps

# Importing pandas to wrangle the data, modules from scikit-learn to fit KNN classifier,
# and numpy to iterate over potential values of k (for a grid search):

import pandas as pd 

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold, GridSearchCV

import numpy as np

# Load input data frame:

data = pd.read_csv("https://gattonweb.uky.edu/sheather/book/docs/datasets/MichelinNY.csv", 
                   encoding ='latin-1')
                   
# Zero-in on X and Y variables:

y = data['InMichelin']

X = data.drop(columns = ['InMichelin', 'Restaurant Name'])# Importing pandas to wrangle the data, modules from scikit-learn to fit KNN classifier,
# and numpy to iterate over potential values of k (for a grid search):

import pandas as pd 

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold, GridSearchCV

import numpy as np

# Load input data frame:

data = pd.read_csv("https://gattonweb.uky.edu/sheather/book/docs/datasets/MichelinNY.csv", 
                   encoding ='latin-1')
                   
# Zero-in on X and Y variables:

y = data['InMichelin']

X = data.drop(columns = ['InMichelin', 'Restaurant Name'])# Importing pandas to wrangle the data, modules from scikit-learn to fit KNN classifier,
# and numpy to iterate over potential values of k (for a grid search):

import pandas as pd 

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold, GridSearchCV

import numpy as np

# Load input data frame:

data = pd.read_csv("https://gattonweb.uky.edu/sheather/book/docs/datasets/MichelinNY.csv", 
                   encoding ='latin-1')
                   
# Zero-in on X and Y variables:

y = data['InMichelin']

X = data.drop(columns = ['InMichelin', 'Restaurant Name'])# Importing pandas to wrangle the data, modules from scikit-learn to fit KNN classifier,
# and numpy to iterate over potential values of k (for a grid search):

import pandas as pd 

from sklearn.neighbors import KNeighborsClassifier

from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold, GridSearchCV

import numpy as np

# Load input data frame:

data = pd.read_csv("https://gattonweb.uky.edu/sheather/book/docs/datasets/MichelinNY.csv", 
                   encoding ='latin-1')
                   
# Zero-in on X and Y variables:

y = data['InMichelin']

X = data.drop(columns = ['InMichelin', 'Restaurant Name'])

Performing Train-Test Split, Estimating First Model (

k

= 5)

# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)# Perform train-test split:

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify = y, test_size = 0.2, random_state = 905)
                                                    
# Initializing KNN classifier with k = 5, fitting model:

knn = KNeighborsClassifier(n_neighbors = 5)

knn.fit(X_train.values, y_train)

# Stratified k-fold cross-validation:

skfold = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 905)

# Cross-validation score:

cross_val_score(knn, X_train, y_train, cv = skfold).mean()

# Measure of predictive performance:

knn.score(X_test.values, y_test)

Finding An Optimal Value of

k

# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))# Creating a grid of potential hyperparameter values (odd numbers from 1 to 13):

k_grid = {'n_neighbors': np.arange(start = 1, stop = 15, step = 2) }

# Setting up a grid search to home-in on best value of k:

grid = GridSearchCV(KNeighborsClassifier(), param_grid = k_grid, cv = skfold)

grid.fit(X_train, y_train)

# Extract best score and hyperparameter value:

print("Best Mean Cross-Validation Score: {:.3f}".format(grid.best_score_))

print("Best Parameters (Value of k): {}".format(grid.best_params_))

print("Test Set Score: {:.3f}".format(grid.score(X_test, y_test)))

Machine Learning A Gentle Introduction Sakeef M. Karim New York University CONSORTIUM ON ANALYTICS FOR DATA-DRIVEN DECISION-MAKING

1. Machine Learning
2.1. Roadmap
2.2. What We’ll Cover Today
3.1. What is Machine Learning?
3.2. Defining Machine Learning
3.3. Supervised Machine Learning (SML)
3.4. Unsupervised Machine Learning (UML)
4.1. Machine Learning vs Classical Statistics
4.2. The Two Cultures
4.3. The Two Cultures (cont.)
4.4. The Affordances of Machine Learning
5.1. Key Terms and Concepts in the SML Setting
5.2. Bias-Variance Tradeoff
5.3. Training, Validation and Testing
5.4. $k$ -Fold Cross-Validation
5.5. Hyperparameter Optimization
6.1. $k$ -Nearest Neighbours
6.2. Brief Overview of KNN
6.3. KNN in Python
6.4. A Quick Exercise
7.1. The End (for now)
7.2. References