An Introduction to Statistical Learning with Applications in Python

100%

An Introduction to Statistical Learning with Applications in Python

Preface

Statistical learning refers to a set of tools for making sense of complex datasets. In recent years, we have seen a staggering increase in the scale and scope of data collection across virtually all areas of science and industry. As a result, statistical learning has become a critical toolkit for anyone who wishes to understand data - and as more and more of today's jobs involve data, this means that statistical learning is fast becoming a critical toolkit for everyone.

One of the first books on statistical learning - The Elements of Statistical Learning - was published in two thousand one, with a second edition in two thousand nine. This book has become a popular text not only in statistics but also in related fields. One of the reasons for its popularity is its relatively accessible style. But this book is best-suited for individuals with advanced training in the mathematical sciences.

An Introduction to Statistical Learning, With Applications in R - first published in two thousand thirteen, with a second edition in twenty twenty-one - arose from the clear need for a broader and less technical treatment of the key topics in statistical learning. In addition to a review of linear regression, this book covers many of today's most important statistical and machine learning approaches, including resampling, sparse methods for classification and regression, generalized additive models, tree-based methods, support vector machines, deep learning, survival analysis, clustering, and multiple testing.

Since it was published in two thousand thirteen, this book has become a mainstay of undergraduate and graduate classrooms worldwide, as well as an important reference book for data scientists. One of the keys to its success has been that, beginning with Chapter two, each chapter contains an R lab illustrating how to implement the statistical learning methods seen in that chapter, providing the reader with valuable hands-on experience.

However, in recent years Python has become an increasingly popular language for data science, and there has been increasing demand for a Python-

Preface

Preface based alternative to this book. Hence, this book, An Introduction to Statistical Learning, With Applications in Python, covers the same materials as its R counterpart but with labs implemented in Python - a feat accomplished by the addition of a new co-author, Jonathan Taylor. Several of the labs make use of the Python package, which we have written to facilitate carrying out the statistical learning methods covered in each chapter in Python. These labs will be useful both for Python novices, as well as experienced users.

The intention behind this book (and its R counterpart) is to concentrate more on the applications of the methods and less on the mathematical details, so it is appropriate for advanced undergraduates or master's students in statistics or related quantitative fields, or for individuals in other disciplines who wish to use statistical learning tools to analyze their data. It can be used as a textbook for a course spanning two semesters.

One Introduction

Wage Data

Stock Market Data

Gene Expression Data

A Brief History of Statistical Learning

This Book

Who Should Read This Book?

Notation and Simple Matrix Algebra

Organization of This Book

Data Sets Used in Labs and Exercises

Book Website

Two Statistical Learning

Two point one point one Why Estimate F ?

Prediction

Inference

Two point one point two How Do We Estimate F?

Parametric Methods

Non-Parametric Methods

Two point one point three The Trade-Off Between Prediction Accuracy and Model Interpretability

Two point one point four Supervised Versus Unsupervised Learning

Two point one point five Regression Versus Classification Problems

Two point two Assessing Model Accuracy

Two point two point one Measuring the Quality of Fit

Two point two point two The Bias-Variance Trade-Off

Two point two point three The Classification Setting

The Bayes Classifier

K-Nearest Neighbors

Two point three Lab: Introduction to Python

Two point three point two Basic Commands

Two point three point three Introduction to Numerical Python

Two point three point four Graphics

Two point three point five Sequences and Slice Notation

Two point three point six Indexing Data

Indexing Rows, Columns, and Submatrices

Boolean Indexing

Two point three point seven Loading Data

Reading in a Data Set

Two point three point eight For Loops

String Formatting

Two point three point nine Additional Graphical and Numerical Summaries

Two point four Exercises

Three. We now revisit the bias-variance decomposition.

Four. You will now think of some real-life applications for statistical learning.

Ten. This exercise involves the Boston housing data set.

Three Linear Regression

Three point one Simple Linear Regression

simple linear regression

Three point one point one Estimating the Coefficients

Three point one point two Assessing the Accuracy of the Coefficient Estimates

null hypothesis

alternative hypothesis

p-value

Three point one point three Assessing the Accuracy of the Model

Residual Standard Error

R squared Statistic

Three point two. Multiple Linear Regression

Three point two point one. Estimating the Regression Coefficients

Three point two point two. Some Important Questions

One: Is There a Relationship Between the Response and Predictors?

Two: Deciding on Important Variables

Three: Model Fit

Four: Predictions

Three point three Other Considerations in the Regression Model

Predictors with Only Two Levels

Qualitative Predictors with More than Two Levels

baseline

Three point three point two Extensions of the Linear Model

Removing the Additive Assumption

Non-linear Relationships

Three point three point three Potential Problems

One. Non-linearity of the Data

Two. Correlation of Error Terms

Three. Non-constant Variance of Error Terms

Four. Outliers

Five. High Leverage Points

Six. Collinearity

Three point four. The Marketing Plan

Three point five. Comparison of Linear Regression with K-Nearest Neighbors

K-nearest neighbors regression

New imports

Inspecting Objects and Namespaces

Three point six point two Simple Linear Regression

Using Transformations: Fit and Transform

Defining Functions

Three point six point three Multiple Linear Regression

Three point six point four Multivariate Goodness of Fit

List Comprehension

Three point six point five Interaction Terms

Three point six point six Non-linear Transformations of the Predictors

Three point six point seven Qualitative Predictors

Three point seven Exercises

Nine. This question involves the use of multiple linear regression on the Auto data set.

Ten. This question should be answered using the Carseats data set.

Twelve. This problem involves simple linear regression without an intercept.

Fourteen. This problem focuses on the collinearity problem.

Four Classification

Four point one An Overview of Classification

Four point two Why Not Linear Regression?

Four point three Logistic Regression

Four point three point one The Logistic Model

Four point Classification

Four point three point two. Estimating the Regression Coefficients

Four point three point three. Making Predictions

Four point three point four Multiple Logistic Regression

Four point three point five Multinomial Logistic Regression

Four point four Generative Models for Classification

Four point four point one Linear Discriminant Analysis for p equals one

Four point four point two Linear Discriminant Analysis for p greater than one

ROC curve

Four point four point three Quadratic Discriminant Analysis

quadratic discriminant analysis

Four point four point four Naive Bayes

naive Bayes

Four point five A Comparison of Classification Methods

Four point five point two An Empirical Comparison

Four point six Generalized Linear Models

Four point six point one Linear Regression on the Bikeshare Data

Four point six point two Poisson Regression on the Bikeshare Data

Four point six point three Generalized Linear Models in Greater Generality

Four point seven Lab: Logistic Regression, Linear Discriminant Analysis, Quadratic Discriminant Analysis, and K-Nearest Neighbors

Four point seven point two Logistic Regression

Four point seven point three Linear Discriminant Analysis

Four point seven point four Quadratic Discriminant Analysis

Four point seven point five Naive Bayes

Four. Classification

Tuning Parameters

Comparison to Logistic Regression

Four point seven point seven Linear and Poisson Regression on the Bikeshare Data

Four point seven Lab: Logistic Regression, LDA, QDA, and KNN

Four point eight Exercises

Five. We now examine the differences between LDA and QDA.

Five Resampling Methods

Five point one. Cross-Validation

Five point one point one. The Validation Set Approach

Five point one point two. Leave-One-Out Cross-Validation

Five point one point three k-Fold Cross-Validation

Five point one point four Bias-Variance Trade-Off for k-Fold Cross-Validation

Five point one point five Cross-Validation on Classification Problems

Five point two The Bootstrap

Five point three Lab: Cross-Validation and the Bootstrap

Five point three point two Cross-Validation

Shuffle Split()

Five point three point three The Bootstrap

Estimating the Accuracy of a Statistic of Interest

Estimating the Accuracy of a Linear Regression Model

Five point four Exercises

Three. We now review k-fold cross-validation.

Eight. We will now perform cross-validation on a simulated data set.

Nine. We will now consider the Boston housing data set, from the ISLP library.

Six. Linear Model Selection and Regularization

Six point one Subset Selection

Six point one point one Best Subset Selection

Algorithm Six point one Best subset selection

Six point one point two Stepwise Selection

Backward Stepwise Selection

Hybrid Approaches

Six point one point three Choosing the Optimal Model

Cp, AIC, BIC, and Adjusted R squared

Cp Akaike information criterion information criterion adjusted R squared

Validation and Cross-Validation

Six point two Shrinkage Methods

Six point two point one Ridge Regression

An Application to the Credit Data

Why Does Ridge Regression Improve Over Least Squares?

Six point two point two The Lasso

Another Formulation for Ridge Regression and the Lasso

The Variable Selection Property of the Lasso

Comparing the Lasso and Ridge Regression

A Simple Special Case for Ridge Regression and the Lasso

Bayesian Interpretation of Ridge Regression and the Lasso

Six point two point three Selecting the Tuning Parameter

Six point three Dimension Reduction Methods

Six point three point one Principal Components Regression

An Overview of Principal Components Analysis

The Principal Components Regression Approach

Six point three point two Partial Least Squares

Six point four. Considerations in High Dimensions

Six point four point two. What Goes Wrong in High Dimensions?

Six point four point three Regression in High Dimensions

Six point four point four Interpreting Results in High Dimensions

Six point five Lab: Linear Models and Regularization Methods

Six point five point one Subset Selection Methods

Forward Selection

Choosing Among Models Using the Validation Set Approach and Cross-Validation

Best Subset Selection

Six point five point two Ridge Regression and the Lasso

Ridge Regression

Estimating Test Error of Ridge Regression

Fast Cross-Validation for Solution Paths

Evaluating Test Error of Cross-Validated Ridge

The Lasso

Six point five point three PCR and PLS Regression

Partial Least Squares

Six point six Exercises

Six. We will now explore further.

Seven. We will now derive the Bayesian connection to the lasso and ridge regression discussed in Section six point two point two.

Eleven. We will now try to predict per capita crime rate in the Boston data set.

Seven Moving Beyond Linearity

Seven point one Polynomial Regression

polynomial regression

Seven point one Polynomial Regression

Seven point two Step Functions

Seven point three Basis Functions

regression spline

Seven point four point one Piecewise Polynomials

Seven point four point two Constraints and Splines

Seven point four point three The Spline Basis Representation

Seven point four point four Choosing the Number and Locations of the Knots

Seven point four point five Comparison to Polynomial Regression

Seven point five point one. An Overview of Smoothing Splines

Seven point five. Smoothing Splines

Seven point five point two. Choosing the Smoothing Parameter

Seven point six Local Regression

Seven point seven Generalized Additive Models

generalized

Seven point seven point one Generalized Additive Models for Regression Problems

Pros and Cons of Generalized Additive Models

Seven point seven point two Generalized Additive Models for Classification Problems

Seven point eight Lab: Non-Linear Modeling

Seven point eight point one Polynomial Regression and Step Functions

Seven point eight point two Splines

Seven point eight point three Smoothing Splines and GAMs

Additive Models with Several Terms

ANOVA Tests for Additive Models

Seven point eight point four Local Regression

Seven point nine Exercises

Six. In this exercise, you will further analyze the Wage data set considered throughout this chapter.

Ten. This question relates to the College data set.

decision tree

Eight point one The Basics of Decision Trees

Eight point one point one Regression Trees

Predicting Baseball Players' Salaries Using Regression Trees

Prediction via Stratification of the Feature Space

recursive binary splitting

Tree Pruning

cost complexity weakest link pruning

Eight point one point two Classification Trees

classification tree

Gini index

entropy

Eight point one point three Trees Versus Linear Models

Eight point one point four Advantages and Disadvantages of Trees

Eight point two. Bagging, Random Forests, Boosting, and Bayesian Additive Regression Trees

ensemble

Eight point two point one. Bagging

bagging

Out-of-Bag Error Estimation

Variable Importance Measures

Eight point two point two Random Forests

Eight point two point three Boosting

Algorithm eight point two Boosting for Regression Trees

Eight point two point four Bayesian Additive Regression Trees

Algorithm Eight point three Bayesian Additive Regression Trees

Eight point two point five. Summary of Tree Ensemble Methods

Eight point three. Lab: Tree-Based Methods

Eight point three point one. Fitting Classification Trees

Eight point three point two. Fitting Regression Trees

Eight point three point four Boosting

Classifier

Eight point three point five Bayesian Additive Regression Trees

Eight point four Exercises

Applied

Ten. We now use boosting to predict Salary in the Hitters data set.

Eleven. This question uses the Caravan data set.

Nine point one Maximal Margin Classifier

Nine point one point one What Is a Hyperplane?

Nine point one point two Classification Using a Separating Hyperplane

Nine point one point three The Maximal Margin Classifier

Nine point one point four Construction of the Maximal Margin Classifier

Nine point one point five The Non-separable Case

Nine point two Support Vector Classifiers

Nine point two point two Details of the Support Vector Classifier

Nine point three Support Vector Machines

Nine point three point one Classification with Non-Linear Decision Boundaries

Nine point three point two The Support Vector Machine

Nine point three point three An Application to the Heart Disease Data

Nine point four SVMs with More than Two Classes

Nine point four point one One-Versus-One Classification

Nine point four point two One-Versus-All Classification

Nine point five Relationship to Logistic Regression

Nine point six Lab: Support Vector Machines

Nine point six point two Support Vector Machine

Nine point six point three ROC Curves

Nine point six point four SVM with Multiple Classes

Nine point seven Exercises

Three. Here we explore the maximal margin classifier on a toy data set.

Eight. This problem involves the OJ data set which is part of the ISLP package.

Ten. Deep Learning

Ten point one. Single Layer Neural Networks

Ten point two Multilayer Neural Networks

Ten point three Convolutional Neural Networks

Ten point three point one Convolution Layers

Ten point three point two Pooling Layers

Ten point three point three Architecture of a Convolutional Neural Network

Ten point three point four Data Augmentation

Ten point three point five Results Using a Pretrained Classifier

Ten point four Document Classification

Ten point five Recurrent Neural Networks

Ten point five point one Sequential Models for Document Classification

embedding layer

LSTM RNN

Ten point five point two Time Series Forecasting

lag

RNN forecaster

lag

Autoregression

Ten point five point three Summary of R N Ns

Ten point six When to Use Deep Learning

Seq two Seq

Ten point six When to Use Deep Learning

Ten point seven Fitting a Neural Network

Ten point seven point one Backpropagation.

Ten point seven point two Regularization and Stochastic Gradient Descent

Ten point seven point three Dropout Learning

Ten point seven point four Network Tuning

Ten point eight Interpolation and Double Descent

Ten point nine Lab: Deep Learning

Torch-Specific Imports

Ten point nine point one Single Layer Network on Hitters Data

Linear Models

Specifying a Network: Classes and Inheritance

Cleanup

Ten point nine point two Multilayer Network on the MNIST Digit Data

Ten point nine point three Convolutional Neural Networks

Hardware Acceleration

Ten point nine point four Using Pretrained CNN Models

Ten point nine point five IMDB Document Classification

Ten point nine point six Recurrent Neural Networks

Sequential Models for Document Classification

Time Series Prediction

Ten point ten Exercises

Applied

Eleven Survival Analysis and Censored Data

Eleven point one Survival and Censoring Times

Eleven point two A Closer Look at Censoring

Eleven point three The Kaplan-Meier Survival Curve

Eleven point four The Log-Rank Test

Eleven. Survival Analysis and Censored Data

Eleven point five Regression Models With a Survival Response

Eleven point five point one The Hazard Function

hazard function

probability density function

Eleven point five point two Proportional Hazards

baseline hazard

Cox's Proportional Hazards Model

Cox's proportional hazards

Connection With The Log-Rank Test

Additional Details

Eleven point five point three Example: Brain Cancer Data

Eleven point five point four Example: Publication Data

Eleven point six Shrinkage for the Cox Model

Eleven point seven Additional Topics

Eleven point seven point two Choice of Time Scale

Eleven point seven point three Time-Dependent Covariates

Eleven point seven point four Checking the Proportional Hazards Assumption

Eleven point seven point five Survival Trees

Eleven point eight Lab: Survival Analysis

Eleven point eight point two Publication Data

Eleven point eight point three Call Center Data

Eleven point nine Exercises

Ten. This exercise focuses on the brain tumor data, which is included in the ISLP library.

Eleven. This exercise makes use of the data in Table eleven point four.

Twelve Unsupervised Learning

Twelve point one The Challenge of Unsupervised Learning

Twelve point two Principal Components Analysis

principal components analysis

Twelve point two point one What Are Principal Components?

Twelve point two point two Another Interpretation of Principal Components

Twelve point two point four More on PCA

Uniqueness of the Principal Components

Deciding How Many Principal Components to Use

Twelve point two point five Other Uses for Principal Components

Twelve point three Missing Values and Matrix Completion

Principal Components with Missing Values

Algorithm twelve point one Iterative Algorithm for Matrix Completion

Recommender Systems

Twelve point four Clustering Methods

clustering

K-means clustering hierarchical clustering dendrogram

Twelve point four point one K-Means Clustering

Algorithm Twelve point two K-Means Clustering

Twelve point four point two Hierarchical Clustering

Interpreting a Dendrogram

Algorithm twelve point three Hierarchical Clustering

The Hierarchical Clustering Algorithm

linkage

Twelve point four point three Practical Issues in Clustering

Small Decisions with Big Consequences

Validating the Clusters Obtained

Other Considerations in Clustering

A Tempered Approach to Interpreting the Results of Clustering

Twelve point five Lab: Unsupervised Learning

Twelve point five point two Matrix Completion

Twelve point five point three Clustering

Hierarchical Clustering

Agglomerative Clustering()

Twelve point five point four NCI sixty Data Example

Clustering the Observations of the NCI sixty Data

Twelve point six Exercises

Two. Suppose that we have four observations, for which we compute a dissimilarity matrix, given by

Thirteen Multiple Testing

Thirteen point one. A Quick Review of Hypothesis Testing

Thirteen point one point one. Testing a Hypothesis

Step one: Define the Null and Alternative Hypotheses

Step two: Construct the Test Statistic

Step three: Compute the p-Value

Step four: Decide Whether to Reject the Null Hypothesis

Thirteen point one point two Type One and Type Two Errors

Thirteen point two The Challenge of Multiple Testing

Thirteen point three The Family-Wise Error Rate

Thirteen point three point one What is the Family-Wise Error Rate?

Thirteen point three point two Approaches to Control the Family-Wise Error Rate

The Bonferroni Method

Holm's Step-Down Procedure

Holm's method

Two Special Cases: Tukey's Method and Scheffé's Method

Tukey's method

Scheffé's method

Thirteen point three point three Trade-Off Between the FWER and Power

Thirteen point four The False Discovery Rate

Thirteen point four point two The Benjamini-Hochberg Procedure

Algorithm thirteen point two Benjamini-Hochberg Procedure to Control the FDR

Benjamini-Hochberg procedure

Thirteen point five A Re-Sampling Approach to P-Values and False Discovery Rates

Thirteen point five point one A Re-Sampling Approach to the P- Value

Algorithm thirteen point three Re-Sampling P-Value for a Two-Sample t-Test

Thirteen point five point two A Re-Sampling Approach to the False Discovery Rate

Thirteen point five A Re-Sampling Approach to p-Values and False Discovery Rates

Thirteen point five point three When Are Re-Sampling Approaches Useful?

Algorithm thirteen point four Plug-In FDR for a Two-Sample T-Test

Thirteen point six Lab: Multiple Testing

Thirteen point six point two Family-Wise Error Rate

Thirteen point six point three False Discovery Rate

q-values

Thirteen point six point four A Re-Sampling Approach

Thirteen point seven Exercises

Six. For each of the three panels in Figure thirteen point three, answer the following questions:

Seven. This problem makes use of the Carseats dataset in the ISLP package.

Eight. In this problem, we will simulate data from m equals one hundred fund managers.

Overview

This text provides an accessible introduction to statistical learning methods with a focus on applications in Python. Intended for students and professionals in quantitative fields, it covers a range of topics from linear regression to machine learning approaches, equipping readers with practical skills for data analysis.

Key Points

1Introduces statistical learning tools with applications in Python
2Offers hands-on lab exercises throughout the chapters
3Covers essential topics including linear regression, classification, and deep learning
4Designed for advanced undergraduates and master’s students in quantitative fields
5Emphasizes practical application over complex mathematical theory

Details

Authors: Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor
Category: Mathematics and Statistics

PDF
Quantitative Methods with Simulations and Modeling
This module introduces students to quantitative methods and basic statistics essential for decision-making in various fields, particularly Information Technology. It covers descriptive statistics, probability, visualizations, and practical applications through hands-on activities.
PDF
THE CIRCLE
This document provides a comprehensive explanation of the properties and equations of circles, including the standard and general forms, as well as special cases and parametric forms. It covers mathematical principles related to circles, catering to students and individuals interested in geometry.
PDF
Probability of Simple and Compound Events Concepts and Problem Solving
This document provides an in-depth explanation of probability concepts related to simple and compound events, including definitions and examples of various event types, probability calculations, and worked examples to illustrate the ideas.
PDF
Early Ideas and Innovations in Bayesian and Model-Assisted Multiobjective Optimization
This document reviews the advancements in Bayesian and model-assisted multiobjective optimization since the early 2000s, detailing the impact of evolutionary algorithms, Bayesian optimization, and surrogate models in the field. It emphasizes historical contributions, methodologies, and applications, particularly in engineering design and simulation contexts.
PDF
Introduction to Exponents and Powers
This document serves as a comprehensive guide to understanding exponents and powers, detailing their definitions, laws, and applications in simplifying large numbers. It includes examples and exercises for better comprehension.