Supervised Learning

This course explores the foundations of Supervised Machine Learning through a rigorous probabilistic and statistical lens. Students will learn how to predict labels and continuous targets from observed data, treating each algorithm not as a black box but as an estimator with explicit distributional assumptions, an associated likelihood, and quantifiable error behavior.

Final Objectives

The goal is to provide a deep understanding of how supervised algorithms learn a mapping from inputs to outputs, and why they work. By the end of this course, students will be able to bridge algorithmic concepts like decision trees, regularization, ensembles, and neural networks with their theoretical statistical roots, including maximum likelihood estimation, Bayesian inference, the bias-variance decomposition, and constrained optimization.

Course Content

Click on each lesson to access its detailed planning and study materials.

Part 1: Foundations of Probability and Partitioning

• Lesson 1: Data, Distributions, and Anomaly Detection (Slides)

  • ML Concept: One-dimensional binary classification and outlier detection.
  • Statistical Concept: Fitting continuous densities to bounded data (Beta distribution) and defining decision thresholds from low-probability regions. The unavoidable trade-off between Type I and Type II errors as a consequence of overlapping class-conditional densities.
  • Objectives: Understand classification as a comparison between fitted densities, and thresholds as statistical decisions with explicit error costs.
  • Expected Competencies: Ability to fit a Beta distribution to observed data, place a decision threshold based on tail probabilities, and quantify the resulting Type I and Type II error rates.

• Lesson 2: Conditional Distributions and Generative Models (Slides)

  • ML Concept: Multidimensional classification under the assumption of feature independence (Naive Bayes).
  • Statistical Concept: Bayes’ theorem decomposed into prior, likelihood, and posterior. The distinction between joint and conditional distributions, and the conditional independence assumption as a deliberate simplification to mitigate the curse of dimensionality.
  • Objectives: Understand the generative approach to classification and the cost-benefit of independence assumptions in high dimensions.
  • Expected Competencies: Ability to derive the Naive Bayes classifier from Bayes’ theorem and explain when the independence assumption degrades — or fails to degrade — predictive performance.

• Lesson 3: Decision Trees — Greedy Partitioning

  • ML Concept: The CART algorithm for classification and regression.
  • Statistical Concept: Non-parametric estimation of the conditional density as a piecewise-constant function. Impurity measures (Entropy and Gini) interpreted as the uncertainty of categorical distributions, and splits understood as the greedy maximization of a profile log-likelihood.
  • Objectives: Reinterpret recursive partitioning as a likelihood-driven, non-parametric estimation procedure rather than a heuristic.
  • Expected Competencies: Ability to implement CART, compute impurity gains, and justify split criteria in terms of log-likelihood maximization.

• Lesson 4: Model Selection and Resampling Techniques

  • ML Concept: Model validation and overfitting prevention.
  • Statistical Concept: Empirical error versus expected generalization error. Cross-validation framed as a simulation of repeated samples from the joint distribution \(P(X,Y)\), and the Bootstrap as a tool for estimating the sampling variance of an estimator.
  • Objectives: Treat validation as statistical estimation of an unobservable quantity, not as a bookkeeping step.
  • Expected Competencies: Ability to design sound cross-validation schemes and apply the Bootstrap to quantify the uncertainty of a performance estimate.

Part 2: The Parametric Linear Story

• Lesson 5: Linear Regression and Maximum Likelihood

  • ML Concept: Prediction of continuous targets via hyperplanes.
  • Statistical Concept: The homoscedastic Gaussian noise model \(Y \mid X \sim \mathcal{N}(\beta^T X, \sigma^2)\), the construction of the likelihood function, and the mathematical proof that Ordinary Least Squares (OLS) and Maximum Likelihood Estimation (MLE) coincide under normality.
  • Objectives: Establish maximum likelihood as the unifying principle underlying the entire parametric part of the course.
  • Expected Competencies: Ability to write the likelihood of a linear model and derive the OLS solution as its maximizer.

• Lesson 6: Logistic Regression and the GLM Framework

  • ML Concept: Parametric binary classification.
  • Statistical Concept: The Bernoulli distribution, the link function (logit) that maps a linear combination into a valid probability, and the derivation of cross-entropy loss directly from the Bernoulli log-likelihood.
  • Objectives: Understand classification losses as log-likelihoods of a chosen response distribution, and place logistic regression within the wider GLM family.
  • Expected Competencies: Ability to derive cross-entropy from first principles and generalize the construction to other exponential-family responses.

• Lesson 7: Regularization and Bayesian MAP Estimation

  • ML Concept: Ridge (\(L_2\)) and Lasso (\(L_1\)) penalties for complexity control.
  • Statistical Concept: The transition from frequentist to Bayesian thinking (parameters as random variables), Maximum A Posteriori (MAP) estimation, and the equivalence of Ridge to a Gaussian prior and of Lasso to a Laplace prior (which induces sparsity).
  • Objectives: Recognize regularization as prior belief made explicit rather than as an arbitrary penalty term.
  • Expected Competencies: Ability to derive Ridge and Lasso as MAP estimators and explain geometrically why the Laplace prior produces exact zeros.

• Lesson 8: The Theoretical Bias-Variance Tradeoff

  • ML Concept: Mathematical analysis of the capacity and limits of learning algorithms.
  • Statistical Concept: Formal decomposition of the expected mean squared error into three predictive components: squared bias, estimator variance, and irreducible error (\(\sigma^2\)).
  • Objectives: Provide the analytical vocabulary used throughout Part 3 to explain why ensembles and boosting work.
  • Expected Competencies: Ability to derive the decomposition and diagnose whether a given model is bias-limited or variance-limited.

Part 3: Advanced Models and Optimization

• Lesson 9: Ensemble Theory and Variance Reduction

  • ML Concept: Bagging and Random Forests.
  • Statistical Concept: Statistical properties of the average of random variables, the variance formula for a sum of correlated variables (\(\rho\)), and how feature subsampling mathematically breaks that correlation.
  • Objectives: Explain ensemble gains quantitatively through the variance term of the decomposition from Lesson 8.
  • Expected Competencies: Ability to derive the variance of an averaged correlated ensemble and predict how it responds to changes in tree count and feature subsampling rate.

• Lesson 10: Gradient Boosting and Numerical Optimization

  • ML Concept: AdaBoost and Gradient Boosting.
  • Statistical Concept: Sequential additive modeling aimed at bias reduction, approximation of loss functions via Taylor expansion, and the interpretation of residuals and pseudo-residuals as the negative gradient of the loss in function space.
  • Objectives: Understand boosting as gradient descent performed in the space of functions rather than parameters.
  • Expected Competencies: Ability to derive pseudo-residuals for an arbitrary differentiable loss and contrast the bias-reduction mechanism of boosting with the variance reduction of bagging.

• Lesson 11: Support Vector Machines (SVM)

  • ML Concept: Maximum-margin classifiers and the kernel trick.
  • Statistical Concept: Constrained optimization via the Lagrangian dual formulation, non-probabilistic loss functions (Hinge Loss, which induces sparsity in the support vectors), and the need for subsequent statistical calibration (such as Platt scaling) to recover probabilities.
  • Objectives: Examine a model deliberately built outside the likelihood framework and understand what is gained and lost by that choice.
  • Expected Competencies: Ability to formulate the dual problem, identify support vectors, and calibrate SVM scores into usable probabilities.

• Lesson 12: Neural Networks and Representation Learning

  • ML Concept: Multilayer Perceptrons (MLPs) and backpropagation.
  • Statistical Concept: Neural networks as a hierarchical, chained composition of GLMs; final activation functions (such as Softmax) defining the parameters of multinomial distributions; and gradient computation via the chain rule for the optimization of the global likelihood.
  • Objectives: Close the course by showing deep networks as a stacked generalization of the parametric models studied in Part 2.
  • Expected Competencies: Ability to describe an MLP as a composition of GLMs, derive backpropagation from the chain rule, and connect the output layer to an explicit likelihood.