Statistics | Marcos M. Raimundo

Supervised Learning

Fri, 24 Jul 2026 00:00:00 +0000

This course explores the foundations of Supervised Machine Learning through a rigorous probabilistic and statistical lens. Students will learn how to predict labels and continuous targets from observed data, treating each algorithm not as a black box but as an estimator with explicit distributional assumptions, an associated likelihood, and quantifiable error behavior.

Final Objectives

The goal is to provide a deep understanding of how supervised algorithms learn a mapping from inputs to outputs, and why they work. By the end of this course, students will be able to bridge algorithmic concepts like decision trees, regularization, ensembles, and neural networks with their theoretical statistical roots, including maximum likelihood estimation, Bayesian inference, the bias-variance decomposition, and constrained optimization.

Course Content

Click on each lesson to access its detailed planning and study materials.

Part 1: Foundations of Probability and Partitioning

Lesson 1: Data, Distributions, and Anomaly Detection (Slides)
- ML Concept: One-dimensional binary classification and outlier detection.
- Statistical Concept: Fitting continuous densities to bounded data (Beta distribution) and defining decision thresholds from low-probability regions. The unavoidable trade-off between Type I and Type II errors as a consequence of overlapping class-conditional densities.
- Objectives: Understand classification as a comparison between fitted densities, and thresholds as statistical decisions with explicit error costs.
- Expected Competencies: Ability to fit a Beta distribution to observed data, place a decision threshold based on tail probabilities, and quantify the resulting Type I and Type II error rates.
Lesson 2: Conditional Distributions and Generative Models (Slides)
- ML Concept: Multidimensional classification under the assumption of feature independence (Naive Bayes).
- Statistical Concept: Bayes’ theorem decomposed into prior, likelihood, and posterior. The distinction between joint and conditional distributions, and the conditional independence assumption as a deliberate simplification to mitigate the curse of dimensionality.
- Objectives: Understand the generative approach to classification and the cost-benefit of independence assumptions in high dimensions.
- Expected Competencies: Ability to derive the Naive Bayes classifier from Bayes’ theorem and explain when the independence assumption degrades — or fails to degrade — predictive performance.
Lesson 3: Decision Trees — Greedy Partitioning
- ML Concept: The CART algorithm for classification and regression.
- Statistical Concept: Non-parametric estimation of the conditional density as a piecewise-constant function. Impurity measures (Entropy and Gini) interpreted as the uncertainty of categorical distributions, and splits understood as the greedy maximization of a profile log-likelihood.
- Objectives: Reinterpret recursive partitioning as a likelihood-driven, non-parametric estimation procedure rather than a heuristic.
- Expected Competencies: Ability to implement CART, compute impurity gains, and justify split criteria in terms of log-likelihood maximization.
Lesson 4: Model Selection and Resampling Techniques
- ML Concept: Model validation and overfitting prevention.
- Statistical Concept: Empirical error versus expected generalization error. Cross-validation framed as a simulation of repeated samples from the joint distribution \(P(X,Y)\), and the Bootstrap as a tool for estimating the sampling variance of an estimator.
- Objectives: Treat validation as statistical estimation of an unobservable quantity, not as a bookkeeping step.
- Expected Competencies: Ability to design sound cross-validation schemes and apply the Bootstrap to quantify the uncertainty of a performance estimate.

Part 2: The Parametric Linear Story

Lesson 5: Linear Regression and Maximum Likelihood
- ML Concept: Prediction of continuous targets via hyperplanes.
- Statistical Concept: The homoscedastic Gaussian noise model \(Y \mid X \sim \mathcal{N}(\beta^T X, \sigma^2)\), the construction of the likelihood function, and the mathematical proof that Ordinary Least Squares (OLS) and Maximum Likelihood Estimation (MLE) coincide under normality.
- Objectives: Establish maximum likelihood as the unifying principle underlying the entire parametric part of the course.
- Expected Competencies: Ability to write the likelihood of a linear model and derive the OLS solution as its maximizer.
Lesson 6: Logistic Regression and the GLM Framework
- ML Concept: Parametric binary classification.
- Statistical Concept: The Bernoulli distribution, the link function (logit) that maps a linear combination into a valid probability, and the derivation of cross-entropy loss directly from the Bernoulli log-likelihood.
- Objectives: Understand classification losses as log-likelihoods of a chosen response distribution, and place logistic regression within the wider GLM family.
- Expected Competencies: Ability to derive cross-entropy from first principles and generalize the construction to other exponential-family responses.
Lesson 7: Regularization and Bayesian MAP Estimation
- ML Concept: Ridge (\(L_2\)) and Lasso (\(L_1\)) penalties for complexity control.
- Statistical Concept: The transition from frequentist to Bayesian thinking (parameters as random variables), Maximum A Posteriori (MAP) estimation, and the equivalence of Ridge to a Gaussian prior and of Lasso to a Laplace prior (which induces sparsity).
- Objectives: Recognize regularization as prior belief made explicit rather than as an arbitrary penalty term.
- Expected Competencies: Ability to derive Ridge and Lasso as MAP estimators and explain geometrically why the Laplace prior produces exact zeros.
Lesson 8: The Theoretical Bias-Variance Tradeoff
- ML Concept: Mathematical analysis of the capacity and limits of learning algorithms.
- Statistical Concept: Formal decomposition of the expected mean squared error into three predictive components: squared bias, estimator variance, and irreducible error (\(\sigma^2\)).
- Objectives: Provide the analytical vocabulary used throughout Part 3 to explain why ensembles and boosting work.
- Expected Competencies: Ability to derive the decomposition and diagnose whether a given model is bias-limited or variance-limited.

Part 3: Advanced Models and Optimization

Lesson 9: Ensemble Theory and Variance Reduction
- ML Concept: Bagging and Random Forests.
- Statistical Concept: Statistical properties of the average of random variables, the variance formula for a sum of correlated variables (\(\rho\)), and how feature subsampling mathematically breaks that correlation.
- Objectives: Explain ensemble gains quantitatively through the variance term of the decomposition from Lesson 8.
- Expected Competencies: Ability to derive the variance of an averaged correlated ensemble and predict how it responds to changes in tree count and feature subsampling rate.
Lesson 10: Gradient Boosting and Numerical Optimization
- ML Concept: AdaBoost and Gradient Boosting.
- Statistical Concept: Sequential additive modeling aimed at bias reduction, approximation of loss functions via Taylor expansion, and the interpretation of residuals and pseudo-residuals as the negative gradient of the loss in function space.
- Objectives: Understand boosting as gradient descent performed in the space of functions rather than parameters.
- Expected Competencies: Ability to derive pseudo-residuals for an arbitrary differentiable loss and contrast the bias-reduction mechanism of boosting with the variance reduction of bagging.
Lesson 11: Support Vector Machines (SVM)
- ML Concept: Maximum-margin classifiers and the kernel trick.
- Statistical Concept: Constrained optimization via the Lagrangian dual formulation, non-probabilistic loss functions (Hinge Loss, which induces sparsity in the support vectors), and the need for subsequent statistical calibration (such as Platt scaling) to recover probabilities.
- Objectives: Examine a model deliberately built outside the likelihood framework and understand what is gained and lost by that choice.
- Expected Competencies: Ability to formulate the dual problem, identify support vectors, and calibrate SVM scores into usable probabilities.
Lesson 12: Neural Networks and Representation Learning
- ML Concept: Multilayer Perceptrons (MLPs) and backpropagation.
- Statistical Concept: Neural networks as a hierarchical, chained composition of GLMs; final activation functions (such as Softmax) defining the parameters of multinomial distributions; and gradient computation via the chain rule for the optimization of the global likelihood.
- Objectives: Close the course by showing deep networks as a stacked generalization of the parametric models studied in Part 2.
- Expected Competencies: Ability to describe an MLP as a composition of GLMs, derive backpropagation from the chain rule, and connect the output layer to an explicit likelihood.

Unsupervised Learning

Fri, 24 Jul 2026 00:00:00 +0000

This course explores the foundations of Unsupervised Machine Learning and Representation Learning through a rigorous probabilistic and statistical lens. Students will learn how to extract patterns, model underlying distributions, and discover hidden structures in unlabeled data using advanced statistical frameworks and information theory.

Final Objectives

The goal is to provide a deep understanding of how machine learning algorithms discover structure in data without human supervision. By the end of this course, students will be able to bridge algorithmic concepts like clustering, dimensionality reduction, and self-supervised learning with their theoretical statistical roots, including density estimation, generative modeling, and mutual information maximization.

Course Content

Click on each lesson to access its detailed planning and study materials.

Part 1: Density Structures and Clustering

Lesson 1: Data Space, Parametric Generative Models, and Anomalies (Slides)
- ML Concept: What it means to “model data” without labels. Profiling to identify atypical behaviors.
- Statistical Concept: Empirical vs. theoretical distribution. Assuming observed data was generated by a known theoretical distribution (e.g., Multivariate Gaussian). Using the likelihood of a point under the fitted model to define statistical exclusion regions (anomaly thresholds based on probability tails).
- Objectives: Understand how to fit theoretical distributions to unlabeled data and use likelihood for anomaly detection.
- Expected Competencies: Ability to fit a Multivariate Gaussian to data and define robust, probability-based anomaly thresholds.
Lesson 2: Vector Spaces, KNN, and Kernel Density Estimation (KDE)
- ML Concept: High-dimensional nearest neighbor search (ANN) and local density estimation.
- Statistical Concept: The curse of dimensionality and concentration of measure (analysis of \(L_1\), \(L_2\), and Cosine metrics). The core of the lesson: showing that the distance to the \(k\)-th nearest neighbor, \(d_k(x)\), is inversely proportional to the local density (\(p(x) \propto 1/d_k(x)^d\)). Transitioning from this hard count to KDE, where density is smoothed by a local Gaussian weight function.
- Objectives: Grasp the geometric behavior of high-dimensional spaces and smooth local density estimation.
- Expected Competencies: Ability to evaluate distance metrics in high dimensions and implement KDE to estimate underlying data distributions.
Lesson 3: Density Topography and Graphs: Hierarchical Clustering and HDBSCAN
- ML Concept: Identifying clusters of arbitrary shapes and isolating stochastic noise.
- Statistical Concept: Non-parametric estimation of density level sets (\(p(x) \ge \lambda\)). Using the \(k\)-neighbor distance property (Lesson 2) as a density metric to build graph paths (Minimum Spanning Tree). The concept of statistical persistence in HDBSCAN to cut the density tree where clusters are real rather than sample fluctuations.
- Objectives: Connect density estimation to graph theory to isolate robust, arbitrarily shaped clusters.
- Expected Competencies: Ability to apply HDBSCAN, interpret the cluster hierarchy, and distinguish between true clusters and stochastic noise based on statistical persistence.
Lesson 4: Gaussian Mixture Models (GMM) and the EM Algorithm
- ML Concept: Probabilistic clustering (Soft Clustering) for overlapping sample spaces.
- Statistical Concept: KMeans viewed as the limiting case of a GMM with identical spherical covariances and variance approaching zero. Introduction to Latent Variables (from which hidden population the point was born). The Expectation-Maximization (EM) algorithm to infer posterior probabilities (E-Step) and update parameters via weighted maximum likelihood (M-Step).
- Objectives: Understand latent variable models and iterative probabilistic optimization via EM.
- Expected Competencies: Ability to implement a GMM, mathematically derive the E and M steps, and explain KMeans as a constrained GMM.
Lesson 5: Model Selection and Introduction to Variational Inference
- ML Concept: Choosing the optimal number of components (\(K\)) and evaluating latent models.
- Statistical Concept: The collapse of pure likelihood in complex models. Using BIC as an asymptotic approximation of Model Evidence. Introducing Variational Inference and KL Divergence as tools to bypass the intractability of true Evidence, establishing the Evidence Lower Bound (ELBO) as the definitive criterion for selecting and fitting models with hidden variables.
- Objectives: Evaluate complex models rigorously and introduce the mathematical foundations of variational approximations.
- Expected Competencies: Ability to use BIC for GMM selection and mathematically formulate the ELBO.

Part 2: Dimensionality Reduction and Self-Supervision

Lesson 6: The Linear World: Traditional PCA, PPCA, and Linear Autoencoders
- ML Concept: Linear dimensionality reduction and feature projection.
- Statistical Concept: Maximizing projected variance via spectral decomposition of the sample covariance matrix. Probabilistic PCA (PPCA) formulated as a generative model of linear Gaussian latent variables with isotropic noise. Mathematical proof that the linear Autoencoder shares the same projection subspace as PCA.
- Objectives: Bridge algebraic projections (PCA) with probabilistic generative models (PPCA) and neural architectures.
- Expected Competencies: Ability to perform PCA via eigendecomposition, formulate PPCA, and prove its geometric equivalence to a linear autoencoder.
Lesson 7: The Non-Linear World: Traditional and Variational Autoencoders (VAE)
- ML Concept: Non-linear dimensionality reduction and deep data generation.
- Statistical Concept: Introducing non-linear activation functions creating complex network bottlenecks. Applying Variational Inference and the ELBO (learned in Lesson 5) in the non-linear scenario (VAE): using the reparameterization trick to force the deep latent space to conform to a clean Gaussian prior, without needing to focus on algebraic derivation massacres.
- Objectives: Extend latent variable modeling to non-linear deep networks using variational approximations.
- Expected Competencies: Ability to explain the architecture of a VAE, apply the reparameterization trick, and understand how to optimize the ELBO using gradient descent.
Lesson 8: Reduction for Visualization: MDS and t-SNE
- ML Concept: Collapsing high dimensions into interpretable 2D or 3D visual maps.
- Statistical Concept: Proximity graph optimization. The contrast between preserving raw geometric distances (Stress in MDS) and preserving probabilistic neighborhood structures (t-SNE). Using asymmetric KL Divergence combined with the heavy-tailed Student’s t-distribution to solve the crowding problem.
- Objectives: Understand the mathematical objectives behind topological dimensionality reduction algorithms.
- Expected Competencies: Ability to distinguish when to use MDS vs. t-SNE and explain how heavy-tailed distributions prevent crowding in low dimensions.

Part 3: Information Discovery and Large-Scale Structures

Lesson 9: Bayesian Networks and Probabilistic Graphical Models (PGMs) for Frequent Item Mining
- ML Concept: Association rules (Support, Confidence, Lift) in massive and sparse binary data.
- Statistical Concept: Translating business rules into joint probabilities, conditionals, and independence tests. Using PGMs to model directed dependencies, isolating and eliminating spurious associations caused by latent confounding variables.
- Objectives: Model complex categorical dependencies and distinguish causal structures from spurious correlations.
- Expected Competencies: Ability to extract association rules, construct a basic Bayesian Network graph, and identify conditional independencies.
Lesson 10: Recommendation Systems: Matrix Factorization (SVD and NMF)
- ML Concept: Collaborative filtering and preference prediction in matrices with missing data.
- Statistical Concept: Geometric and additive approach for hidden factors. Presenting Truncated SVD for matrix approximation and NMF (Non-Negative Matrix Factorization) focused on the statistical intuition that positivity constraints completely change the interpretation of components (additive parts vs. abstract combinations that cancel out).
- Objectives: Formulate recommendation as an optimization problem over latent matrix factors.
- Expected Competencies: Ability to implement collaborative filtering via SVD and explain how NMF leads to interpretable, parts-based representations.
Lesson 11: Distribution Alignment and Domain Shift (Data Drift) Detection
- ML Concept: Monitoring models in production and ensuring stochastic data consistency.
- Statistical Concept: High-dimensional non-parametric hypothesis testing to evaluate whether two population samples belong to the same underlying PDF (\(P(X_{\text{train}}) \stackrel{?}{=} P(X_{\text{test}})\)). Introduction to Maximum Mean Discrepancy (MMD) using kernel tricks to compare statistical moments directly in Hilbert spaces.
- Objectives: Rigorously detect data drift using advanced kernel-based statistical tests.
- Expected Competencies: Ability to apply MMD to detect multivariate domain shift between training and production data streams.
Lesson 12: Foundations of Contrastive Representation Learning
- ML Concept: Extracting meaning and creating robust feature vectors (embeddings) from unstructured data without human supervision.
- Statistical Concept: Optimization grounded in Information Theory. Using the InfoNCE loss to maximize the lower bound of Mutual Information between different stochastic views and perturbations of the same data object.
- Objectives: Understand modern self-supervised learning through the lens of mutual information maximization.
- Expected Competencies: Ability to formulate contrastive learning objectives and mathematically explain how InfoNCE approximates mutual information.