from first principles Probability Theory for ML

A probability space is the mathematical bedrock on which everything in statistics and ML rests. It gives a rigorous answer to the question: what is a random experiment?

A random variable is not actually a variable in the usual sense — it is a function from the sample space to the real numbers. This formalism connects the abstract probability space to measurable quantities.

In Machine Learning, probability distributions form the backbone of our loss functions and generative models. When a neural network predicts a class, it outputs a Probability Mass Function (PMF) via a softmax layer. When a variational autoencoder learns the latent space of images, it models a continuous Probability Density Function (PDF).

Machine Learning is fundamentally the science of optimizing expectations. The loss function we minimize during training (Empirical Risk Minimization) is simply a sample-based approximation of the expected loss over the true data distribution. Furthermore, when our models make predictions under uncertainty, they frequently output the expected value of the target.

While expectation tells us our model's average prediction, variance measures its uncertainty and instability—central to understanding the famous bias-variance tradeoff. Covariance extends this to multiple dimensions, forming the foundation of feature correlation, Principal Component Analysis (PCA), and the geometry of multivariate data.

Conditional probability is the mathematical formulation of supervised learning. When we train a discriminative model, we are not trying to learn the distribution of everything; we are specifically trying to model the conditional probability \(P(Y|X)\)—the probability of a target \(Y\) given our observed features \(X\).

Bayes' Rule is the engine of Bayesian ML. It provides a principled mathematical framework for how an AI agent should update its beliefs (its model parameters or weights) as it observes new training data. Instead of finding a single "best" parameter set, Bayesian inference finds an entire distribution of plausible parameters.

A joint distribution models the entire universe of our data variables simultaneously, \(P(X_1, X_2, \ldots, Y)\). While discriminative models focus solely on the conditional, generative AI—such as Variational Autoencoders, GANs, or Diffusion Models—attempts the much harder task of learning this complete joint distribution, allowing us to sample entirely new, synthetic data points.

Independence is the most common (and often violated) assumption in data science. The fundamental "i.i.d." assumption—that training examples are independent and identically distributed—allows us to multiply probabilities and sum log-likelihoods. Furthermore, conditional independence is what makes complex graphical models (like Naive Bayes or Hidden Markov Models) computationally tractable.

These distributions are our algorithmic building blocks. The choice of a target distribution directly dictates the loss function we use: assuming Gaussian noise leads to Mean Squared Error (MSE), assuming a Bernoulli distribution leads to Binary Cross-Entropy, and a Categorical distribution gives us Softmax Cross-Entropy.

The Law of Large Numbers formalizes the intuition that averages stabilize with more data. It is the mathematical foundation of Monte Carlo methods, empirical risk minimization, and every statistical estimator.

The Central Limit Theorem (CLT) is the reason the Gaussian distribution appears everywhere in nature and algorithms. In Deep Learning, it justifies why aggregated gradients in a mini-batch behave predictably, why weight initialization schemes assume normality, and why ensemble methods like Random Forests successfully reduce model variance.

Concentration inequalities bound the probability that a random variable deviates far from its mean. They are the mathematical core of statistical learning theory — generalization bounds, PAC learning, and bandit algorithms all rely on them.

Every ML algorithm has a probabilistic interpretation. Recognizing this reveals what assumptions each method implicitly makes, how to improve it, and how to quantify uncertainty in its predictions.

The complete story as one coherent thread:

1. A probability space \((\Omega, \mathcal{F}, \mathbb{P})\) formalizes randomness: outcomes, measurable events, and a measure satisfying Kolmogorov's three axioms. Every probability rule you've ever used is a theorem derived from these axioms.
2. Random variables are measurable functions \(X: \Omega \to \mathbb{R}\). They carry distributions — described by PMFs (discrete) or PDFs (continuous) — and their CDFs always exist and completely characterize the distribution.
3. Expectation is the Lebesgue integral of \(X\) against \(\mathbb{P}\). It is linear (always), monotone, and satisfies LOTUS (no need to find the distribution of \(g(X)\) to compute \(\mathbb{E}[g(X)]\)). Jensen's inequality connects convexity to expectation inequalities.
4. Variance and covariance measure spread and co-movement. The covariance matrix is symmetric PSD. PCA diagonalizes it. Every loss landscape in ML has a Hessian which is a covariance matrix in disguise.
5. Conditional probability and Bayes' rule follow from the product rule. The chain rule of probability factorizes any joint distribution into conditionals — exactly what autoregressive LMs (GPT) compute when generating text token by token.
6. The Law of Large Numbers guarantees empirical averages converge to expectations. This justifies SGD, mini-batch training, empirical risk minimization, and Monte Carlo integration — the entire computational machinery of ML.
7. The Central Limit Theorem tells us that sums of independent random variables are approximately Gaussian, regardless of the original distribution. This justifies Gaussian likelihood assumptions and enables statistical inference on model metrics.
8. Concentration inequalities — Markov, Chebyshev, Hoeffding, sub-Gaussian — give finite-sample bounds. They are the core of PAC learning theory: "with \(n\) samples and probability \(1-\delta\), the generalization gap is at most \(O(\sqrt{\log(1/\delta)/n})\)."
9. Every ML model is a probabilistic model: the loss function is the negative log-likelihood, regularization is a log-prior, training is MLE or MAP, and the full Bayesian treatment gives a posterior over parameters and an honest predictive distribution.