Deep Learning · Self-Supervision · Mathematical Foundations
the geometry of meaningRepresentation Learning
From hand-crafted features to learned embedding spaces — Word2Vec's
vector arithmetic, autoencoders' compressive bottleneck, contrastive
learning's comparison-based objectives, SimCLR, CLIP, and the
foundation models that read meaning straight from geometry.
Feature LearningEmbeddingsWord2VecAutoencodersContrastive LearningSimCLRCLIPLatent SpacesGeometry of RepresentationsFoundation Models
// the feature engineering era · why raw data resists linear methods · learning φ(x) instead of designing it
Before a model can predict anything, the input must be transformed into a form where the relevant structure is accessible. The entire history of machine learning's progress can be told through one question: who designs that transformation — a human expert, or the data itself?
Why Raw Data Resists Direct Use
Raw pixel intensities, character codes, or audio samples are not linearly related to the high-level concepts we care about. "Catness" is not a linear function of pixel brightness values — two photos of the same cat in different lighting can be farther apart in raw pixel space than photos of entirely different objects. Any simple downstream model (linear classifier, nearest-neighbor) applied directly to raw input performs poorly, not because the model is weak, but because the representation hides the structure it needs.
The Hand-Engineering Era
Domain
Hand-Crafted Feature
What It Captured
Vision
SIFT (Scale-Invariant Feature Transform)
Local gradient orientation histograms, invariant to scale/rotation
Vision
HOG (Histogram of Oriented Gradients)
Edge direction distributions for object/pedestrian detection
Text
TF-IDF, n-grams
Term frequency weighted by corpus-wide rarity
Audio
MFCCs (Mel-Frequency Cepstral Coefficients)
Perceptually-motivated frequency-domain summary
Each of these required deep domain expertise, took years to refine, and — critically — did not transfer: a feature pipeline tuned for pedestrian detection told you nothing useful about handwriting recognition. Each new task demanded new hand engineering from scratch.
The Reframing: Learn φ, Don't Design It
Representation Learning — General ObjectiveCore Reframing
f_θ: X → R^d is the LEARNED representation function (encoder); g is typically a SIMPLE downstream function (often linear) applied on top. Both f_θ and g are trained jointly via gradient descent — backpropagation pushes the loss signal back through g into f_θ, sculpting the representation to make g's job easy. No human designs φ; gradient descent discovers it.
The Operational Definition of "Good"
✦
What Makes a Representation Good
Bengio, Courville & Vincent (2013) crystallized the modern criterion: a representation is good if a simple function — ideally linear — on top of it solves many different downstream tasks well. This single sentence reframes the entire goal of representation learning: not "solve task X," but "build a coordinate system in which task X, Y, and Z all become easy." Every method in this masterclass — Word2Vec, autoencoders, contrastive learning, CLIP — is best understood as a different strategy for achieving exactly this property.
Hierarchical Feature Composition
Deep networks learn features hierarchically: Zeiler & Fergus's (2014) visualizations of trained CNN filters showed early layers detecting edges and color gradients, middle layers detecting textures and simple parts (eyes, wheels), and late layers detecting whole objects — each layer's representation built by composing the previous layer's. This compositional structure is itself a form of representation learning: every layer is \(\phi_\ell = f_\ell(\phi_{\ell-1})\), progressively re-representing the input into a space where the final classification layer's job (linear separation) becomes trivial.
02
Object
Embeddings — The Mathematical Object
// formal definition · one-hot vs dense · the embedding matrix · choosing a similarity measure
An embedding is the single most important mathematical object in representation learning: a map from a discrete or structured domain into a continuous vector space, chosen so geometry in the vector space reflects meaningful structure in the original domain.
V is the original domain (a vocabulary, a set of users, a set of images). sim_V is whatever notion of "similar" matters in that domain (synonymy, co-purchase behavior, visual resemblance). The embedding's entire job is to make this domain-specific similarity computable via simple geometric operations (dot product, cosine, Euclidean distance) in R^d.
One-Hot vs Dense: Why Low Dimension Helps
One-Hot Encoding
Vocabulary of size \(|\mathcal{V}|\): each item is a vector with a single 1 and \(|\mathcal{V}|-1\) zeros. Sparse, high-dimensional, and — critically — every pair of distinct items has dot product exactly 0. There is no notion of similarity whatsoever; "cat" and "dog" are exactly as dissimilar as "cat" and "asphalt."
Dense Embedding
Maps to \(\mathbb{R}^d\) with \(d \ll |\mathcal{V}|\) (e.g., \(d=300\) for a 1-million-word vocabulary). Every dimension is informative; nearby points in this continuous space genuinely share meaning. Crucially, dense embeddings are differentiable — gradients flow smoothly, unlike the discrete, ungraded structure of raw symbols.
The Embedding Matrix and Lookup
Embedding Matrix — ImplementationMechanics
\[\mathbf{E} \in \mathbb{R}^{|\mathcal{V}|\times d}, \qquad \mathbf{e}_i = \mathbf{E}^T\mathbf{x}_i \quad\text{where }\mathbf{x}_i\text{ is the one-hot vector for item }i\]
Multiplying by a one-hot vector simply selects row i of E — in practice this "lookup" is implemented as direct array indexing rather than an actual matrix multiply, for efficiency. E itself is a learned parameter matrix, updated by backpropagation exactly like any other weight matrix in the network.
Choosing a Similarity Measure
Dot product \(\mathbf{u}^T\mathbf{v}\): sensitive to both direction and magnitude — large-norm vectors dominate, which can be desirable (norm encodes confidence/frequency) or undesirable (norm is an artifact of training dynamics).
Cosine similarity \(\mathbf{u}^T\mathbf{v}/(\|\mathbf{u}\|\|\mathbf{v}\|)\): normalizes away magnitude, comparing only direction — the standard choice when embedding norm is known to correlate with something other than semantic content (e.g., word frequency in Word2Vec).
Euclidean distance \(\|\mathbf{u}-\mathbf{v}\|\): the natural choice when the embedding space is trained to literally be metric (e.g., via a triplet or contrastive loss with explicit distance targets).
03
Distributional
Word2Vec — Distributional Semantics to Vector Arithmetic
// the distributional hypothesis · skip-gram with negative sampling · why analogy arithmetic works
Word2Vec (Mikolov et al. 2013) is the canonical demonstration that a simple, scalable training objective can produce embeddings with strikingly rich linear structure — including the famous analogy arithmetic that first convinced the field representation learning was onto something fundamental.
The Distributional Hypothesis
"
Firth, 1957
"You shall know a word by the company it keeps." Operationalized: words that tend to appear in similar contexts tend to have similar meaning. Word2Vec turns this linguistic intuition directly into a training signal — predict a word's context, and in doing so, words with similar contexts are pushed toward similar vectors purely as a side-effect of the prediction objective.
Skip-Gram With Negative Sampling — Full Derivation
The denominator sums over the ENTIRE vocabulary — for |V| ≈ 1 million words, this softmax must be evaluated (and differentiated) for every single training pair. Computationally infeasible at scale.
// Negative sampling reframes softmax classification as binary classification
1
Instead of predicting the exact context word out of all \(|\mathcal{V}|\) candidates, train a binary classifier: given a (center, context) pair, predict whether it came from the real data or was randomly sampled "noise."
2
For a true pair \((w_t,w_c)\), sample \(K\) noise words \(w_1,\ldots,w_K \sim P_n\) (typically \(P_n \propto \text{unigram frequency}^{3/4}\)) and maximize:
\[J = \log\sigma(\mathbf{v}_{w_c}^T\mathbf{v}_{w_t}) + \sum_{k=1}^K\mathbb{E}_{w_k\sim P_n}\left[\log\sigma(-\mathbf{v}_{w_k}^T\mathbf{v}_{w_t})\right]\]
3
This is exactly logistic regression (covered in the Logistic Regression masterclass) distinguishing real pairs from sampled noise — the softmax over a million-word vocabulary is replaced with \(K+1\) sigmoid evaluations (typically \(K=5\) to \(20\)).
Mikolov's negative sampling trick converts an O(|V|) operation into an O(K) operation per training step — the single optimization that made training on billion-word corpora tractable. ∎
Why Vector Arithmetic Works — The Matrix Factorization View
Word2Vec as Implicit Matrix FactorizationLevy & Goldberg, 2014
\[\text{At the optimum of SGNS: } \quad \mathbf{v}_w^T\mathbf{v}_c \approx \text{PMI}(w,c) - \log k\]
\[\text{PMI}(w,c) = \log\frac{P(w,c)}{P(w)P(c)} \quad\text{(pointwise mutual information)}\]
Setting the gradient of the SGNS objective (per word-context pair) to zero and solving shows that the optimal dot product v_w·v_c equals the (shifted) PMI between w and c. Word2Vec is therefore IMPLICITLY factorizing a shifted-PMI matrix — connecting it directly to the SVD/PCA masterclass's low-rank matrix factorization machinery. This is a profound result: a "neural" word embedding method turns out to be doing classical matrix factorization under the hood.
The Analogy Arithmetic, Explained
Why king − man + woman ≈ queen
Because \(\mathbf{v}_w\cdot\mathbf{v}_c \approx \text{PMI}(w,c)-\log k\), the vector difference \(\mathbf{v}_{\text{king}}-\mathbf{v}_{\text{man}}\) approximately captures the PMI difference between (king, ·) and (man, ·) across all contexts \(\cdot\) — roughly, "the contextual shift associated with royalty, independent of gender." Empirically, this contextual shift is approximately consistent across many gender pairs (man/woman, king/queen, actor/actress) because the underlying co-occurrence statistics encode gender as a roughly fixed, additive contextual offset. This is not an engineered property — it falls out of the corpus's statistics interacting with the linear (dot-product, hence additive-in-log-space) form of the objective. It is also why this linear structure is fragile: any semantic relationship NOT well-approximated by an additive PMI shift in the training corpus will not produce clean analogy arithmetic.
04
Compression
Autoencoders — Compression as Representation
// encoder-decoder bottleneck · linear AE ≡ PCA · denoising · the VAE bridge
If Word2Vec learns representations by predicting context, autoencoders learn them by a different, equally elegant trick: force a network to reconstruct its own input through an information-limiting bottleneck, and the bottleneck activations become a learned, compressed representation.
If z could simply copy x (k ≥ dim(x), identity mapping available), the network would trivially achieve zero loss without learning anything useful. The bottleneck k ≪ dim(x) is what FORCES the network to discover and retain only the data's most important structure — assuming the data has lower intrinsic dimensionality than its raw representation (the manifold hypothesis, formalized in §08).
Theorem: Linear Autoencoders Recover PCA
Baldi & Hornik, 1989
If \(f_{\text{enc}}\) and \(f_{\text{dec}}\) are both linear (\(\mathbf{z}=\mathbf{W}_1\mathbf{x}\), \(\hat{\mathbf{x}}=\mathbf{W}_2\mathbf{z}\)) with bottleneck dimension \(k\), and the loss is MSE reconstruction error, then at the global optimum, the row space of \(\mathbf{W}_1\) spans the SAME \(k\)-dimensional subspace as the top-\(k\) principal components of the data (up to an arbitrary rotation within that subspace). A linear autoencoder is, in this precise sense, exactly PCA — bridging this section directly to the PCA masterclass's eigendecomposition of the covariance matrix.
The practical implication runs in both directions: PCA is the special case of representation learning where the encoder/decoder are linear and the loss is MSE; nonlinear autoencoders (with ReLU/tanh activations in \(f_{\text{enc}},f_{\text{dec}}\)) generalize PCA to learn curved manifolds rather than only flat linear subspaces.
Denoising Autoencoders
Denoising Autoencoder ObjectiveVincent et al. 2008
Forcing reconstruction of clean data from corrupted input prevents the trivial near-identity solution and forces the network to learn the data's robust, underlying structure rather than memorizing exact pixel values — a regularization effect closely related to why data augmentation helps supervised training, and a direct conceptual precursor to masked-prediction pretraining objectives used in modern foundation models (§10).
Bridge: Variational Autoencoders
Replacing the deterministic encoder with a probabilistic one, \(q_\phi(\mathbf{z}|\mathbf{x})\), and adding a prior \(p(\mathbf{z})\) (typically \(\mathcal{N}(0,\mathbf{I})\)) turns the autoencoder into a Variational Autoencoder, trained by maximizing the Evidence Lower Bound covered in full in the Bayesian Machine Learning masterclass:
The first term is reconstruction quality (as in a vanilla AE); the second is a regularizer pulling each input's latent distribution toward a shared prior — this is precisely what makes VAE latent spaces smooth and suitable for interpolation, a property explored further in §08.
05
Comparison
Contrastive Learning — Learning by Comparison
// pull positives, push negatives · InfoNCE · the mutual-information lower bound · temperature
A third strategy for learning representations abandons reconstruction entirely. Instead of asking "can you rebuild the input," contrastive learning asks a comparative question: "can you tell which of these items belongs together?"
"Semantically related" is defined however the task demands: two augmented crops of the same image, an image and its matching caption, a sentence and its paraphrase. The negatives are simply everything else in the batch — no labeled "negative" examples are required, making this a self-supervised technique.
InfoNCE — Derivation as a Mutual Information Bound
// Oord, Li & Vinyals, 2018 — Contrastive Predictive Coding
1
Define the InfoNCE loss for one anchor with one positive among \(N\) total candidates (1 positive + \(N-1\) negatives):
\[\mathcal{L}_{\text{InfoNCE}} = -\log\frac{\exp(\text{sim}(\mathbf{z},\mathbf{z}^+)/\tau)}{\sum_{i=1}^N\exp(\text{sim}(\mathbf{z},\mathbf{z}_i)/\tau)}\]
2
This is structurally identical to a softmax cross-entropy loss over \(N\) classes, where the "correct class" is the positive. Minimizing it is precisely a classification task: identify the true positive among the distractors.
3
Oord et al. proved that minimizing \(\mathcal{L}_{\text{InfoNCE}}\) is equivalent to maximizing a lower bound on the mutual information between the anchor and the positive:
\[I(\mathbf{x};\mathbf{x}^+) \geq \log N - \mathcal{L}_{\text{InfoNCE}}\]
As N (the number of negatives) grows, this lower bound tightens — directly explaining the empirical observation that contrastive methods improve with MORE negatives per positive, a recurring theme that motivates SimCLR's large batch sizes in §06. ∎
The Role of Temperature τ
τ
A Familiar Variance-Control Knob
Dividing similarities by \(\tau\) before the softmax plays exactly the same role as the \(1/\sqrt{d_k}\) scaling in attention (covered in the Attention Mechanisms masterclass): it controls how sharply the softmax concentrates. Small \(\tau\) → a peaked distribution that focuses gradient signal almost entirely on the hardest negative (the one closest to the positive); large \(\tau\) → a smoother distribution that treats all negatives more equally. Choosing \(\tau\) is a bias-variance tradeoff between hard-negative emphasis and training stability.
06
Vision
SimCLR — Contrastive Learning for Images
// augmentation as positive-pair generation · the projection head trick · NT-Xent loss in full
SimCLR (Chen et al. 2020) instantiates the contrastive framework of §05 for images with a deceptively simple recipe: generate positive pairs purely through data augmentation, with no labels at all.
The Full Pipeline
Fig 1. SimCLR pipeline: one image augmented into two views, passed through a shared encoder f, then a projection head g, with the contrastive NT-Xent loss applied on the projected z's. After pretraining, g is discarded — only h (pre-projection) is kept for downstream tasks.
For a batch of N images, each generates 2 augmented views, giving 2N total samples. For each sample, the OTHER 2N−2 samples in the batch (not its paired view) all serve as negatives — no separate negative-sampling step is needed; the batch itself supplies the negatives. This loss is summed over BOTH directions of every positive pair (i→j and j→i) for symmetry.
Why Batch Size Matters So Much
Recall from §05 that the InfoNCE mutual-information bound tightens as the number of negatives \(N\) grows. In SimCLR, the negatives come exclusively from within the batch — so a larger batch directly means a tighter bound and a more informative gradient signal per step. This is why SimCLR's original results used batch sizes up to 8192, far larger than typical supervised training batches, and why follow-up work (MoCo) introduced a memory bank / momentum encoder specifically to decouple "many negatives" from "requires a huge batch."
Why the Projection Head Is Discarded
⊘
h, Not z, Transfers Best
SimCLR's empirical finding (confirmed across many follow-ups) is that linear-probe accuracy on downstream tasks is consistently HIGHER using \(\mathbf{h}\) (the encoder output, before projection) than \(\mathbf{z}\) (the projected output actually used in the contrastive loss). The explanation: \(g(\cdot)\) is optimized specifically to discard information the contrastive task doesn't need — e.g., to become invariant to color jitter — but that very information (color) might matter for some downstream task. \(\mathbf{h}\) retains more general-purpose information; \(\mathbf{z}\) over-specializes to the pretraining objective's particular invariances.
07
Multi-Modal
CLIP — Joint Embedding Across Modalities
// symmetric contrastive pretraining · the shared embedding space · zero-shot classification
CLIP (Radford et al. 2021) extends contrastive learning across an entirely different boundary: not two augmented views of the same image, but an image and its natural-language description, learned from 400 million (image, text) pairs scraped from the internet.
The Symmetric Contrastive Objective
CLIP — Bidirectional InfoNCETwo Encoders, One Space
\[\mathbf{I}_k = f_I(\text{image}_k), \quad \mathbf{T}_k = f_T(\text{text}_k), \quad \text{both projected into the same } \mathbb{R}^d\]
\[\text{logits}_{kl} = \frac{\mathbf{I}_k^T\mathbf{T}_l}{\|\mathbf{I}_k\|\|\mathbf{T}_l\|}\cdot\exp(t) \quad\text{(}t\text{ = LEARNED temperature parameter)}\]
\[\mathcal{L} = \frac{1}{2N}\sum_k\left[\text{CE}(\text{logits}_{k,:},\,k) + \text{CE}(\text{logits}_{:,k},\,k)\right]\]
For a batch of N pairs, build the full N×N similarity matrix between every image and every text. The diagonal entries are the true matches; cross-entropy is applied along rows (image→text direction) AND columns (text→image direction), then averaged — exactly the same symmetric structure as SimCLR's NT-Xent, but across two different encoders and two different modalities instead of two augmented views of one modality.
Zero-Shot Classification — Turning Labels Into Text
// Classification without ANY classification-specific training
1
For each candidate class \(c\) (e.g., "cat", "dog", "airplane"), construct a natural-language prompt: "a photo of a {c}".
2
Embed every prompt with the text encoder: \(\mathbf{T}_c = f_T(\text{"a photo of a "}+c)\).
3
Embed the query image: \(\mathbf{I} = f_I(\text{image})\). Classify by nearest neighbor in the shared space:
\[\hat{c} = \arg\max_c \frac{\mathbf{I}^T\mathbf{T}_c}{\|\mathbf{I}\|\|\mathbf{T}_c\|}\]
No classification head was ever trained for this specific label set — the SAME pretrained encoders, with NO gradient updates, perform competitively against fully-supervised classifiers trained explicitly on labeled data. This is a direct application of the "attention as kernel/retrieval" view from the Attention Mechanisms masterclass: classification has been reframed as a similarity search over a learned embedding space. ∎
Why a Shared Space Enables Cross-Modal Arithmetic
Because images and text are projected into the same vector space (not two separate spaces later glued together), cosine similarity between an image embedding and a text embedding is directly meaningful — enabling text-to-image retrieval ("find images matching this caption"), image-to-text retrieval ("find the best caption for this image"), and even rudimentary cross-modal vector arithmetic, all using nothing but dot products in one shared geometry.
08
Topology
Latent Spaces — Manifolds and Interpolation
// the manifold hypothesis · interpolation as a quality signature · disentanglement
Every method covered so far produces a "latent space" — but what, mathematically, are we hoping that space looks like? The manifold hypothesis gives the formal answer, and interpolation gives the empirical test.
The Manifold HypothesisFoundational Assumption
\[\text{Real-world high-dimensional data } \mathbf{x}\in\mathbb{R}^D \text{ concentrates near a low-dimensional manifold } \mathcal{M}\subset\mathbb{R}^D, \quad \dim(\mathcal{M})=d \ll D\]
The set of "all possible photographs of human faces" is a tiny, structured sliver of the full pixel space R^D (D = millions of pixel values) — most points in R^D look like pure noise, not faces. A latent space z ∈ R^d learned by an encoder is, ideally, a coordinate chart — a continuous parametrization — of (an approximation to) this manifold.
If the latent space is "well-behaved" (the manifold is approximately flat/convex in these learned coordinates), decoding along this straight line produces a SMOOTH, semantically meaningful morph between x1 and x2 — e.g., one face's pose, lighting, and identity continuously blending into another's. If the space is poorly structured, intermediate points decode to blurry or nonsensical outputs, since the straight line between z1 and z2 leaves the region the decoder was trained to handle well.
This single empirical test — does straight-line interpolation in latent space produce smooth, meaningful outputs — has become one of the standard qualitative diagnostics for representation quality across generative models (VAEs, GANs, diffusion models' latent spaces).
Disentanglement
⊥
One Dimension, One Factor
An ideally disentangled latent space has each dimension (or small subspace) correspond to a single, independent, semantically interpretable factor of variation — e.g., one dimension controls face pose, another controls lighting, a third controls identity, all independently adjustable. \(\beta\)-VAE (Higgins et al. 2017) encourages this by upweighting the KL term in the ELBO from §04, pushing the learned posterior closer to an independent (diagonal-covariance) prior — trading some reconstruction fidelity for cleaner factor separation.
Classical vs Learned Manifold Discovery
Method
Type
Key Idea
PCA
Linear, classical
Best flat (linear) approximating subspace
Isomap
Nonlinear, classical
Preserve geodesic (manifold) distances via shortest paths on a neighbor graph
t-SNE / UMAP
Nonlinear, classical, non-parametric
Preserve local neighborhood structure for 2D/3D visualization
Autoencoder / VAE
Nonlinear, learned, parametric
Neural network directly parametrizes the encoder/decoder maps
09
Measurement
Geometry of Representations
// cosine vs Euclidean · anisotropy and the cone effect · linear probing · CKA · the linear representation hypothesis
Once a representation exists, how do we measure whether it's any good — and how do we compare two different representation spaces to each other? This section covers the toolkit for studying representations as geometric objects in their own right.
Anisotropy — The Cone Effect
Embedding Space AnisotropyEthayarajh, 2019
\[\text{Isotropic expectation: } \mathbb{E}_{u,v\sim\text{random}}[\cos(u,v)] \approx 0 \quad\text{(for high-dim random vectors)}\]
\[\text{Observed in contextual embeddings (e.g., BERT): } \mathbb{E}[\cos(u,v)] \gg 0\]
If embeddings were spread isotropically (uniformly in all directions) across the space, two RANDOM embeddings would have cosine similarity near zero in high dimensions (concentration of measure). Ethayarajh found that contextual embeddings instead occupy a narrow CONE — most vectors point in roughly similar directions — meaning raw cosine similarity values are systematically inflated and less discriminative than they appear. Practical fix: center and whiten embeddings (subtract the mean, decorrelate) before computing similarities.
Linear Probing
Linear Probe — Representation Quality MetricOperational Test
\[\text{Freeze } f_\theta. \quad \text{Train only a linear head: } \hat{y} = \mathbf{W}f_\theta(\mathbf{x}) + \mathbf{b}\]
\[\text{Linear probe accuracy} \approx \text{how linearly accessible task-relevant info is in } f_\theta(\mathbf{x})\]
This directly operationalizes the Bengio et al. criterion from §01: a representation is "good" precisely to the extent that a SIMPLE (linear) readout — i.e., logistic regression, covered in its own masterclass — succeeds on many different probe tasks. Linear probing has become the standard benchmark protocol for comparing self-supervised representation learning methods (SimCLR, CLIP, masked autoencoders) against each other.
X and Y are two representation matrices (e.g., the same inputs passed through two different layers, or two different networks). CKA measures how much shared geometric structure exists between the two — invariant to orthogonal rotation and isotropic scaling of either space (since such transformations don't change the underlying geometric relationships between points). Used extensively to study whether different architectures, or different layers of the same network, converge to similar internal representations.
The Linear Representation Hypothesis
An Emerging Unifying Principle
A growing body of interpretability research (studying both word embeddings and large language model internals) suggests that many semantic concepts — not just gender, as in Word2Vec's analogies, but abstract concepts like sentiment, truthfulness, or even specific facts — are represented as roughly linear directions in activation space, rather than encoded in some hopelessly tangled nonlinear form. This hypothesis directly explains why linear probes (this section) work so well, why vector arithmetic works in Word2Vec (§03), and why simple techniques like "activation steering" (adding a learned direction vector to a model's internal activations) can reliably control model behavior. It is one of the most actively studied open questions connecting representation learning to mechanistic interpretability.
Every idea in this masterclass — distributional context prediction, compressive reconstruction, contrastive comparison — converges in modern foundation models: single large networks, pretrained once on broad data, whose representations are reused across a vast range of downstream tasks.
Foundation Model — DefinitionBommasani et al. 2021
\[\text{A single model } f_\theta \text{ trained on broad data, such that } f_\theta(\mathbf{x}) \text{ transfers to many downstream tasks}\]
\[\text{with minimal task-specific adaptation (zero-shot, few-shot, or lightweight fine-tuning)}\]
The defining property is BREADTH of transfer, not any specific architecture or training objective — a single pretrained representation function expected to serve as the substrate for an open-ended set of future tasks, many unknown at training time.
Pretraining Objectives — Representation Learning at Scale
Masked Language Modeling
predict masked tokens from context
BERT-style. A direct, deep-bidirectional generalization of Word2Vec's context-prediction idea (§03) to full sentences with attention-based context aggregation.
Autoregressive LM
predict next token given all previous
GPT-style. Every internal activation is, implicitly, a representation trained to be maximally useful for predicting what comes next — at every position, simultaneously.
Masked Image Modeling
reconstruct masked image patches
MAE-style. The denoising-autoencoder idea from §04, scaled up: mask large fractions of an image, reconstruct from the visible remainder.
Contrastive Image-Text
align matching (image, text) pairs
CLIP-style (§07). Self-supervision derived from naturally co-occurring multi-modal pairs, with no manual labeling at all.
Why Scale Changes Qualitative Behavior
↗
Emergent Linear Separability
As foundation models scale, downstream tasks increasingly become solvable with NO gradient updates at all — via zero-shot similarity (CLIP, §07) or via in-context learning, where a few examples placed directly in the input context are enough to adapt behavior. The Attention Mechanisms masterclass showed that in-context learning is mathematically equivalent to performing Nadaraya-Watson kernel regression live, at inference time, using the in-context examples as the "training set." Representation learning's entire historical arc — from hand-crafted features, through learned embeddings, to this point — culminates in representations rich enough that even a NONPARAMETRIC kernel readout, with no weight updates whatsoever, suffices.
The Readout Spectrum
Readout Method
Weight Updates?
Mechanism
Zero-shot
None
Direct similarity / prompting against the frozen representation
Linear probing
Only a new linear head
Train \(\mathbf{W}\) on top of frozen \(f_\theta(\mathbf{x})\) — directly tests representation quality (§09)
In-context learning
None
Condition on examples in the input; kernel-regression-like readout (Attention masterclass)
Fine-tuning
Some or all of \(\theta\)
Continue gradient descent on \(f_\theta\) itself for the specific downstream task
11
Synthesis
The Complete Mental Model
// everything unified · one diagram · the full thread from features to foundation models
Fig 2. Complete representation learning map — three foundational strategies (distributional, compressive, contrastive) converging through latent space geometry into foundation models and their spectrum of readout methods.
The complete story in one coherent thread:
Feature learning reframes the core ML problem: instead of a human hand-designing \(\phi(x)\), gradient descent discovers it — judged not by performance on one task, but by how many different SIMPLE downstream readouts it enables.
Embeddings are the basic mathematical object: maps from discrete or structured domains into continuous vector spaces, chosen so geometric operations (dot product, cosine, distance) recover meaningful similarity.
Word2Vec demonstrates that a simple context-prediction objective, trained at scale, implicitly performs PMI matrix factorization — and that the resulting linear structure (king − man + woman ≈ queen) is an emergent statistical regularity, not a designed property.
Autoencoders learn representations through a reconstruction bottleneck; the LINEAR case is provably equivalent to PCA, while nonlinear variants generalize this to curved manifolds, and the VAE extension connects directly to the Bayesian ELBO.
Contrastive learning replaces reconstruction with comparison: the InfoNCE loss is a softmax classification task that provably lower-bounds the mutual information between paired views, explaining why more negatives (bigger batches) consistently help.
SimCLR generates positive pairs purely via image augmentation, with the empirical twist that the pre-projection representation \(\mathbf{h}\) transfers better than the contrastively-trained \(\mathbf{z}\) itself.
CLIP extends contrastive learning across modalities, producing a single shared image-text space that turns classification into nearest-neighbor search and enables zero-shot transfer with no task-specific training at all.
Latent spaces are best understood through the manifold hypothesis; smooth interpolation and disentangled, independently-controllable factors are the empirical signatures of a well-structured learned coordinate system.
The geometry of representations — anisotropy, linear probing, CKA, and the linear representation hypothesis — gives the precise tools for measuring representation quality and comparing different learned spaces to each other.
Foundation models are the convergence point: all three learning strategies, deployed at massive scale, producing representations rich enough to be read out with zero weight updates — via similarity, prompting, or in-context kernel regression — completing the arc from hand-crafted features to representations that read meaning straight from geometry.
Representation Learning Is the Search for Useful Geometry
Every method in this masterclass, despite wildly different mechanics — predicting context, reconstructing through a bottleneck, contrasting positive against negative pairs — is solving the exact same underlying problem: build a coordinate system in which the structure that matters becomes geometrically simple. Distance should mean dissimilarity. Direction should mean meaning. Straight lines should mean smooth semantic change. When this geometry is built well enough, classification becomes nearest-neighbor search, analogy becomes vector subtraction, and even the seemingly mysterious in-context learning of foundation models reveals itself as ordinary kernel regression performed in a space where the hard work — learning what matters — was already done at pretraining time.
SHAHID||Portfolio