building thoughtful ai systems

it seems like a key to understanding where we are now in the ai revolution involves understanding the historical precedence and paradigm shifts of ai, from expert systems and symbolic ai (i.e. hardcoded cases) to supervised machine learning (training on top of human-based feature identification), unsupervised learning (i.e. edge detection within neural nets that were not explicit), self-supervised representation learning (i.e. BERT/FLAN missing-piece language prediction), and finally, the GPT-moment. at where we are now, our most interesting questions look at how we can shape model behavior after pre-training (i.e. in the post-training process) through reward models. the primary two being RLHF (reinforcement learning with human feedback, which requires human annotations of preferring option A or B), and reasoning RL (which removes human-labeled data in fields with verifiable rewards, such as math or coding where test cases passed can indicate success.)

knowing the history and techniques, i find that what we "owe" machines poses an interesting question. in lecture, five paradigms of ways we should treat models were introduced: good teaching (i.e. constitutional ai rather than benchmark optimization like the SAT or IMO scores), care (investigating WHY a model breaks, rather than patching up symptoms), honesty (understanding the drivers behind model dev, such as publication), patience (iteration even w/failure), and imagination (treating philosophers/artists/psychologists at the same level as engineers.)

imagination is an interesting point here, because at the core of this pillar (and the involvement of psychologists, artists, writers, philosophers etc.) is this fundamental question: what is it that we really want models to be able to do? personally, i DO want models to be able to handle menial coding tasks that i don't particularly care about doing in favor of the product and the creative idea behind it (think: vibe coding this website haha!). i DO want it to help find patterns in data that accelerate cancer drug discovery, or perhaps help outline some good must-see spots when i'm planning my vacation to new york. i would also love a personalized teaching assistant that poses questions and guides learning in a way that fundamentally optimizes my OWN learning. what i DON'T want AI to do is what i believe is soulless—which is taking out the "human" storytelling aspect of our world. i don't want it to create art for me that has some hidden deep meaning, because i don't think art by a machine has any meaning. i don't want it to philosophize about existence—i think we as the living creatures here have the fundamental right to that. and i CERTAINLY do not want AI to write and publish novels, because again—i believe that the "human experience" is what makes storytelling what it is. i don't care how good an ai writes a fictional sci-fi fantasy—if it didn't come from the thought process of a human, is it really a story at all? it lacks soul. i think ai is like any other innovation—a tool. and as questions are asked about what we want these models to be able to do, i believe it's imperative that we are optimizing its assistance and tool capabilities—not its human ones. (things like moltbook, this reddit-based social network for ai agents, scare me, for example.)

that note aside, i quite enjoyed what i found were incredibly good insights about the best practices in ai research at this day and age—particularly because they apply both to research and to daily life in general. my favorite point that the lecturers mentioned was "choosing the right problem and iterating fast." choosing the right problem is more of an art than it is a science, but by starting somewhere, iterating fast, and experimenting fast, seemingly small incremental improvements compound quickly. it's almost the bread and butter of post-training, this kind of "compound interest" mindset where small actions/tweaks bring incrementally better performance, and it is—in my opinion—what powers any kind of skillbuilding.

lastly, in the spirit of not turning these thoughts into an even larger rambling than it is, i love the way riccardo colleti broke down our lecture and will just list karina and kevin's research insights here again for my own convenience: (1) choose the right problem, (2) iterate fast, (3) design experiments for max information / evaluate whether something is worth burning compute on or not and how much, (4) pre-training is a moonshot, post-training is compound interest, (5) check the stack before blaming the model, (6) benchmarks steer the field, and (7) start from the user, not the benchmark / capabilities. (this reminds me a lot of my experience with amazon's customer obsession principles).

thanks for reading these ramblings! onto next week :)

this lecture has two big underlying ideas.

first of all, limitations in llm development are not necessarily what we think; rather than remaining strictly constrained by compute, llm advancement is largely dictated by the "hardware lottery" and the "memory wall," as described below:

• the "hardware lottery" dictates the direction of llm advancement. the uncomfortable truth reveals itself when we ask ourselves: are transformers, the current state of the art (sota) architecture, necessarily the best architecture for large reasoning models? we may never know, because the success or failure of research ideas often depends on what available hardware we have: in our case, NVIDIA GPUs. better ideas may never be explored.

• the "memory wall" is the largest bottleneck for efficiency. it's not how fast a GPU can compute matrix multiplications, nor is it their compute capacity, but rather how fast data can be fed to GPUs in the first place. in other words, memory accesses are the most costly, the prime bottleneck. that's why many of the most common optimization techniques developed in research are meant to skip re-loading data (KV caching), avoid unnecessary memory trips (FlashAttention), or optimize computations on data that is ALREADY loaded (batching).

the second big idea we talked about was the sota training pipeline for llms: (1) pre-training and (2) postraining, the latter being further split into supervised finetuning (SFT) -> preference optimization -> safety layers. i provide a quick rundown below:

• pretraining: the training process that takes months and is often done by frontier labs ($10M-$100M+). they create general capabilities, and decisions made during pretraining are baked into the weights...which means that there's no changing them later on (unless you re-train the entire model). three key decisions constrain downstream inference: attention variants (i.e. mult-headed attention, grouped-query attention), vocab size, and context length. luckily, we (as individuals without millions of dollars) are not too concerned with pre-training in this class, as we primarily have leverage over the post-training process :)

• SFT: takes the base, pre-trained model (which literally just predicts the next most likely word), and trains it on examples of desired behavior via thousands of high-quality human prompts and ideal responses. examples of desired behavior include conversational formatting (i.e. Q&A), when to stop generating etc.

• RLHF: method for aligning responses with human preferences. trains a separate reward model (based on humans ranking whether they like model_a or model_b's responses better) and then uses PPO to optimize the language model against this reward model. this pipeline requires FOUR models in memory: policy, reference, reward, AND value ... so it's a lot of memory being used. but it's very powerful nevertheless!

• DPO: super popular method proposed as an alternative to traditional RLHF (and a lot cheaper compared to PPO): instead of training another reward model, DPO directly optimizes the original language model based on (prompt, good_model, bad_model) input labels

• GRPO: a method specifically used in verifiable domains (i.e. coding, math, where there is a right answer) and requiring NO human labels (unlike RLHF and DPO). given a prompt, GRPO starts by generating a group of N responses, and then verifying which ones are correct. correct answers become "preferred," incorrect answers become "not preferred," and these outcome-based pairs are then used to train the model.

overall, this lecture was a great high-level overview of modern architectures and machine learning pipelines for training llms. i hear a lot of these terms (SFT, PPO, DPO etc.) thrown around in my lab (shameless plug for sky lab!) so it's nice to have a big picture that connects all of their functionalities together.

takeaways from this lecture:

• prediction is compression, and compression requires the ability to understand and model the world. kolmogorov complexity provides us with a theoretical limit of compression. you can think of it as the minimum description or smallest program needed to completely capture some [string] x

• math, logic, code, grammar, syntax, and factual knowledge (i.e. rule-based methods) are easy to compress (low K). emotional nuance, creative taste, and individual voice (i.e. per-instance and unique traits) are difficult to compress (high K)

• during model training, next-token prediction loss rewards easily-compressible patterns first (i.e. ones with low K) before learning more difficult patterns (i.e. ones with high K)

• the bias-variance tradeoff from classical ML is somewhat misleading (see double descent)--at some point, the more complexity (i.e. the more parameters and the more data), the more effective modeling and compression. in simple words, "scale works." instead of being explicitly taught physics, the model learns physics because physics helps with prediction.

• however, there is a ceiling to this idea of "scale", as instances that have extremely high Kolmogorov complexity often cannot be adequately compressed no matter how grand our scale is. (in lecture, we hypothesized that this area may be where the most distinctly-human capabilities live)

• post-training is all about compressing a specific subset of what you actually want. i won't go into details here, since my notes describe the post-training landscape (i.e. SFT, RLHF, DPO, CAI, RL on Verifiable Rewards, GRPO) far better than i can here. the biggest takeaway is that each method (listed previously) has its own purpose, benefits, and drawbacks. see my lecture notes for more, :)

kullback–leibler (KL) divergence is the bread and butter of modern llm post-training. it essentially measures the "dissimilarity" between two distributions P and Q, and serves as a pretty essential regularization mechanism to ensure that even when models follow reward signals, they don't deviate too far from their general capabilities. if a model did not have a KL constraint, it could lead to a myriad of problems including but not limited to drifting, collapsing, losing language fluency, producing unnatural output, and reward hacking--all in an attempt to maximize some reward signal. the KL constraint essentially forces the model not to stray too far from its original, pre-trained behavior.

KL divergence can be mathematically expressed as the following:

where we can interpret $\text{KL}(P || Q)$ as the extra bits needed when using distribution $Q$ to encode data actually drawn from $P$. almost all post-training methods implicitly or explicitly manage some kind of KL budget, and we can see that in SFT (forward KL) and RLHF (reverse KL), amongst other methods.

due to the asymmetric nature of KL divergence, two distinct usages arise:

(1) forward KL can be represented as $\text{KL}(P_\text{data} || \pi_\text{model}$). we can interpret this as sampling from $P_\text{data}$ and asking ourselves, "for each data point, does my model $\pi_\text{model}$ assign a high probability?" if no, (i.e. there exists mass at some part of $P_\text{data}$ but not in $\pi_\text{model}$), then the penalty is large. this effectively encourages broad, generic coverage for our model. (in other words, forward KL is data-driven: to minimize the penalty, our model seeks to cover as many data points as possible.)

(2) reverse KL can be represented as $\text{KL}(\pi_\text{model} || P_\text{target}$). we can interpret this as sampling from our model $\pi_\text{model}$, and penalizing our model for generating things that our reference distribution $P_\text{target}$ does not approve of. this results in narrow, mode-seeking behavior. (reverse-KL is model-driven. the model essentially makes an educated guess by sampling from $x\sim\pi_\text{model}$, and we only want to generate mass probability where our data actually exists.) the model is not encouraged full coverage of all regions; rather, it is encouraged to cover high-density regions of the target distribution $P_\text{target}$.

because supervised finetuning (SFT) loss utilizes forward-KL, we can say that it is data-driven. reinforcement learning with human feedback (RLHF) utilizes reverse KL, so we can say that it is model-driven.

one of the biggest debates today is whether RL actually "teaches" a model new things, versus sharpening what the model knows. one argument is that because RL concentrates mass on the correct paths that the model already has, it simply acts as a "sharpener"; the counterargument is that RL introduces emergent behaviors like self-verification and progressive lengthening of reasoning chains, so if one were to measure the reasoning steps of the model, they would discover behaviors that did not exist during the SFT checkpoint (see CoT-Pass@K, 2025 Wen et al).

regardless of which side ultimately gains favor (one could argue that the question of teaching v. sharpening is mainly a question of benchmarks), i think this is all great food for thought.

that's all i have for today! onto next week :)