The Stanford University campus seen from Hoover Tower (photographed May 19, 2014). Photo: Pmlineditor. Source: Wikimedia Commons. License: CC BY-SA 4.0.
The latest offering of Stanford CS336 "Language Modeling from Scratch" is the Spring 2026 term. Its course website, YouTube lecture recordings and GitHub assignments are all public, so you can follow along for free. The five assignments begin with writing a tokenizer and a Transformer, then cover GPU systems optimization, scaling laws, pretraining data processing and alignment, in that order. The barriers to entry are solid Python and PyTorch skills, a large amount of coding time and GPU costs that you pay yourself. Public self-study does not come with a certificate.
Public Materials and Course Dates
The course website, cs336.stanford.edu, is currently labeled "Stanford / Spring 2026". It also keeps archive pages for the two previous spring offerings, 2025 and 2024. The Spring 2026 offering is taught by Percy Liang and Tatsunori Hashimoto. It starts on March 30 and runs for 19 sessions through June 3, and the last two sessions are guest lectures. The schedule attaches lecture materials to each of the first 17 lectures. Percy's lectures come as executable Python lecture notes (lecture_XX.py), and Tatsu's lectures come as PDF slides. The recordings are collected in the YouTube playlist "Stanford CS336: Language Modeling from Scratch | Spring 2026", and the website's schedule also has links next to the lecture titles.
The course code is hosted under the GitHub organization stanford-cs336. The lectures repository holds the lecture source code for this offering, and assignment1-basics through assignment5-alignment are the student versions of the five assignments. Lecture notes from earlier offerings are in two separate repositories, spring2024-lectures and spring2025-lectures. A GitHub check on October 22, 2026 showed that the course website repository's most recent commit was on June 1, 2026. The lectures repository's most recent push was on September 12, 2026, and assignment1-basics's most recent push was on April 7, 2026. The Assignment 1 repository uses the MIT License, but the lectures repository does not declare a license, so check separately before republishing or adapting the lecture materials. There is also one small discrepancy. The README titles for Assignments 2 through 5 all say Spring 2026, but the README title on Assignment 1's main branch still says Spring 2025. Before you start, use the code and preview links for Assignment 1 in the website's schedule as the authoritative version.
The Five Assignments and Their Lectures
The course is built around five assignments, and the lecture sequence largely follows them. The table below is compiled from the website's Spring 2026 schedule. The dates are the release and due dates for that offering, and the website notes that the due dates are tentative.
| Assignment | What you do | Corresponding lectures | Spring 2026 release—due |
|---|---|---|---|
| Assignment 1: Basics | Implement your own tokenizer, Transformer model architecture and optimizer, then train a small language model; the data are TinyStories and an OpenWebText subsample | Lectures 1–4: tokenization; PyTorch and resource accounting (FLOPs, GPU memory, arithmetic intensity); architecture and hyperparameters; attention alternatives and mixture of experts | March 30—April 15 |
| Assignment 2: Systems | Measure each layer of the model with profiling tools, write your own FlashAttention2 in Triton, then implement memory-efficient distributed training | Lectures 5–8: GPUs and TPUs; kernels and Triton; parallelism (two lectures) | April 15—April 29 |
| Assignment 3: Scaling | Understand the role of each Transformer component, then call a training API to fit scaling laws and extrapolate from them | Lectures 9 and 11: scaling laws | April 29—May 6 |
| Assignment 4: Data | Process raw Common Crawl dumps into usable pretraining data, including filtering and deduplication | Lectures 13–14: data sources; filtering, deduplication, mixing and synthetic data | May 6—May 20 |
| Assignment 5: Alignment and Reasoning RL | Use supervised fine-tuning and reinforcement learning to train a model to solve math problems; an optional second part implements safety alignment methods such as DPO | Lectures 15–17: mid-training and post-training (SFT/RLHF); RLVR; alignment and multimodality | May 20—June 3 |
The schedule also includes Lecture 10 on inference, Lecture 12 on evaluation and the two guest lectures on June 1 and 3 (Daniel Selsam and Dan Fu). These sessions do not map directly to an assignment. Self-learners may also find the website's rules for enrolled students useful. The assignment handouts are self-contained, so you should not look at existing implementations online unless a handout says otherwise. You may ask ChatGPT or similar tools low-level programming or conceptual questions, but you may not have them solve the problems for you. The course also strongly recommends turning off AI autocomplete tools such as Cursor Tab and GitHub Copilot, because they make it hard to understand the material deeply.
Prerequisites, Compute Costs and Certificates
The website lists five prerequisites:
- proficiency in Python, plus software engineering skills;
- experience with deep learning and systems optimization, familiarity with PyTorch and an understanding of basic systems concepts such as the memory hierarchy;
- college calculus and linear algebra, including the ability to read matrix and vector notation;
- basic probability and statistics, such as Gaussian distributions, means and standard deviations;
- machine learning fundamentals equivalent to one of Stanford's CS221, CS229, CS230, CS124 or CS224N.
The website specifically warns that the assignments provide almost no scaffolding and require at least an order of magnitude more code than other courses. CS336 is a 5-unit course, and Stanford Online lists the expected commitment as 10 weeks at 20–25 hours per week.
You have to arrange your own compute. The website lists public prices for a single B200 GPU as of March 28, 2026:
- Modal, the course sponsor: $6.25 per hour, plus $30 of free credit each month;
- Lambda: $6.69;
- RunPod: $4.99;
- Nebius: $5.50 ($3.05 for preemptible instances);
- Together: $7.49, with a minimum rental of 8 GPUs.
The website recommends getting your code correct on CPU first. After that, use GPUs for the training in Assignments 1, 4 and 5 and for the GPU benchmarking in Assignment 2, with the number of GPUs recommended for each assignment. Assignment 4 needs the most compute. Its README states that final training uses 8 B200s, which comes to about $50 per hour at Modal's list price. The actual training time depends on the assignment handout (to be verified). Assignment 3's hosted training API requires a Stanford student ID, so learners who are not enrolled must run the training scripts themselves by following the README and pay for that compute too. The course does not estimate the total cost of the whole course, so the total is still to be verified. These prices may also change as the platforms update them.
A compute cluster equipped with NVIDIA Tesla GPUs, shown only to illustrate GPU compute. These are not the B200s used for the CS336 assignments. Photo: ChrisDag. Source: Wikimedia Commons. License: CC BY 2.0.
The free public website, recordings and assignments do not come with a certificate. The website says enrolled students submit through Gradescope, and it does not mention any grading option for self-learners. If you need a formal credential, Stanford Online offers CS336 as a graduate course that you can apply to. Tuition is $7,875, although the page notes that it may change. The course is worth 5 units, and the credential is a Stanford University transcript. The course can count toward the Artificial Intelligence Graduate Certificate program, and the class dates for the 2026 offering are March 30 to June 10. As of October 22, 2026, the page shows that enrollment is currently closed and lists the spring quarter as "tentative". Because of compute and workload limits, enrollment uses a waitlist. Applicants must submit a non-degree application and a course-specific application. They must also have a bachelor's degree, an undergraduate GPA of at least 3.0 and credit for CS224N or a similar natural language processing course. The page does not state the application deadline or eligible regions for the next offering, so these details are still to be verified.
Suggested Self-Study Order
This site compiled the order below from the course schedule and each assignment's README. It is a suggestion, not an official course requirement.
- Cover the prerequisites first: If you are not yet comfortable with PyTorch, practice writing tensor operations, using autograd and building a complete training loop before you start CS336. Karpathy's eight-lecture neural network series, which this site has covered before, can serve as a bridge.
- Lectures 1–4 + Assignment 1: Clone the repository, manage the environment with uv and run
uv run pytest. At first, every test fails with NotImplementedError. Connect your own implementations intests/adapters.pyand get the tests passing one at a time. Once the code is correct on CPU, train a small model on TinyStories. - Lectures 5–8 + Assignment 2: From this point on, you need a GPU. The Assignment 2 repository includes the official reference implementation of Assignment 1, so you can continue even if you have not fully finished Assignment 1. You can also watch Lecture 10 on inference during this stage.
- Lectures 9 and 11 + Assignment 3: Focus on understanding how scaling laws are fitted and extrapolated. Learners who are not enrolled cannot use the hosted training API, so estimate the compute cost of running the training yourself before you start.
- Lectures 12–14 + Assignment 4: Start by downloading a small number of files in offline mode and building the filtering and deduplication pipeline. The README notes that you can reduce the number of WET files downloaded (about 2,500 by default). On a limited budget, you can work at a smaller scale instead of reproducing the 8-GPU final training run.
- Lectures 15–17 + Assignment 5: Complete the supervised fine-tuning and reinforcement learning parts. If you still have time and energy, do the optional safety alignment part (DPO and similar methods). You can leave the two guest lectures until the end.
The official pace covers all the material in 10 weeks. If you are studying while working, you can stretch the timeline to fit the time you actually have each week. Work through one assignment at a time, and move on to the next group of lectures after you finish each assignment.
References
-
Stanford CS336 | Language Modeling from Scratch, course website, accessed October 22, 2026
-
Stanford CS336: Language Modeling from Scratch | Spring 2026, YouTube lecture recording playlist
-
stanford-cs336, GitHub organization page
-
stanford-cs336/lectures, lecture materials repository
-
stanford-cs336.github.io commit history, course website repository
-
Language Modeling from Scratch | Stanford Online, accessed October 22, 2026
-
Smart Academy · Your own academy for the intelligent era
-
Compiled and published by Scholay, the one-stop academic platform. If this content infringes your rights, please let us know and it will be taken down.
-
Also published on EnterScholar, a community for scholars in the intelligent era