Learn AI Create the Future — with Pranay
Main site

All sections

The whole curriculum, grouped into 14 areas. Start at the top if you are new. If you came here for one thing, filter to its area or search — everything is on this page or one click from it.

Every lesson here is written by .

14 areas 117 sections 1,162 of 1,167 lessons published 10 new lessons every day

Showing Deep Learning — 11 sections, 113 lessons. Show all 14 areas

Deep Learning

Neural networks explained in plain words, then built by hand in PyTorch, all the way to training on many GPUs.

11 sections 113 of 113 lessons published

Deep Learning

Neural networks — what they are, how they learn, and how to build them.

13 published

  1. What is a neural network?
  2. How neural networks learn
  3. Activation functions
  4. Backpropagation
  5. Gradient descent
  6. Loss functions
  7. Convolutional neural networks (CNN)
  8. Recurrent neural networks (RNN)
  9. LSTM
  10. GANs
  11. Autoencoders
  12. Transformers
  13. PyTorch basics

PyTorch Tensors

The one object every PyTorch program is made of, and the handful of shape tricks that stop most errors before they happen.

10 published

  1. Installing PyTorch with the right CUDA build
  2. Creating tensors and choosing a dtype
  3. Shapes and broadcasting
  4. reshape, view, permute and contiguity
  5. Reductions and the dim argument
  6. Indexing and boolean masks
  7. gather, scatter and index_select
  8. einsum for tensor operations
  9. Moving tensors between CPU and GPU
  10. Seeds and reproducible runs

Autograd in Depth

PyTorch keeps a record of every calculation so it can work out who caused the error. This section is what that record really is.

10 published

  1. How the autograd graph is built and freed
  2. requires_grad and leaf tensors
  3. detach, no_grad and inference_mode
  4. Calling backward on a non-scalar output
  5. In-place operations and when they break autograd
  6. Gradient accumulation for large batches
  7. Gradient clipping
  8. Writing a custom autograd Function
  9. Checking your gradients with gradcheck
  10. Jacobians, vjp and jvp with torch.func

Building Models with nn.Module

How PyTorch keeps track of your layers, and the building blocks you reach for in almost every model you write.

11 published

  1. How nn.Module tracks parameters
  2. Sequential, ModuleList and ModuleDict
  3. Buffers vs parameters
  4. Weight initialisation
  5. model.train() and model.eval()
  6. BatchNorm and its running statistics
  7. nn.Embedding and index errors
  8. Padding and packing variable-length sequences
  9. Hooks for inspecting a running model
  10. Writing a custom loss function
  11. Transfer learning: swapping the head and freezing a backbone

Datasets and DataLoaders

Getting data into the model quickly and correctly — the part that quietly decides how fast your training runs.

10 published

  1. Writing your own Dataset
  2. What DataLoader actually does
  3. Writing a collate_fn
  4. num_workers, prefetching and the Windows spawn trap
  5. Samplers and weighted sampling
  6. IterableDataset for streams and huge files
  7. Image transforms with torchvision v2
  8. Datasets that do not fit in memory
  9. Surviving corrupt and missing files
  10. Finding out whether the GPU is waiting for data

Optimisers, Schedulers and the Training Loop

Everything that surrounds the model: how weights are updated, how the learning rate moves, and how you know it is working.

9 published

  1. Choosing between SGD, Adam and AdamW
  2. Parameter groups and per-layer learning rates
  3. Learning rate schedulers
  4. Finding a learning rate that works
  5. Logits, softmax and picking the right loss
  6. Writing a validation loop that reports the truth
  7. TorchMetrics
  8. Early stopping and keeping the best model
  9. PyTorch Lightning

Debugging PyTorch

The errors everyone hits, and a method for the worst case of all — code that runs perfectly and learns nothing.

9 published

  1. Reading a shape error
  2. Expected all tensors to be on the same device
  3. Expected scalar type Float but found Double
  4. When the loss will not go down
  5. Overfit a single batch first
  6. Tracking down a NaN loss
  7. Checking that gradients reach every layer
  8. Memory that grows every epoch
  9. Finding label and target bugs

GPU Memory and Speed

Why training runs out of memory or crawls, and the small number of changes that reliably fix it.

10 published

  1. Reading and fixing CUDA out of memory
  2. What is using your GPU memory
  3. Mixed precision with autocast and GradScaler
  4. Gradient checkpointing
  5. torch.compile
  6. Profiling with torch.profiler
  7. Timing GPU code correctly
  8. Fast attention with scaled_dot_product_attention
  9. channels_last for convolution networks
  10. When the GPU is slower than the CPU

Multi-GPU and Distributed Training

Training on more than one GPU: what actually gets sent between them, and the ways it goes wrong.

11 published

  1. DataParallel vs DistributedDataParallel
  2. Your first DDP run with torchrun
  3. DistributedSampler and set_epoch
  4. When gradients are synchronised, and no_sync
  5. Scaling the learning rate with the number of GPUs
  6. Saving and logging from one rank only
  7. all_reduce, all_gather and broadcast
  8. FSDP: sharding a model across GPUs
  9. Pipeline and tensor parallelism
  10. Training across several machines
  11. Debugging a distributed job that hangs

Checkpoints, Export and Inference

Saving a model so it loads again, and getting it out of your training script in one piece.

7 published

  1. What a state_dict really is
  2. Resuming training exactly where it stopped
  3. Fixing missing and unexpected keys
  4. Why torch.load can run code, and safetensors
  5. Exponential moving average of weights
  6. TorchScript: tracing vs scripting
  7. torch.export and the ExportedProgram

Reinforcement Learning

How an AI learns by trying things and getting rewarded — the method behind game-playing AI and RLHF.

13 published

  1. What is reinforcement learning?
  2. Agents, environments and rewards
  3. Markov decision processes
  4. Exploration vs exploitation
  5. Multi-armed bandits
  6. Q-learning
  7. Deep Q-networks (DQN)
  8. Policy gradients
  9. Actor-critic methods
  10. PPO
  11. Reward shaping
  12. RLHF — learning from human feedback
  13. Gymnasium and training environments

Other ways to use this site

Learning paths

An ordered route through the lessons for one job: ML developer, AI engineer, computer vision engineer.

Projects

Build real things with full code, a dataset and a deployment guide.

AI glossary

Every term you keep seeing, defined in one plain sentence first.

Error database

Paste the error you got. Find out what it means and how to fix it.