🐦 Follow me on Twitter • ➡️ Jump to LLMs! 📧 Feedback

TorchLeet is broken into two sets of questions:

1. Question Set: A collection of PyTorch practice problems, ranging from basic to hard, designed to enhance your skills in deep learning and PyTorch.
2. LLM Set: A new set of questions focused on understanding and implementing Large Language Models (LLMs) from scratch, including attention mechanisms, embeddings, and more.

Note

Avoid using GPT. Try to solve these problems on your own. The goal is to learn and understand PyTorch concepts deeply.

Table of Contents

Question Set

🔵Basic

Mostly for beginners to get started with PyTorch.

🟢Easy

Recommended for those who have a basic understanding of PyTorch and want to practice their skills.

🟡Medium

These problems are designed to challenge your understanding of PyTorch and deep learning concepts. They require you to implement things from scratch or apply advanced techniques.

1. Implement parameter initialization for a CNN (Solution)
2. Implement a CNN from Scratch
3. Implement an LSTM from Scratch (Solution)
4. Implement AlexNet from scratch
5. Build a Dense Retrieval System using PyTorch
6. Implement KNN from scratch in PyTorch

🔴Hard

These problems are for advanced users who want to push their PyTorch skills to the limit. They involve complex architectures, custom layers, and advanced techniques.

1. Write a custom Autograd function for activation (SILU) (Solution)
2. Write a Neural Style Transfer
3. Build a Graph Neural Network (GNN) from scratch
4. Build a Graph Convolutional Network (GCN) from scratch
5. Write a Transformer (Solution)
6. Write a GAN (Solution)
7. Write Sequence-to-Sequence with Attention (Solution)
8. [Enable distributed training in pytorch (DistributedDataParallel)]
9. [Work with Sparse Tensors]
10. Add GradCam/SHAP to explain the model. (Solution)
11. Linear Probe on CLIP Features
12. Add Cross Modal Embedding Visualization to CLIP (t-SNE/UMAP)
13. Implement a Vision Transformer
14. Implement a Variational Autoencoder

LLM Set

An all new set of questions to help you understand and implement Large Language Models from scratch.

Each question is designed to take you one step closer to building your own LLM.

1. Implement KL Divergence Loss
2. Implement RMS Norm
3. Implement Byte Pair Encoding from Scratch (Solution)
4. Create a RAG Search of Embeddings from a set of Reviews
5. Implement Predictive Prefill with Speculative Decoding
6. Implement Attention from Scratch (Solution)
7. Implement Multi-Head Attention from Scratch (Solution)
8. Implement Grouped Query Attention from Scratch (Solution)
9. Implement KV Cache in Multi-Head Attention from Scratch
10. Implement Sinusoidal Embeddings (Solution)
11. Implement ROPE Embeddings (Solution)
12. Implement SmolLM from Scratch (Solution)
13. Implement Quantization of Models
    1. GPTQ
14. Implement Beam Search atop LLM for decoding
15. Implement Top K Sampling atop LLM for decoding
16. Implement Top p Sampling atop LLM for decoding
17. Implement Temperature Sampling atop LLM for decoding
18. Implement LoRA on a layer of an LLM
    1. QLoRA
19. Mix two models to create a mixture of Experts
20. Apply SFT on SmolLM
21. Apply RLHF on SmolLM
22. Implement DPO based RLHF
23. Add continuous batching to your LLM
24. Chunk Textual Data for Dense Passage Retrieval
25. Implement Large scale Training => 5D Parallelism

What's cool? 🚀

• Diverse Questions: Covers beginner to advanced PyTorch concepts (e.g., tensors, autograd, CNNs, GANs, and more).
• Guided Learning: Includes incomplete code blocks (... and #TODO) for hands-on practice along with Answers

Getting Started

1. Install Dependencies

• Install pytorch: Install pytorch locally
• Some problems need other packages. Install as needed.

2. Structure

• /: Easy/Medium/Hard along with the question ID.
• /qname.ipynb: The question file with incomplete code blocks.
• /qname_SOLN.ipynb: The corresponding solution file.

3. How to Use

• Navigate to questions/ and pick a problem
• Fill in the missing code blocks (...) and address the #TODO comments.
• Test your solution and compare it with the corresponding file in solutions/.

Happy Learning! 🚀

Contribution

Feel free to contribute by adding new questions or improving existing ones. Ensure that new problems are well-documented and follow the project structure. Submit a PR and tag the authors.

Authors

💻 AI/ML Dev

💻 Developer

Stargazers over time