Mastering the Full LLM Pipeline for 3 Yuan and 2 Hours: Pure PyTorch in MiniMind

How open-source project MiniMind unpacks the full LLM training pipeline—pre-training, SFT, LoRA, RL, and Tool Use—in pure PyTorch without high-level framework a

tau · September 10, 2026

#LLM #MiniMind #PyTorch #ModelTraining #OpenSource #AIDevelopment

Mastering the Full LLM Pipeline for 3 Yuan and 2 Hours: Pure PyTorch in MiniMind

AI researcher and practitioner Ren (@Ryrenz) has highlighted MiniMind, an open-source educational project that demonstrates how to train a conversational large language model from scratch in about two hours for just 3 RMB (less than $0.50) in cloud compute. Created in July 2024 by Nanjing University researcher jingyaogong—who authored 194 of the repository's commits compared to just 6 from the second contributor—MiniMind has rapidly surpassed 59,000 GitHub stars by stripping away framework abstraction layers and implementing the entire LLM pipeline in transparent, native PyTorch.

Architectural diagram and code structure of MiniMind, an open-source pure PyTorch end-to-end LLM training pipeline

Image source: MiniMind (jingyaogong) / X (@Ryrenz)

The vast majority of existing LLM tutorials focus narrowly on downloading pre-trained base models and applying LoRA fine-tuning. While this enables quick experimentation, the core engineering mechanics—from data ingestion and parameter initialization to multi-head attention and loss backpropagation—remain opaque black boxes. MiniMind directly tackles this knowledge gap by laying out the complete pipeline from scratch, allowing developers to inspect every single mathematical calculation and gradient update line by line.

Stripping Away High-Level Abstractions with Pure PyTorch

MiniMind's foundational principle is to eliminate wrapper dependencies such as transformers, trl, and peft, implementing every critical algorithm directly with native PyTorch primitives.

The project decomposes the full lifecycle of modern large language models into modular, self-contained stages:

  • Pre-training: Building foundational language patterns from scratch via token prediction over randomly initialized weights.
  • Supervised Fine-Tuning (SFT): Aligning raw model output using conversational instruction datasets.
  • LoRA (Low-Rank Adaptation): Implementing parameter-efficient fine-tuning via direct low-rank matrix decomposition.
  • Direct Preference Optimization (DPO): Optimizing directly on human preference pairs without training a separate reward model.
  • PPO and GRPO Reinforcement Learning: Controlling reasoning trajectories and structured generation via Proximal Policy Optimization and Group Relative Policy Optimization.
  • Knowledge Distillation: Transferring logit distributions from larger teacher models into compact student architectures.
  • Tool Use and Agentic RL: Structuring function-calling mechanisms and autonomous action policies.

By making the forward pass, backward pass, and loss computation fully explicit in standard Python and PyTorch syntax, learners can read and trace the codebase sequentially without hitting hidden library internals.

Hardware Feasibility, Cost, and Qwen3 Ecosystem Alignment

A major factor behind MiniMind's popularity is its exceptionally accessible hardware footprint and economic barrier to entry.

According to the official repository benchmarks, running one complete pre-training round on a single consumer-grade NVIDIA RTX 3090 GPU takes approximately 1.21 hours, costing around 1.57 RMB in rental compute. The full training cycle can be executed within roughly two hours for approximately 3 RMB:

  • Ultra-Compact 64M Parameter Baseline: The smallest default model contains only 64 million parameters, enabling seamless training and inference on modest hardware setups.
  • Qwen3 Ecosystem Alignment: The architectural structure is deliberately aligned with the Qwen3 ecosystem, making the learned modules directly applicable to modern production open-weight models.
  • Standard Serving Integrations: Once trained, weights can be exported and served via llama.cpp, vllm, or ollama. The repository includes an OpenAI-compatible API server alongside a lightweight web chat interface.
  • Multimodal Evolution: The ecosystem has expanded beyond text models to include MiniMind-V (vision), MiniMind-O (multimodal), and diffusion-based language model variants.

As Ren (@Ryrenz) summarized, reading a cleanly structured, end-to-end training implementation from beginning to end delivers far deeper conceptual clarity than skimming dozens of high-level survey papers.

Community Observations and Educational Reality Checks

As the repository went viral, experienced developers in the community shared practical caveats alongside technical endorsements:

  • Loopsaaage (@daotagoto): The published RTX 3090 benchmark reflects minimal baseline experimental settings. Furthermore, a 64M parameter model demonstrating conversational capability is fundamentally different from a model that is viable in production environments. Readers should not let the headline "3 RMB" cost foster unrealistic expectations for enterprise use. However, as an educational teaching aid that elevates understanding from superficial API wrappers and LoRA scripts to fundamental pre-training mechanics, its value is exceptional.
  • ErenHe (@sirHe12): The most rewarding way to study the project is to contrast its internal mechanics directly with the official Qwen3 ecosystem, modifying training hyperparameters and analyzing weight diffs. The project's massive traction stems from its clarity as an educational tool where every line of code can be audited and understood directly.

Sources

The technical breakdown of MiniMind's architecture and native PyTorch training pipeline is verified from AI researcher Ren's (@Ryrenz) technical post and the official open-source GitHub repository led by Nanjing University researcher jingyaogong. Complete implementation scripts for pre-training, SFT, LoRA, DPO, PPO/GRPO, distillation, and OpenAI-compatible serving interfaces are accessible via the primary reference links below.