SmolHub Playground
SmolHub Playground
Source: https://www.smolhub.com/smolhub/
📌 SmolHub is Yuvraj Singh's web playground and open-source collection of small language models (roughly 88M–200M parameters) built from scratch in PyTorch. It showcases proof-of-concept LLMs, translation models, and narrative generators with Gradio demos and Hugging Face–hosted weights.
🧪 Six Playground Projects
The hub lists six experiments: Smol Llama (130M instruct model), Smol Mixtral (124M MoE), Smol Transformer (English→Hindi seq2seq), Story Kimi (DeepSeek V3–inspired MoE), Story Llama (88M bedtime-story model), and Story Mixtral (story-focused MoE).
🔥 From-Scratch PyTorch
Every model is implemented and trained from scratch—no weight transfer from large pretrained checkpoints. Architectures include RoPE, SwiGLU, Flash Attention, GQA/MQA, RMSNorm, and Mixture-of-Experts with top-2 routing.
🏋️ Full Training Pipelines
Smol Llama runs a three-stage pipeline: pretraining on FineWeb, SFT on Alpaca, and DPO on UltraFeedback. Other models use TinyStories, Samanantar, or FineWeb, with multi-GPU DDP, WandB logging, and published checkpoints.
🌐 Try It in the Browser
Each project ships a Gradio web interface for interactive text generation or translation. Pretrained weights are on Hugging Face (e.g., SmolLlama Space, YuvrajSingh9886/SmolMixtral), so you can experiment without training locally.
💻 Built for Modest Hardware
Models target the 100–300M parameter range and are optimized for consumer GPUs—designed to fit in about 8 GB VRAM with mixed precision, gradient checkpointing, and efficient attention kernels.
Key facts
| Fact | Value |
|---|---|
| Creator | Yuvraj Singh (yuvrajsingh.io) |
| Framework | PyTorch 2.0+ with Gradio demos |
| Model Size Range | ~88M to ~200M parameters |
| GitHub Repository | YuvrajSingh-mist/SmolHub (MIT License) |
| Notable Smol Llama Pipeline | Pretrain → SFT → DPO (130M params) |
| Deployment | Hugging Face Spaces and model hubs |
Details
SmolHub Playground at smolhub.com/smolhub is the public-facing index for Yuvraj Singh's experimental AI work. Described as a space for "small-scale models, proof-of-concepts, and AI experiments," it links to six self-contained projects, each with its own GitHub subdirectory, documentation, training reports, and interactive demo.
The collection spans multiple architectures and tasks. Smol Llama (130M) is the most complete instruct-tuned model, trained through pretraining, supervised fine-tuning, and DPO alignment. Smol Mixtral and Story Mixtral implement Mixtral-style MoE transformers with Flash Attention and Liger kernels. Story Kimi explores DeepSeek V3–inspired latent attention and MoE. Story Llama (88M) focuses on narrative generation from TinyStories. Smol Transformer is an encoder-decoder model for English-to-Hindi translation using the Samanantar dataset.
All projects share a common engineering philosophy: implement architectures from the ground up, train on curated Hugging Face datasets, log experiments to Weights & Biases, and publish weights to Hugging Face. Training supports distributed data parallel (DDP), cosine learning-rate schedules, gradient accumulation, and optional optimizations like torch.compile and Liger kernels.
SmolHub is both a learning resource and a research sandbox. The accompanying GitHub repo positions it as a platform for exploring attention mechanisms, scaling laws, and efficient training of small models that remain practical on modest hardware while still producing coherent text.