Loading course…
Build Private Offline AI Agents
Created by Shaunak Ghosh
Run capable local LLMs with Ollama or LM Studio, then build a private RAG workflow over your own documents with grounding and citations. Finally, connect your local-first agent to MCP tools safely, and package it for reproducible, “works on my machine” startup.
Requirements
- Comfort using a terminal and editing config files
- Basic Python or JavaScript reading ability
- Basic understanding of localhost, ports, and HTTP
- Familiarity with filesystems and directory structure
What you'll learn
- Explain, and debug, the core local-first architecture: LLM generation vs embeddings vs the RAG retrieval loop.
- Install and run local models reliably, and choose between Ollama and LM Studio based on workflow and constraints.
- Diagnose common local performance failures by relating context length to VRAM/RAM pressure and latency.
Learning path
8 modules • Each builds on the previous one
1
LLM and RAG core mental model
Build a correct mental model of what runs locally: the generative LLM (text output) vs the embedding model (vectorization for retrieval), and why local-first mainly changes data boundaries and cost—not magically model capability.
2 videos9 min
2
Local inference tooling: Ollama and LM Studio
Compare Ollama and LM Studio as local inference toolchains: installation paths, model management, serving APIs, updates, and how to keep setups offline/low-cost and repeatable.
2 videos10 min
3
Hardware limits: RAM, VRAM, tokens
Understand local LLM constraints: RAM/VRAM sizing, context window memory cost, tokens/sec throughput, CPU vs GPU behavior, and practical monitoring for ‘works on my machine’ reliability.
1 video3 min
4
Model quantization: GGUF and EXL2
Learn what quantization changes (weight precision), why it reduces RAM/VRAM footprint, and how formats like GGUF and EXL2 trade quality, speed, and compatibility across backends.
2 videos8 min
5
Choosing local models for tasks
Pick models strategically for reasoning vs writing vs code vs small-footprint use, using lightweight evaluation: latency, context needs, tool-use ability, and task-specific benchmarks.
1 video4 min
6
Vector embeddings and similarity search
Understand embeddings as vectors, how similarity search works (cosine/dot), and how local vector indexes support private semantic retrieval for RAG over personal docs.
1 video4 min
7
Private RAG: chunking, retrieval, grounding
Build a private RAG pipeline over local documents (PDFs, notes, repos): chunking strategies, embeddings, retrieval tuning, and grounding checks (citations, quote verification) to reduce hallucinations.
2 videos10 min
8
MCP basics: safe tools and packaging
Learn MCP fundamentals and connect local-first agents to tools (filesystem, git, task manager) with safe defaults (read-only, directory scoping), then harden privacy boundaries against prompt injection and package a reproducible ‘one-command’ setup.
3 videos16 min
Start Learning
Begin your learning journey
Modules8
Duration61 min
Science-backed learning
In-video quizzes and scaffolded content to maximize retention.
Key concepts
Local-First Privacy Boundaries And Offline TradeoffsRunning And Managing Local Models (Ollama And LM Studio)Hardware Limits: RAM, VRAM, Context Window Performance