Mining Reddit communities with a live RAG pipeline
A Blazor Server application that crawls any subreddit, embeds the discussion into a vector store, and answers questions about it (with citations) through a real-time chat interface. Built for researchers and founders who want signal from a community without reading a thousand threads by hand.
The problem
Researchers and founders want to extract insight from subreddits: recurring pain points, emerging trends, common questions, without reading thousands of posts manually. Keyword search on Reddit surfaces individual threads, not synthesis, and generic web-scraping tools produce raw text with no way to ask it a question.
What I built
A Blazor Server app that crawls any subreddit through the Reddit API, chunks and embeds the posts into a PostgreSQL vector store using a local Ollama model, and exposes a real-time chat interface where users get cited, context-aware answers via retrieval- augmented generation. Ingestion runs as a streaming background pipeline with live progress pushed to the browser over SignalR, so a crawl of a large subreddit shows real status instead of a blank spinner.
Engineering decisions
Why rerank before synthesis, not just top-K retrieval?
Raw vector similarity on short Reddit comments returns a lot of near-duplicate or tangential matches. A rerank pass over the initial candidate set measurably improved answer relevance without needing a larger K, which would have diluted the context window with noise.
Why Blazor Server instead of a JS SPA?
The app is UI-light and update-heavy: a chat stream and a live ingestion status feed. Blazor Server's built-in SignalR circuit meant real-time updates without hand-rolling a WebSocket layer or a separate frontend build pipeline.
Why a local embedding model instead of a hosted embeddings API?
Embedding runs continuously as new posts are ingested. Routing that volume through a metered hosted API adds cost and an external dependency for a step that doesn't need frontier model quality — Ollama running locally keeps ingestion cost-flat and offline- capable.
Why OpenRouter for the synthesis step specifically?
Unlike embedding, answer quality on synthesis matters a lot, and the right model for that job changes as new ones ship. OpenRouter keeps the generation step swappable without a code change, so the deployment can move to a better model as soon as one's worth switching to.
Operational details
Rate limiting
Per-IP rate limiting on the chat endpoint prevents a single client from exhausting the LLM budget or the crawl queue.
Lead capture
Optional lead capture with email notifications via Resend and PostgreSQL persistence, for turning tool usage into follow-up conversations.
Configurable retrieval
Retrieval parameters — chunk size, top-K, rerank depth — are configurable per deployment rather than hardcoded.
Deployment
Runs in Docker behind Nginx, matching the rest of the production infrastructure on this domain.