arnitly

1 video archived 204,676 total views

A developer just built a fully searchable RAG pipeline over 2M+ pages of the Epstein Files and the architecture decisions are worth studying. The dataset is public on Hugging Face (teyler/epstein-files-20k). The MMR swap is the real lesson here. Pure cosine similarity at 100K+ chunks returns redundant results from the same document. Maximal Marginal Relevance balances relevance with diversity. Each retrieved chunk has to earn its place by adding new information, not just scoring high. Stack breakdown: LangChain RecursiveCharacterTextSplitter for chunking, all-MiniLM-L6-v2 for embeddings, ChromaDB as vector store, Groq + LLaMA 3.3 for inference. Runs on 8GB RAM minimum, free Groq API key required. Cleaning was deliberately lightweight — regex boundary detection, line buffer reconstruction, dropping docs under 100 characters. Nothing exotic. The point was pipeline architecture, not preprocessing complexity. Repo linked in comments. #ai #epsteinfiles #artificialintelligence #technews #algorithm 1:17
A developer just built a fully searchable RAG pipeline over ...
204,676 14,156
Feb 25, 2026