techwithprateek

2 videos archived 59,277 total views

Here’s the production-grade way to think about it 👇 The Ingestion Illusion Embedding 10M PDFs upfront is pure waste Most documents are never queried Instead: → Fingerprint PDFs → dedupe 30–40% instantly → Chunk semantically, not by fixed tokens → Embed on access, not on arrival Payoff: 5 TB shrinks to ~3 TB. Embedding bill drops 60% The Vector Tax Vector search is expensive when it’s your first filter Cosine similarity shouldn’t touch cold data Instead: → Keyword + metadata filter first → Narrow to top 1–5% corpus → Run vectors only on survivors Payoff: P95 latency improves 4–6× The Retrieval Funnel One retriever is brittle at this scale Instead: → BM25 for recall → vectors for relevance → Rerank top 50, not top 5,000 → Cache query embeddings aggressively Payoff: Recall stays high. Cost stays flat. The Context Budget Trap More context ≠ better answers It’s noise inflation Instead: → Compress chunks with summaries → Enforce hard token caps → Track answer attribution coverage Payoff: Token usage drops 70%. Accuracy goes up. Reframe: RAG is a retrieval system, not an embedding project. 🔖 Save this for your next RAG system design interview 💬 Comment “RAG” if you are also building a real-world architecture ➕ Follow for production-grade system design, not toy demos 0:07
Here’s the production-grade way to think about it 👇 The Ing...
49,908 687
Feb 15, 2026
Everyone thinks LLM leaks are “model problems.” Actually: they’re architecture problems. Here’s the framework I use in production Access, Context, Output. ⚡ The Over-Entitled Retriever Insight: Most leaks happen before generation because your retriever sees too much • Enforce row-level ACLs → filter before embedding search • Partition vector indexes by tenant → zero cross-org bleed • Sign queries with user identity → audit every retrieval Result: 100% tenant isolation. Zero accidental cross-access — ⚡ The Prompt Injection Trap Insight: A single malicious sentence can override your system prompt. “Ignore previous instructions…” → goodbye guardrails. • Strip tool instructions from retrieved text → no tool hijacking • Freeze system prompts server-side → never client-controlled • Run injection classifier → block risky queries pre-generation Payoff: 80% of jailbreak attempts stopped before inference. — ⚡ The Memory Time Bomb Insight: Long-term memory becomes long-term liability. • Encrypt embeddings at rest → reduce blast radius • Set TTL on conversation memory → auto-expire after 24h • Disable training retention → no vendor data reuse Outcome: Sensitive data lifespan drops from months to hours. — ⚡ The Output Spill Insight: The model can echo secrets it shouldn’t. Especially in summarization and Q&A. • Add regex + NER redaction → mask PII before response • Apply policy LLM pass → secondary compliance filter • Log every response with hash → traceability under 200ms Result: 90% fewer policy violations. — Secure LLM ≠ better prompts. It’s layered defense. Access → Context → Output. 🔖 Save this before your next security review 💬 Comment “SECURE” if you also building safe LLM Appa ➕ Follow for production-grade AI system design breakdowns 0:09
Everyone thinks LLM leaks are “model problems.” Actually: th...
9,369 187
Feb 23, 2026