Why RAG Systems Fail at Enterprise Scale

Core Summary: This document addresses why Retrieval-Augmented Generation (RAG) systems often suffer from dramatic accuracy drops when scaling from small (5k) to large (500k) document corpora. The core issue is 'neighborhood density' within the embedding space, where thematically related documents (e.g., Slack threads, Jira tickets, and emails regarding the same project) cluster together, causing specific, relevant information to be pushed out of the top-k retrieval results. Important Details and Facts: Research from Onyx utilizing the open-source EnterpriseRAG-Bench dataset demonstrates this phenomenon. Findings showed that vector search accuracy plummeted from 90.7% at 5k documents to 50.6% at 500k. BM25 performed more robustly, dropping from 85.8% to 68.4%. High neighborhood density was found to monotonically correlate with lower recall across all scales. Entities and Tools: The document references several platforms including Slack, Gmail, Jira, GitHub, Confluence, Google Drive, HubSpot, Fireflies, and Linear. The primary research entity is Onyx, and the core dataset used is the EnterpriseRAG-Bench. Technologies mentioned include vector search and BM25 (Best Matching 25) algorithms. Recommendations: The author warns that small-scale testing (e.g., 5k docs) is insufficient for predicting production performance. Developers are advised to test retrieval systems against realistic production volumes using the EnterpriseRAG-Bench dataset to measure how the accuracy curve breaks as the corpus expands.