Retrieval-Augmented Generation (RAG) has emerged as a powerful technique for enhancing the performance of generative AI models. RAG is the process of optimizing the output of a large language model (LLM) with domain-specific information injected before sending the request to the LLM.  

For more information on RAG, read our blog post, What Is RAG and How Does It Work? 

To date, the RAG strategies being discussed and developed have utilized vector databases and vector-based searching. While a vector-based RAG strategy works well in specific scenarios, others require a more “real-time” approach. This blog will compare these approaches, highlighting their key features, advantages, and limitations.

Vector-Based Retrieval-Augmented Generation (Offline RAG)

RAG uses specific knowledge sets to enrich and enhance outputs, and that information needs to be readily available. The most common approach for this is to use a vector database.  A vector database breaks large datasets and content into smaller, manageable parts or “chunks.” Each chunk is then transformed into a mathematical representation, or vector, based on the features or attributes of the “chunk.” Each vector has multiple dimensions ranging from tens to thousands, depending upon the complexity of the underlying data and transformations performed. Vector design has similar objects close together in the vector space. This approach allows for efficient retrieval of relevant chunks based on similarity searches. 

A RAG strategy leveraging a vector database transforms the query into a vector, and the vector database returns similar vectors. LangChain is an example of a product that can chunk documents and embed them in a vector database to enable RAG. 

Advantages of Vector-based RAG

Vector-based RAG offers several compelling benefits, particularly in terms of domain specificity, efficient retrieval, and scalability. By storing domain-specific information and enabling fast retrieval of relevant chunks, this approach ensures that generative AI models can access enriched, contextually appropriate data. Let’s delve into the key advantages of utilizing vector-based RAG.

1. Domain Specificity – Storing domain-specific information contained in documents in a vector database gives the AI LLM domain-specific knowledge contained in the documents.
2. Efficient Retrieval – The vector database enables fast and efficient retrieval of similar content chunks, making it suitable for scenarios where quick responses are needed.
3. Scalability – Vector databases can handle large datasets since the documents are pre-processed and stored in a structured manner.

Limitations of Vector-based RAG

While vector-based RAG is powerful, certain limitations can impact its effectiveness. These include challenges related to changing content, security concerns, and the potential for noise and hallucinations in retrieval. This section will highlight these limitations and discuss their implications for deploying vector-based RAG systems.

1. Changing Content – Since documents are chunked and embedded into a vector database, when content is updated or deleted, a separate process must re-embed affected documents and replace or remove associated content that is no longer valid from the vector database. This drawback is significant for content that changes or is removed over time.
2. Security – For organizations with secure documents and information, the offline vector database approach does not support honoring this security during the RAG retrieval process.  The only options are to allow all content to have read access for all users or to develop a secondary lookup of security before augmenting the prompt in the RAG engine.  This latter approach is challenging to develop and, more likely, impossible due to complex security structures and loss of document-level context with chunked vector embeddings.
3. Noise and Hallucinations – The primary search mechanism is based on similarity, which may yield inaccurate results. Similarity searches will always respond with an answer, even if it is outdated or incorrect. This nuance can lead to irrelevant or misleading information. For example, a query like “Analyze all contracts from vendor ‘Acme’ over $1 million” might retrieve unrelated contracts due to the similarity-based retrieval process and the inability to target relevant contracts by metadata search.

Real-Time Retrieval-Augmented Generation (Real-Time RAG)

In real-time RAG, the content remains in context since information and documents are not chunked or embedded in a vector database. The information is kept in its original context, allowing for more accurate and contextually relevant retrieval. Since content is kept intact, it can be associated with metadata and retrieved using full-text searching and metadata queries, providing further enrichment and searching strategies.

Advantages of Real-Time RAG

Real-time RAG provides dynamic content management, enhanced security, improved accuracy, and better contextual understanding. Keeping information in its original form and utilizing advanced search capabilities minimizes the risk of retrieving irrelevant or outdated data. This section will explore the significant benefits of real-time RAG in various applications.

1. Dynamic Content – The ability to update information in real-time (add, update, and delete) makes real-time RAG ideal for applications where information is frequently added or changed. Users can keep their datasets current without complex reprocessing.
2. Security – Real-time RAG allows content security models to be honored since content is accessed directly from the enterprise content store. Organizations that need to enforce specific permissions for users or roles need to ensure their RAG engine utilizes a content services repository with relevant security embedded. This approach enhances privacy and, in some industries, honors regulatory requirements for content access.
3. Accuracy – Enhanced search capabilities based on content and metadata reduce noise and improve the relevance of the retrieved information. This focus minimizes the risk of hallucinations and irrelevant data.
4. Contextual Understanding – Real-time RAG can provide more nuanced and accurate responses by keeping information in context, especially for complex queries.
5. Ease of Attribution Support – With real-time RAG, responses can contain links to documents utilized during the generation process. This enrichment allows users to gain insights into how the LLM generated the response by seeing source documents, provides the user easy access to further reading, and enhances user trust in the LLM response.

Limitations of Real-Time RAG

Despite its numerous benefits, real-time RAG has some limitations, particularly concerning context windows and efficiency. Managing large volumes of data within an LLM’s context window can be challenging, and poor implementation can lead to inefficiencies. This section will discuss these constraints and how they can be managed.

1. Context Window – All LLMs have a context window, limiting how much text users can include in a prompt. This restriction must be considered when augmenting prompts with entire documents versus chunks.  If a text overflows this context window, the LLM will not have access to the information lost in a “first-in-first-out” manner.  Some models have context windows of 8-64 thousand “tokens,” each being a word or part of a word.  Other models have context windows of 1 million tokens. Context windows are growing daily, but users must be aware of this limitation. 
2. Efficiency Paradox – When implemented poorly, utilizing very large context windows can be inefficient, both in processing time and cost, as most LLMs are metered by tokens. We recommend that the RAG retrieval engine is tuned to find relevant documents based on metadata and full-text content to ensure that the minimum amount of tokens is utilized in the context window, improving LLM response quality, performance, and cost-effectiveness. 

Vector vs Real-time RAG – The Verdict

Both vector database and real-time RAG approaches have unique strengths and weaknesses.  With efficient retrieval and scalability, Vector database-driven RAG is suitable for scenarios where the dataset is static and open to all LLM users. However, its limitations regarding changing content, difficulty honoring content security, and potential for noise due to only supporting similarity searches make it less ideal for dynamic enterprise content environments.

On the other hand, real-time RAG offers flexibility, security, improved accuracy, and better contextual understanding, making it suitable for applications where content changes and content-level security must be honored. The advantages of dynamic content management and enhanced search capabilities make real-time RAG a powerful tool for modern enterprise AI applications.

Choosing the right RAG approach, vector, real-time, or a combination of both, depends on the application’s specific requirements, including the nature of the dataset, the need for content updates, the need to honor content security, and the importance of retrieval accuracy. By understanding these factors, organizations can leverage the appropriate RAG strategy to enhance their AI capabilities and achieve better outcomes. Get in touch today to learn more about how Veladocs can enrich AI RAG strategies for end-user organizations and take LLM offerings to the next level for software vendors.