Organizations that want to implement generative AI (GenAI) in their businesses have increasingly looked to retrieval-augmented generation, or RAG, as a foundational framework. With RAG, the generative outputs of an AI application are based on a trustworthy source, such as a company’s knowledge base, ensuring accuracy, speed, and security.

Doing RAG properly requires putting together an architecture that’s right for your organization. It’s a bit like building a house: you wouldn’t lay the foundation without ensuring you have a solid blueprint for the entire building, so why would you try to implement a RAG-based AI application without the right architecture in place?

The best RAG architecture ensures a great implementation and, ultimately, a safe and trustworthy deployment. In this article, we’ll explore what a RAG architecture should include, what it looks like in practice, mistakes to avoid, and more.

‍

What is a RAG architecture?

Put simply, a RAG architecture defines the structure of a RAG-based AI application. It includes the key components that make RAG possible, such as an ingestion engine, retrieval engine, and generative engine. A RAG architecture also includes the foundational elements critical to any business-ready application, such as a security layer and platform configuration capabilities.

‍

Do I need RAG?

Since AI emerged on the scene just a few years ago, organizations have eagerly explored the deployment of AI applications to unlock new workflows, accelerate existing ones, and drive productivity like never before. After playing around with consumer-grade generative AI tools like ChatGPT, enterprises found that these tools could hallucinate, or make up answers, instead of simply admitting they didn’t know the answer to every question. This isn’t exactly ideal for an enterprise AI application.

Retrieval-augmented generation (RAG) has emerged as an ideal way to implement AI responsibly. AI applications built on a RAG framework retrieve information from a trusted knowledge library before delivering responses to users. This is different from how tools like ChatGPT work; those tools have been trained to sound intelligent, like another human, but in reality they’re more like sophisticated autocomplete systems.

Not all RAG solutions are equal, though. Organizations — especially large ones — that must ensure their AI applications prioritize accuracy, privacy, speed, security, and other capabilities need an enterprise-grade RAG architecture. A RAG architecture built for the enterprise is capable of:

1. Understanding the content wherever it is, as it is.
2. Understanding the question.
3. Matching the best answer(s).
4. Delivering a delightful use answer experience.
5. Adhering to security, governance, and operational requirements.

‍

‍

LEARN MORE: Retrieval-Augmented Generation Tutorial:
Master RAG for Your Enterprise

‍

‍

What a RAG architecture looks like

An enterprise RAG architecture includes everything needed for a large organization to implement RAG at scale. At a high level, here’s what one looks like:
‍

‍

Here’s a quick overview of each of these pieces before we dive deeper on some of the key parts of a RAG architecture.
‍

• Deployment Platform: An enterprise RAG application (like every application) needs to be deployed on a platform. This could be a cloud platform, such as Google Cloud, Microsoft Azure, or Amazon AWS. Some RAG applications also run in a private cloud or federal cloud environment, if an enterprise wants to keep their data segregated from where other companies’ data is stored. Finally, select RAG platforms, such as Pryon RAG Suite, can run entirely on-premises — no cloud connection required.
• Security and Operations: Enterprise RAG architectures must include a security and operations layer to ensure compliance with data governance and security rules. For example, an enterprise RAG architecture should include document-level user access controls so users can only access information they should have access to — for instance, a low-level employee on the Marketing team shouldn’t be able to see payroll documents listing peoples’ salaries. Compliance with cybersecurity frameworks, like SOC 2, is also critical.
• Ingestion Engine: An ingestion engine reads content and converts it into structured data, ready for retrieval. It often includes prebuilt connectors to commonly used data repositories and more capabilities described below.
• Retrieval Engine: Responsible for matching user queries to the ingested content, a retrieval engine is the heartbeat of any enterprise RAG architecture. Doing retrieval well is a major challenge for most RAG application builders.
• Generative Engine: The generative engine enables generative LLM orchestration. A generative engine should play well with different LLMs, offer prompt engineering for developers, and allow users to provide feedback on the generative outputs.
• Platform Configuration: An enterprise RAG application should offer flexible configurability. Everything from the data that’s being ingested, how it’s being retrieved, what the generative experience looks like, and more should be easily modifiable by those deploying and administering the application.
• Reporting & Analytics: No enterprise application is complete without the ability to provide admins with a robust set of details about who’s using it, how often, and for which purpose(s). Advanced reporting and analytics for a RAG-based application can also help subject matter experts (SMEs) learn which queries are most asked and if any new content needs to be spun up based on these queries.

‍

A deeper dive on the key components of an enterprise RAG architecture

Now that you’ve got an overview of what’s included in a RAG architecture, let’s take a closer look at the many pieces that make a RAG application tick.

‍

RECOMMENDED READING: RAG Definition and LLM Glossary

‍

What’s included in an ingestion engine

Any ingestion engine reads content, but the best ingestion engine can read content just like a human would. This is made possible through:

• Enterprise content connectors: Prebuilt pathways connecting the RAG application with an organization’s trusted data sources, such as SharePoint, Zendesk, Salesforce, and more.
• Content updating and management: The ability for administrators to easily define what content the RAG application should ingest and to regularly have that content updated, either manually or on a predefined schedule.
• File type handling: The ability to ingest and read files in multiple formats, such as Word docs, PDFs, images, and videos.
• Layout analysis: The process of analyzing the specific layout of each document to extract important context.
• OCR/HTR: Reading and understanding the words on a page, including handwritten text.
• Sematic segmentation: The intelligent segmentation of text into potential answers, while still maintaining the semantic meaning behind the text.
• Metadata capture: The capture of metadata (e.g., content author, date modified) to better understand the content.
• Segment prioritization: The prioritization of certain text over others based on context. For example, the text from a body paragraph would likely rank higher in importance than text from a table of contents.
• Embeddings: The high-dimensional vectors that LLMs use to capture the semantic meaning of words, sentences, or entire documents.  Once an ingestion engine ingests content, it divides the content into chunks of text. These chunks are encoded into an embedding vector and stored in a vector database.
• Vector database: The entity that stores the embedding vectors produced during the ingestion process. These embeddings are retrieved by the retrieval engine and used to form generative outputs.

‍

What’s included in a retrieval engine

RAG simply isn’t possible with a retrieval engine. You can ingest all the content you want, but if you can’t properly match queries to that content, the RAG application won’t work. The capabilities of a RAG retrieval engine can be separated into two categories:

‍

Query processing:

• Query context handling: Applying the necessary context to handle the incoming query.
• NER (named entity recognition) and intent recognition: The extraction of key entities and intent from a query.   For instance, an employee at a mobile phone store might ask a chatbot, “When will the new iPhones be available?” A sophisticated query engine should be able to infer the user’s intent (in this case, the employee is likely asking about iPhones launching in the next few months, not iPhones that have already launched) and retrieve the best answer related to the user’s query, if available.
• Query type handling: Handling each question by type. For example, an irrelevant question (“What’s for lunch today?”) may be given a standard response or ignored, while another question might prompt an answer sourced from content stored in the knowledge library.
• Query transformation: The breaking down of complex queries into simpler, more manageable questions.
• Query expansion: The expansion of acronyms and other organization-specific terms to return more accurate results (e.g., “SME” may be “small or medium enterprise” or “subject matter expert,” depending on the organization).
• Embedding generation: Generating the embeddings (high-dimensional vectors that LLMs use to capture semantic meaning) that go with the question.
• Chit-chat detection: The detection of queries that don’t require a retrieval-augmented response, e.g., “How’s it going?”

‍

Query matching:

• Deterministic controls: A set of features to ensure the model is answering the question appropriately.
• Out-of-domain detection: Ensuring the model can answer the question using information from the knowledge library. If the information is unavailable, the model must tell the user it cannot answer the question rather than fabricating a response.
• Access control check: Ensuring the user has access to the documents the information is being retrieved from.
• Metadata-based filtering: Allowing the user to filter content and responses based on things like content author, date modified, etc.
• Matching models: The identification of a shortlist of potential answers.
• Reranking models: The ranking of the shortlist of potential answers, so the correct response is fed to the generative engine.

‍

What’s included in a generative engine

Of the three engines included in an enterprise RAG architecture, the generative engine is the one that users will interact with most — and whose underperformance will be most obvious. Here’s what a generative engine should include to ensure users get the responses they’re expecting:

• Response selection model: Determining which retrieved responses are most relevant.
• Answer summarization: The summarization of retrieved responses, performed by the LLM.
• Response attribution: Providing users with original source attribution with a response, so users can reference and verify the answer and explore additional context.
• Generative model: The LLM that generates responses. Some RAG applications (like those built with Pryon RAG Suite) provide bespoke LLMs but are compatible with open-source LLMs, like GPT-4 and LLaMa, as well.
• Feedback mechanism: The ability for users to provide feedback on responses (e.g., thumbs-up/thumbs-down or a star rating) to improve output over time.
  

‍

What’s included in the security and operations layer

• Cloud and on-premises portability: The ability of the RAG application to run in a variety of environments.
• Platform security: The assurance that the RAG application is resistant to hacking by bad actors.
• Content security: The assurance that an organization’s data won’t leak.
• Dashboard and analytics/activity logging: The interface admins can use to track usage of the RAG application, including who’s using it and how often users receive satisfactory answers. This data can also help subject matter experts populate the RAG application with the most important content users are looking for.

‍

How long does it take to implement a RAG architecture?

The time to implement a RAG architecture can vary between enterprises. Unfortunately, many AI applications take many months to deploy, owing to the time needed to scope a business case, determine technical feasibility, source and prepare data, build and test models, and actually implement. On the other hand, enterprises that adopt the pre-built Pryon RAG Suite can be production-ready in just 2-6 weeks.

‍

FAQs
‍

Why do I need a RAG architecture?
If you’re serious about deploying a RAG-based AI application in a thoughtful manner, you need a RAG architecture. A RAG architecture includes the key elements that make Enterprise RAG possible, including a security layer, platform configuration capabilities, and ingestion, retrieval, and generative engines.
‍

Can I implement an enterprise RAG architecture quickly?
With Pryon RAG Suite, you can implement an enterprise-class RAG architecture in just 2-6 weeks. Within this short period of time, you can scope, build, and test multiple use cases; connect your RAG application directly with your existing content sources; and get help from Pryon’s solution experts.
‍

Should I cobble together my own RAG architecture, or go with a prebuilt RAG architecture?
The decision to build or buy an enterprise AI application is complex. In general, if your goal is to develop a single or a couple of bespoke applications, a custom-built solution might make sense. However, if you envision a broader enterprise RAG architecture to support multiple applications across various departments, a proven, purpose-built platform could provide greater value and scalability. Purchased solutions have often been better tested, offer more support, and boast more security features, reducing the risks associated with custom development.
‍

Should my RAG architecture be totally verticalized, or can I use a RAG architecture composed of parts from multiple companies?
If you’re starting from scratch, it would likely be easier and less expensive to choose a single vendor for the many components of your RAG architecture. However, some organizations that already have some aspects of a RAG architecture in place (e.g., a generative engine they’re already comfortable with and that they’ve already built a front-end interface around) may choose to get other parts of the RAG stack, such as the ingestion and retrieval pieces, from a vendor like Pryon that offers modular solutions for RAG application builders.
‍

Why does data governance matter, and what's included in it?
As companies integrate generative AI applications (including those built on a RAG framework) into their systems, they must align these implementations with established data governance pillars to maintain data integrity, security, and compliance. These pillars include data quality, data security & privacy, data architecture & integration, metadata management, data lifecycle management, regulatory compliance, and data stewardship.