RAG Definition and LLM Glossary

Most Popular Definitions
‍

RAG: RAG stands for Retrieval-Augmented Generation. RAG is a framework for implementing generative AI using a curated set of trusted content as the basis of the generative output. It's a way for organizations to implement generative AI with confidence, because the large language models (LLMs) that create the output are grounded in trusted content — rather than whatever content a consumer-grade tool, like ChatGPT, might find somewhere on the Internet.

General AI/RAG/LLM Terms
‍

Bounding box and coordinates: To provide attribution to a specific sentence or paragraph within a document, a RAG system needs to be able to assign some kind of value to each content chunk. This is done through bounding box/coordinate values. The RAG system’s computer vision model analyzes the layout of a document and assigns each chunk a coordinate value. These coordinates are stored as metadata within a vector database. When the LLM delivers a response, it retrieves the coordinates of the source chunk, so a link to the source chunk (attribution) can be added to the response.

This concept is also referenced in the realm of object detection, a technique used in computer vision. Object detection models receive an image as input and output coordinates of the bounding boxes and associated labels of the detected objects. An image can contain multiple objects, each with its own bounding box and a label (e.g. it can have a car and a building), and each object can be present in different parts of an image (e.g. the image can have several cars). This task is commonly used in autonomous driving for detecting things like pedestrians, road signs, and traffic lights. Other applications include counting objects in images, image search, and more.

GPUs: GPUs, or graphics processing units, power LLMs and SLMs (small language models). Historically, computers were powered solely by CPUs, but in the last couple of decades, computer engineers realized GPUs were more efficient at certain computing tasks — not just producing graphics, as the name implies, but also tasks related to AI. As their usage has skyrocketed, GPUs — made by companies like NVIDIA — have consumed more and more energy and require more cooling methods, which can have major negative effects on the environment. Pryon’s proprietary set of LLMs and SLMs use less GPU resources than many competing language models, making Pryon RAG Suite an energy-efficient way to deploy generative AI at enterprise scale.

Neural networks: Neural networks are used in machine learning as a way to mimic how neurons in the human brain communicate. Just as neurons "fire" together to enable us to do things like walk, talk, and understand a book, neural networks are a series of nodes that work together to accomplish a certain task, such as summarizing text, doing natural language processing (NLP), and recognizing images.

The reason Pryon Ingestion Engine can understand enterprise content like a human would is its advanced deep learning techniques, such as visual segmentation and OCR (optical character recognition). These deep learning techniques are powered by neural networks.

Orchestration: The process of contributing and managing the interactions between different AI components to work together effectively. This process involves a central platform that manages the deployment, integration, and interaction of AI components such as databases, algorithms, AI models, and neural networks.

REST APIs: APIs make applications extensible, providing the ability for one application to integrate with another. They establish the content required from the consumer (the call) and the content required by the producer/provider (the response). For example, Weather.com’s API might call for a ZIP code and respond with the high and low temperature for that ZIP code.

REST APIs (aka RESTful APIs) are a kind of API that conforms to a specific architectural standard. Without going into a lot of technical detail, REST APIs are faster, more lightweight, and offer increased scalability than other kinds of APIs.

‍Small language models (SLMs): Like large language models (LLMs), SLMs are what provide the responses that generative AI application users receive. The difference is that unlike LLMs, which are built for general purpose use, SLMs are built for specific purposes. For instance, an LLM might be capable of everything from writing a poem to penning an academic dissertation, but an SLM might be trained specifically to answer questions posed by users (instead of generating original content). Because SLMs aren’t built for as many use cases as LLMs, they’re smaller in size, which is not only beneficial from a performance standpoint but also makes them far more energy-efficient.

Tokens: In natural language processing (NLP), a token is a fundamental building block of text. Tokens are the smallest unit into which a piece of text, such as a sentence or document, can be divided. This process of converting text to tokens is called tokenization.

Tokens are typically words, where each word in a sentence is considered a separate token. For example, in the sentence "I love NLP," there are three words and therefore three tokens: "I," "love," and "NLP." However, tokenization can also involve subword, character, or even sentence-level units, depending on the specific task and the tokenization approach used.

Tokens are the basic units of data processed by LLMs, so the more tokens an LLM can process, the more powerful the LLM. (It’s like cellular organisms; the more cells an organism has, the more complex that organism is.)

Transformers: Transformers are a kind of deep learning architecture designed to process natural language and other data. The transformer-based architecture was first described in a Google whitepaper in 2017 and gave rise to the modern LLM-based AI applications many of us are familiar with, such as ChatGPT. (Unrelated to the toys or Michael Bay films.)

Ingestion-Related Terms
‍

Connectors: An enterprise RAG product like Pryon RAG Suite uses connectors to ingest content directly from different repositories, such as SharePoint, Box, Amazon S3, Confluence, and Google Drive. This enables seamless, no-code content updates.

Embeddings/vectors: LLMs use vectors to represent text or data in a numerical form. Vectors are easy for an LLM to parse and process.

Embeddings are the high-dimensional vectors that LLMs use to capture the semantic meaning of words, sentences, or entire documents.

An ingestion engine divides the content into chunks of text. These chunks are encoded into an embedding vector and stored in a vector database. These embeddings are retrieved by the retrieval engine and used to form generative outputs.

‍

See also

Node/chunk retrieval

‍

File type handling: When an ingestion engine ingests and reads files in multiple formats, such as Word docs, PowerPoints, PDFs, images, and videos, in different ways. For instance, the ingestion engine might first convert an image-heavy PowerPoint into a PDF to enable better comprehension of the content.

Layout analysis: Analyzes the specific layout of each document (e.g., titles, headers, body copy, bullet points, footnotes, diagrams, and more) to understand the flow and hierarchy of information and extract the most important content.

Metadata capture: Captures additional document metadata, such as content author and date modified, to gather more context and better understand the content.‍

‍

Optical character recognition (OCR)/ Handwritten text recognition (HTR): Understands the words on the page, including handwritten text. Most RAG solutions can do OCR well for typed text, but understanding handwritten text can prove challenging for systems that aren’t designed to capture content that isn’t “neat.”

Segment prioritization: Prioritizes embeddings based on the original context in which text appeared in a document. For example, text from a body paragraph should rank higher in importance than text from a table of contents. As a result, a retrieval engine should place greater priority on retrieving the embeddings for the text chunks in a body paragraph than the embeddings for the text chunks in a table of contents.

Semantic segmentation: RAG solutions typically segment chunks of text arbitrarily (for instance, every chunk of text might be a fixed length of three sentences). With semantic segmentation, a RAG solution would instead intelligently segment the text into potential answers, whether those answers are two sentences or five sentences long.

Query-Related Terms
‍

Embedding generation: LLMs use embeddings to capture the semantic meaning of words, such as those in a query. When a question is asked, the LLM generates an embedding and feeds it to the retrieval engine. The retrieval engine matches this embedding with the closest embedding(s) from the vector database to provide the user with the most relevant response.

LLM inference: The process of entering a prompt and generating a response from an LLM. It involves a language model drawing conclusions or making predictions (so, inferring) from a prompt, so it can generate an appropriate output based on the patterns and relationships to which it was exposed during training. The better an LLM is at inference (understanding a user’s prompt), the better.

Named entity recognition (NER) and intent recognition: Extracts key entities and intent from the question. For instance, an employee at a mobile phone retailer might ask a chatbot, “How much does a galaxy cost?” The word “galaxy” can either refer to a type of Samsung phone or a celestial system. A sophisticated query engine should be able to infer the user’s intent (in this case, the employee is asking about phones, not stars), generate the appropriate embedding, and retrieve the best answer related to the user’s query.

Query backoff: Automatically retries failed queries by substituting specific entities with their broader topic families based on configurations, increasing the likelihood of retrieving accurate responses. To retrieve better answers for the question “What do I do if my Dell XPS is overheating?” a query engine might replace “Dell XPS” with “laptop” and retrieve a support article that mentions how to deal with overheating laptops.

Query disambiguation: When a RAG application prompts a user for additional context or clarification when necessary to ensure a more accurate response. For example, if a user's input is simply “Revenue,” the application might respond back, “Would you like to know the company’s revenue for the last quarter, last fiscal year, or another timeframe?”.

Query expansion: Expands acronyms and other organization-specific terms to return more accurate results. For instance, “SME” could stand for “subject matter expert” or “small or medium enterprise” depending on the organization and user; a RAG application should know which one the user intended to ask about and expand it to provide an appropriate response.

Query transformation: Breaks down complex queries into simpler, more manageable questions. Say, for example, you’re using a RAG application designed to answer questions about the Wizarding World franchise. If you ask, “How long was the first book in the series?” the RAG application might break this query down to first identify “the series” (Harry Potter), then the “first book” (Harry Potter and the Sorcerer’s Stone) before asking the retrieval engine for the needed information (the length of Harry Potter and the Sorcerer’s Stone) and providing an answer.

Query type handling: Determines how to handle each question by type. For example, a conversational (“chit-chat”) query, like “What’s for lunch?” or one that’s out of domain (e.g., “What are the nuclear launch codes?” might receive a canned response, while a legitimate query would get routed to the retrieval engine so an answer can be provided using content from the knowledge library.

Retrieval-Related Terms‍

Access control checks: Ensures the user has access to the documents the information is being retrieved from. If a user doesn’t pass an access control check for a certain document, that document and its contents won’t be available to the user. For instance, a low-level employee in the marketing department likely wouldn’t be able to gain access to payroll data.
‍

See also

User-based document-level access controls
‍

Context length configuration: If desired, a developer may choose to configure the length of a context chunk. The longer the context, the more expensive (compute-wise) it is, but longer context lengths also increase the accuracy of a response, since the prompt the LLM receives contains more information. Of course, LLMs can only handle a finite number of tokens, which impacts the maximum context length.

Deterministic controls: Measures put in place to ensure a generative AI system follows certain rules or protocols to answer user questions. Deterministic controls can help ensure greater consistency in answers provided to users. For instance, a content admin might implement a deterministic control such that a user query that includes a certain word always leads to an answer being retrieved from a certain set of documents. Another form of deterministic controls include Pryon RAG Suite’s Verified Answers, which enable admins and SMEs to predefine answers to certain questions (e.g., “Where is our company overview?” will always provide a link to a particular file).

Expanded Context Retrieval (Chunk / Parent Document Retrieval): Sometimes, a chunk is small — this is great for compute and cost reasons, but a small chunk on its own may not provide enough information to deliver a comprehensive-enough answer. Parent Document Retrieval allows a retrieval engine to retrieve additional context from the parent document that contains a chunk, thereby enabling the retrieval engine to deliver an answer with complete context.

Matching models: Identify a shortlist of potential answers by matching the semantic meaning of the user’s query (as captured through embeddings) with the closest matching chunks from the vector database.

Metadata-based filtering: Allows the user to filter content and responses by document properties, including owner, date last modified, and file type.

Metadata retrieval: Metadata includes information related to a document, such as file path, owner, date last modified, and more.  Having this information on hand can provide the retrieval engine with additional context, enabling the retrieval engine to decide whether the document in question is relevant to the user’s query. This metadata can also be shown to the user to enable metadata-based filtering.

Node/chunk retrieval: Foundational to RAG is the idea of chunking. A document’s contents are ingested and then chunked; those chunks are encoded into an embedding vector and stored in a vector database.

A retrieval engine, such as Pryon Retrieval Engine, retrieves chunks and feeds a prompt to the LLM consisting of the chunks that closely match a user’s query as well as the user’s (vectorized) query itself.  Nodes are another term for chunks.
‍

See also

Embeddings/vectors
‍

Out-of-domain detection: Ensures the model can answer the question using information from the knowledge library. If the information is unavailable, the model should tell the user it cannot answer the question rather than fabricating (hallucinating) a response.
‍

See also

Hallucinations
‍

Reranking models: Rank the shortlist of potential answers using a series of methods, including vector matching, to identify the correct response to feed into the generative model.

‍

Spell check: The ability for a retrieval engine to perform spell check on a user’s misspelled query.

Generative-Related Terms

Ways to enhance the performance of an LLM
‍

Fine-tuning: Most large language models (LLMs) are versatile, but if you have a specific task in mind, such as answering questions, you might benefit from fine-tuning a model for that purpose.

Fine-tuning is done by changing the parameters of the LLM to train the LLM so it’s better at a particular task.

Fine-tuning can significantly enhance an LLM’s performance for specific tasks, though it may come at the expense of no longer being able to perform other tasks. For instance, an LLM fine-tuned so it performs well at information retrieval might not do as good a job at things like writing advertising copy or brainstorming new product ideas.

At Pryon, our engineering team found dramatic improvements in accuracy and latency from fine-tuning our generative LLMs to focus on question-answer tasks.

Hyperparameter tuning: Hyperparameters are adjustable parameters, such as the quantity of units in a neural network and the learning rate, that determine the rate at which a machine learning algorithm learns. Hyperparameter tuning is the process of optimizing these parameters to maximize the algorithm’s performance. Hyperparameters are set before an LLM is trained, so they must be re-tuned every time the LLM trains on new data.

Parameter-efficient fine-tuning (PEFT): A fine-tuning method that, as the name states, is parameter-efficient — meaning this method minimizes the amount of required memory. PEFT improves the performance of an LLM with minimal computational cost and data.

Prompt engineering: When a data scientist strategically provides an LLM with input queries to guide the LLM in generating accurate and relevant responses.

Essentially, with prompt engineering, the data scientist molds the input for a generative LLM to produce a better output for whatever application they’re trying to build. For example, the LLM of a RAG application focused on helping lawyers may be prompted never to inject humor into a response, lest a joke be misinterpreted.

‍

Additional generative-related terms
‍

Answer snippets: With RAG, responses are delivered in the form of answer snippets, so people don’t have to read through an entire document to find the answer they’re looking for (this is the main selling point of RAG!).

Answer snippets should include source attribution, so users can refer back to the source and gather additional context if desired.

See also

Response attribution
‍

Feedback mechanism: Enables users to provide feedback on responses (e.g., thumbs-up/thumbs-down or a star rating) to improve output over time.

Hallucinations: Hallucinations are incorrect or nonsensical responses provided by generative LLMs. Consumer-grade generative AI tools, such as ChatGPT, are especially prone to produce hallucinations because they search for information across the internet and can’t separate fact from fiction. RAG-based applications are less likely to hallucinate because their LLMs retrieve highly accurate, fully attributed answers solely from trusted content and can identify when a question is out of domain/scope.

Security-Related Terms

Data segregation: Determines which departments should access specific data. For example, your finance team should not have access to your HR team's data, and vice versa. Consequently, a RAG solution could choose to ingest the two departments’ documents separately or store the documents’ embeddings in different vector databases.

Data retention policies: Determines how long information, such as ingested data and input queries, is retained. Data retention policies can apply either to a RAG application or a generative LLM.

Encryption in transit and at rest: Ensures content and questions remain encrypted at all times using industry-standard protocols (e.g., 256-bit AES) so bad actors can’t gain access to sensitive data or to users’ queries.

Filtering functionality: Sometimes, you don’t want to expose certain sensitive information, like confidential or proprietary data or PII, to LLMs. Private content filters can identify sensitive data, redact or mask it, encrypt it, enforce access controls, and integrate with a RAG pipeline.

Segment security score: Groups data into categories, such as personal identifiable information (PII) or protected health information (PHI), and applies security and authentication measures to each.

Service and Organization Controls (SOC) 2: A cybersecurity compliance framework that helps ensure third-party service providers store and process client data securely. The framework establishes criteria for maintaining high data security, availability, processing integrity, confidentiality, and privacy. Pryon RAG Suite is SOC 2-compliant.

User-based document-level access controls: Identifies whom should and should not be given access to certain content by user level. As an example, a junior accountant should probably be restricted from accessing a payroll document listing the salaries of all senior-level executives at a company.
‍

See also


Access control checks