This work was collaboratively completed by Idan Arbiv and Gal Fadlon.
Large language models (LLMs) have undoubtedly changed the way we interact with information. However, they come with their fair share of limitations as to what we can ask of them. Base LLMs (ex. Llama-2-70b, gpt-4, etc.) are only aware of the information that they’ve been trained on and will fall short when we require them to know information beyond that. Retrieval augmented generation (RAG) based LLM applications address this exact issue and extend the utility of LLMs and their generative reasoning abilities to our unique datasets.
​
In this guide, we will develop a toy example of a RAG-based LLM application. The goal is to demonstrate how RAG enhances LLM capabilities by integrating retrieval mechanisms that fetch relevant documents before generating responses. Specifically, we will build an assistant capable of answering questions about specific CVs.

Building a RAG Based LLM Application

Retrieval-augmented generation (RAG) is an AI framework designed to enhance the quality of responses generated by large language models (LLMs) by supplementing the model’s internal knowledge with external sources of information. This approach improves the reliability and accuracy of the model’s output.​In a question-answering system, RAG offers two key advantages: it ensures the model has access to up-to-date, verified facts, and it provides users with transparency by allowing them to trace the model’s sources, making it easier to verify the accuracy of its claims.​Additionally, RAG helps mitigate risks associated with relying solely on the model’s pre-trained knowledge. By grounding the model in external, verifiable data, the chances of the LLM inadvertently disclosing sensitive information or generating false or misleading content—often referred to as “hallucinations”—are significantly reduced.​Lets see briefly the key components and the flow: ​Indexing Phase:​Load data → Split data into chunks → Embed chunks → Store embeddings in an index.​​Query Phase:​Input query → Retrieve relevant data from the index → Augment the query with retrieved data → Generate a response using the augmented data.​1. Indexing Phase: Data Loading, Splitting, Embedding, StoreIndexing is the process of preparing the external knowledge sources to be efficiently queried by the model during the generation phase. This step involves several key actions:​​Data Loading: First, the relevant external data (e.g., text documents, knowledge bases, research papers, etc.) is loaded into the system. This could involve reading from a database, a set of files, or online sources.​​Data Splitting: The data is split into manageable chunks. For example, a large document might be split into smaller sections, paragraphs, or sentences. This helps the retrieval system focus on specific, relevant pieces of information.​​Embedding: Each chunk of data is transformed into a vector representation using a method like sentence embeddings (e.g., using models like BERT, RoBERTa, or more advanced embedding models). These embeddings capture the semantic meaning of the data and make it easier for the system to compare and retrieve relevant information later.​​Store: The embeddings, along with the original text data, are stored in an index or a vector database. The index allows for fast retrieval of the most relevant data chunks based on a given query. This setup ensures that the model can quickly search through large amounts of external knowledge.​​
​​2. Query Phase - Retrieval, Augmenting, GenerationOnce the data is indexed and ready, the system proceeds with the retrieval-augmentation-generation process:​​Retrieval: When a user query or prompt is received, the system first retrieves relevant information from the indexed external data. The query is embedded into a vector using the same embedding model used during indexing, and then the system searches for the most similar vectors (i.e., data chunks) to the query in the index. The result is a set of relevant documents or text passages.​​Augmenting: The retrieved information is augmented by the model, i.e., it is used to provide additional context for answering the query. This helps the model answer more accurately by grounding its response in real-world facts or up-to-date data. The model combines its internal knowledge with the external, retrieved data to create a response that is both coherent and factually grounded.​​Generation: After augmenting the input with relevant external information, the model proceeds to generate a response. In this step, a language model (typically based on architectures like GPT, T5, or BART) is used to formulate a response, taking into account both the user’s query and the augmented external knowledge. This response is more informed and reliable because it’s grounded in verifiable data from the retrieval step.​​
​This flow ensures that the model can produce accurate, fact-based, and up-to-date answers while minimizing the risk of “hallucination” or generating incorrect information. The key is that external knowledge is continually used to refine and validate the LLM’s responses, resulting in a more trustworthy output.
RAG vs Fine-tuning​Fine-tuning refers to the process of adjusting a pre-trained model’s parameters using a smaller, task-specific dataset. It involves training the model further on the new dataset to specialize it for a particular application, such as answering questions about a specific domain or generating responses in a certain style.​​Key Differences Between Fine-Tuning and Retrieval-Augmented Generation (RAG):​Purpose: Fine-tuning: Specializes a model on a specific task or dataset. RAG: Enhances a model’s responses with external knowledge. ​Data Usage:​ Fine-tuning: Relies on a labeled dataset for further training. RAG: Uses external, unstructured data for retrieval. ​Knowledge Source:​ Fine-tuning: Knowledge comes from the model’s pre-trained parameters. RAG: Knowledge is dynamically retrieved from an external source. ​Model Modification:​ Fine-tuning: Changes model parameters based on new data. RAG: No change to the model parameters, just augmentation with retrieved information. ​Flexibility:​ Fine-tuning: Limited to the domain covered in the fine-tuning data. RAG: Flexible, can use any external knowledge dynamically.​​​​

Indexing Pipeline

Data Loading

Data Splitting

We choose to split the text into chunks of 100 tokens with no overlap, but you can adjust this based on your data by experimenting with different chunk sizes and adding overlap to preserve more context.

Embedding

This line normalizes each embedding vector in chucksEmbeddings by applying the Normal function to each element, ensuring that all embeddings have a consistent scale, which improves similarity comparisons in the RAG system.
Mapping between the normalized embeddings and their corresponding text chunks by pairing each embedding from normalizedChucksEmbeddings with its original text from chucks, forming a list of key-value pairs for efficient retrieval.
FeatureSpacePlot[dataWithLabels] and FeatureSpacePlot3D[embeddingsToText] allow us to visually explore the embedding space of our text chunks by plotting their vector representations in 2D and 3D, respectively. These plots help us understand how the model clusters similar chunks together based on their semantic meaning. By hovering over a specific data point, the user can see the corresponding text, making it easier to interpret relationships between different chunks and assess the effectiveness of the embedding model in capturing meaningful similarities.

RAG Pipeline

Retrieval

Retrieval step in a Retrieval-Augmented Generation (RAG) system focuses on finding the most relevant chunks for a given query. When a user submits a query, its embedding is computed and compared against the stored embeddings using a similarity metric (e.g., cosine distance). The system then retrieves the most semantically similar text chunks, ensuring that the model has relevant contextual information before generating a response. This retrieval process enhances the model’s accuracy by grounding its output in factual, retrieved data rather than relying solely on its internal knowledge.
nfunction as a nearest neighbor search using the CosineDistance metric to find the most similar text chunks based on their embeddings, though other distance metrics like Euclidean or Manhattan can also be used to measure similarity.
An example of the term embedding vector of the term “Diffusion Models”
An example of how we find the most similar text chunks to the term “Diffusion Models” by first computing its embedding with getEmbedding[“Diffusion Models”] and then using nfunction to retrieve the top 3 closest matches based on cosine similarity. The result, wlExamplesContext, contains the three most relevant text chunks, which can be used to provide context for answering related questions.

Augmentation

Augmentation part refers to enriching the model’s ability to answer a query by first retrieving relevant external context or information. In this example, the query (query) is augmented with additional context (content) obtained through a retrieval process, where related tasks or information are identified and provided to the model. This process helps the model generate a more accurate and informed response by incorporating external knowledge into the generation process, as seen in the template where the retrieved context directly informs the final answer generation. The model leverages this “augmented” input to provide a more relevant and context-aware answer.

Generation

We integrate the previously discussed components into a single agent, cvAgent, which processes a query by first obtaining an embedding and then retrieving related tasks using a function. The flow begins with generating the task embedding for the query, followed by identifying contextually relevant tasks, and finally using the context to craft a prompt, which is sent to the OpenAI service for generating a response based on the provided content.

Example Queries and Outputs with our RAG -Agent

Conclusion

In this notebook, we explored a toy example of how Retrieval-Augmented Generation (RAG) can be used to enhance an LLM’s ability to answer queries about an unfamiliar data source. While this was a simple demonstration, the same principles can be extended to larger document collections and more advanced RAG architectures, incorporating techniques like vector databases, hybrid search, and reranking models. The goal here was to illustrate the core ideas behind RAG—retrieving relevant context and leveraging an LLM to generate informed responses—providing a foundation for building more scalable and sophisticated retrieval-based AI systems.

CITE THIS NOTEBOOK

Enhancing LLMs with retrieval-augmented generation: a practical example using CV data​
by Idan Arbiv & Gal Fadlon​
Wolfram Community, STAFF PICKS, February 7, 2025
​https://community.wolfram.com/groups/-/m/t/3387426