Retrieval-Augmented Generation

Have you ever encountered a scenario like this:

• You have a product manual that’s hundreds of pages long, and a user asks a very specific question. You search for a long time but can’t find where the answer is.

• Your company has accumulated several years of internal documents, meeting minutes, and technical specifications. New employees who want to understand a particular business rule have no idea where to even start looking.

• You have a pile of ebooks, research reports, and papers, and you want AI to answer questions based on this content, but if you paste the entire book directly to AI, the context window won't fit it.

This is the problem RAG aims to solve.

RAG (Retrieval-Augmented Generation) is a technology that enables large language models to answer questions based on specific knowledge bases.。

The core idea of RAG is simple: first retrieve the most relevant content from your knowledge base, then send that content together with the user's question to the LLM, and let the LLM answer based on this context.

Without RAG, the LLM only knows what's in its training data; with RAG, the LLM can use your private, up-to-date data.

Why is RAG needed?

RAG is not the only solution to these problems, but it is currently one of the most practical and lowest-cost options.

LLM knowledge cutoff issue.

• All LLMs have a knowledge cutoff date—the training data only goes up to a certain point in time, and it doesn't know about events after that.

• For example, GPT-4's knowledge cutoff is October 2023. If you ask it about news from 2024, it can only say 'I don't know.'

• More importantly, your private data (internal documents, product manuals, company policies) has never appeared in the LLM's training data—how could it possibly know?

Private data cannot be directly input into an LLM.

You might think: can't I just paste the documents directly into the LLM?

The answer is: short documents work, but long documents don't.

• First, the LLM's context window is limited. For example, GPT-3.5 only has 16K tokens, roughly 10,000 characters. A manual of several hundred pages simply won't fit.

• Second, even if the context window is large enough, stuffing the entire document in doesn't work well. LLMs tend to get lost in long text and fail to find the truly relevant information.

• Finally, there's the cost. GPT-4 with a 128K context costs $10 per million input tokens. If you stuff a whole book in every time, your wallet will cry.

RAG vs Fine-tuning: How to Choose

Another way to give an LLM new data is fine-tuning—continuing to train the model with new data.

But fine-tuning and RAG are fundamentally different, with entirely different use cases:

A simple rule of thumb: if you want the LLM to "know" certain factual knowledge (such as product descriptions, company policies), use RAG; if you want the LLM to "learn" a certain style or capability (such as writing style, code conventions), use fine-tuning.

RAG Architecture Overview

RAG is divided into two phases: the offline indexing phase and the online retrieval phase.

First, let's look at a complete architecture diagram:

Offline indexing stage (data preparation)

This phase runs in the background, and users won't see it directly.

Its task is to process your documents into a format that can be quickly retrieved.

The steps are as follows:

• 1. Document loading: Read documents in various formats such as PDF, TXT, DOCX, and web pages.

• 2. Document splitting: Split long documents into smaller text chunks, usually a few hundred to a thousand words.

• 3. Vectorization: Use an Embedding model to convert each text chunk into a vector (a string of numbers).

• 4. Storage: Store the vectors and original text together in a vector database.

This phase only needs to be done once, or rerun when documents are updated.

Online retrieval stage (user query)

This phase happens in real time when the user asks a question.

The steps are as follows:

• 1. Question vectorization: Use the same Embedding model to convert the user's question into a vector as well.

• 2. Similarity retrieval: Find the text chunks closest to the question vector in the vector database.

• 3. Build Prompt: Use the retrieved text chunks as context, and assemble them together with the user's question into a Prompt.

• 4. LLM generation: Send the Prompt to the LLM, and let it answer based on the context.

• 5. Return the answer: Return the LLM's answer to the user, usually along with the cited document sources.

Introduction to Vector Databases

The vector database is one of the core components of RAG. To understand it, you first need to understand what a vector is.

What is a vector (Embedding)?

A vector (Embedding) is a string of numbers converted from text.

For example, the word "cat" might be converted into a vector with several hundred dimensions like [0.23, -0.45, 0.12, 0.89, ...].

The key point is:Texts with similar semantics will have vectors that are close together in space.。

For example:

• The vector distance between "cat" and "kitty" is very close.
• The vector distance between "cat" and "dog" is closer than between "cat" and "car".
• The vector distance between "I like cats" and "I love cats" is very close.

This is the foundation for "semantic search" — matching by meaning, not by keyword.

The computational principle of semantic similarity.

The similarity of two vectors is usually calculated using "cosine similarity".

The range of cosine similarity is -1 to 1:

• 1 means exactly the same
• 0 means unrelated
• -1 means completely opposite

In practice, we usually only care about positive similarity; the closer to 1, the more relevant.

When you ask a question, the vector database quickly computes the similarity between the question vector and all document vectors, returning the most relevant top few.

Mainstream vector database

There are many vector databases on the market, each with its own characteristics:

For beginners, we recommend starting with Chroma — it is easy to install, requires no extra configuration, and is very suitable for learning and prototyping.

Document Processing Workflow

Document processing is a step in RAG that is easily overlooked but actually very important.

Whether your retrieval works well depends largely on how well the documents are processed.

Supported document formats.

LangChain supports many document formats:

• Plain text: .txt, .md
• PDF：.pdf
• Office：.docx, .pptx, .xlsx
• Web pages: HTML, URL
• Code: .py, .js, .java, etc.
• JSON：.json, .jsonl
• CSV：.csv

Each format has a corresponding loader, making it very convenient to use.

Document segmentation strategy

Document chunking is the process of splitting long documents into smaller pieces.

This looks simple, but there are many nuances:

• Too small: may lose context, a complete meaning gets cut off
• Too large: may contain irrelevant information, wasting the context window

Common chunking strategies:

In most cases, using LangChain's RecursiveCharacterTextSplitter is sufficient.

Tips for choosing the segmentation size

Chunk Size is a parameter that needs to be adjusted according to the scenario.

Some reference guidelines:

• QA scenarios: 500-1000 characters, enough to contain a complete question-answer pair
• Summarization scenarios: 1000-2000 characters, more context is needed
• Code scenarios: can split by function or class, keeping code blocks intact

There is also an important parameter: Overlap — the number of characters overlapped between adjacent text chunks.

Overlap can prevent important information from being split across two chunks, usually set to 10%-20% of the chunk size.

Don't blindly chase "optimal" chunking parameters. Start with 500 characters and 50 overlap, test the actual results, then adjust.

Embedding Model

An embedding model is a model that converts text into vectors.

Its quality directly affects retrieval performance — a good embedding model can accurately understand semantics and rank relevant documents higher.

What is an embedding model?

An Embedding model is a pretrained neural network that takes text as input and outputs vectors.

Its training objective is: for texts with similar semantics, the output vectors should also be similar.

For example:

• "How to learn Python" and "Python learning methods" → vectors are close
• "Cats are cute" and "Puppies are cute" → vectors are somewhat distant but still relatively close
• "How to learn Python" and "Cats are cute" → vector distance is very far

OpenAI text-embedding-3 series

OpenAI provides the most popular embedding models currently:

text-embedding-3-small is currently the top choice—it performs better than the previous generation text-embedding-ada-002 and is cheaper.

Open-source Embedding Model

If you don't want to use OpenAI, or have data privacy requirements, there are also many open-source options:

For Chinese scenarios, we recommend the BAAI/bge series or Alibaba-NLP/gte series—they are specifically optimized for Chinese and perform better than general-purpose models.

Retrieval strategy

Retrieval is not as simple as "find the most relevant documents"—different strategies suit different scenarios.

Vector similarity search

This is the most common retrieval method: compute the similarity between the question vector and all document vectors, and return the top K most relevant ones.

Advantages: understands semantics, synonyms and near-synonyms can all be matched.

Disadvantages: may be less accurate than keyword retrieval for precise terms and proper nouns.

For example, if a user asks "example's Python tutorial", vector retrieval can understand that "tutorial" and "course" are similar, but if the user asks "example Python3.10 documentation", keyword search may be more accurate.

Keyword search (BM25)

This is the method used by traditional search engines: match keywords and score based on term frequency, inverse document frequency, etc.

• Advantages: effective for exact matching of terms, product names, model numbers, etc.

• Disadvantages: cannot understand synonyms or semantic relationships.

For example, if a user asks how to learn programming, keyword retrieval can only find documents containing "programming" and cannot find documents that say "writing code" but mean the same thing.

Hybrid Search

Since both methods have their pros and cons, why not use them together?

Hybrid search performs both vector retrieval and keyword retrieval simultaneously, then merges and re-ranks the results.

This way, it can leverage semantic understanding while ensuring exact matching.

Many vector databases support hybrid search, such as Weaviate, Pinecone, Qdrant, etc.

If you're not sure which one to use, choose hybrid search—in most cases, it works best.

Generation phase optimization

Retrieving relevant documents is only the first step. There are also many techniques for getting the LLM to generate good answers based on those documents.

Prompt template design

The prompt template is a key factor affecting output quality in RAG.

A good RAG prompt usually includes:

• Role setting: tell the LLM what role it is
• Context: the retrieved document content
• Task description: what to make the LLM do
• Constraints: for example, "answer only based on the context" and "if you don't know, say you don't know"
• Output format: how you want the LLM to answer

A classic template:

How to reduce hallucinations?

Hallucination is the phenomenon where an LLM fabricates facts—it can sound very convincing, but it's all made up.

RAG can reduce hallucinations, but cannot completely eliminate them.

Some techniques for reducing hallucinations:

• Emphasize "answer only based on the context" in the prompt
• Have the LLM cite excerpts from the original text in its answers
• If the retrieval results are irrelevant, have the LLM say "I don't know"
• Use multiple LLMs to cross-validate answers
• Have the LLM first find relevant content from the context, then answer

Source citation annotation

It's important to let users know where the answer comes from—this not only increases credibility but also makes it easier for users to verify.

Common practices:

• Mark the source in the answer, e.g., "according to Document A, page 2"
• List the titles and links of reference documents at the end of the answer
• Allow users to click and jump to the original text location

Some advanced frameworks in LangChain (such as LlamaIndex) can do this automatically for you.

Hands-On: Building a Document Q&A System Based on LangChain

Enough talk—let's get hands-on and build a complete RAG system.

We will use LangChain + Chroma + OpenAI to build a document Q&A system.

Environment setup

First install the required libraries:

pip install langchain langchain-openai langchain-chroma python-dotenv

Then create a .env file in the project directory and fill in your OpenAI API Key:

OPENAI_API_KEY=你的-api-key-在这里

Complete code implementation

This is a complete RAG system, including index creation and query functionality:

Running this program, you will see output similar to this:

创建新的数据库...
已加载 4 个文档
已将 4 个文档切分为 4 个文本块
向量数据库已创建，共 4 条记录

============================================================
问题：EXAMPLE 是什么时候创立的？
============================================================
回答：EXAMPLE 教程网站创立于 2013 年。

参考来源：
  [1] example_intro.txt (第 1 页)
      片段：EXAMPLE 教程网站创立于 2013 年，由Example团队创建。...
  [2] features.txt (第 1 页)
      片段：除了文字教程，EXAMPLE 还提供了视频教程。...

============================================================
问题：EXAMPLE 的 Python 教程有什么特点？
============================================================
回答：EXAMPLE 的 Python 教程涵盖了 Python 的所有重要知识点，包括基础语法、数据类型、控制语句、函数、模块、文件操作等，并且提供了大量经过精心设计和测试的代码示例，还提供了在线 Python 编辑器，可以直接在浏览器中运行代码。

参考来源：
  [1] python_tutorial.txt (第 1 页)
      片段：EXAMPLE 的 Python 教程涵盖了 Python 的所有重要知识点。...
  [2] features.txt (第 1 页)
      片段：EXAMPLE 教程的特点是内容简洁、示例丰富、注重实践。...

Congratulations! You have successfully built a complete RAG document Q&A system.

Advanced Optimization: Re-ranking, HyDE

Basic RAG works, but if you want better results, you can try these advanced optimization methods.

Re-ranking

The results returned by vector retrieval are sorted by similarity, but high similarity does not necessarily mean it is the most suitable answer to the question.

Re-ranking means: first use vector retrieval to return the top K results (e.g., 20), then use a more precise model to re-rank these results, and select the best few to send to the LLM.

Common Re-ranking models:

• cross-encoder/ms-marco-MiniLM-L-6-v2
• BAAI/bge-reranker-large

Although it adds an extra step, the improvement in results is obvious and worth doing.

HyDE（Hypothetical Document Embedding）

HyDE is a clever idea: first use the LLM to generate a "hypothetical answer" for the user's question, then use this hypothetical answer for retrieval.

Why is this effective? Because questions are usually short and have low information density; while hypothetical answers are longer, more document-like in style, and have higher similarity to real documents.

For example:

• User question: "When was EXAMPLE founded?"
• Hypothetical answer: "The EXAMPLE tutorial website was founded in 2013, created by the Rookie Tutorial team, aiming to provide quality tutorials for programming beginners."

Using the hypothetical answer for retrieval often finds more relevant documents.