Retrieval-Augmented Generation (RAG) extends LLMs with your own documents, giving accurate, source-grounded answers without fine-tuning. This guide builds a working RAG system from document ingestion to response generation.
LLMs have a knowledge cutoff and cannot know about your private documents. RAG solves this by retrieving relevant text chunks from a document store at query time and injecting them into the prompt as context. The model then answers based on the retrieved context rather than its training data.
RAG is ideal for: Q&A over internal documentation, customer support bots with a knowledge base, code search over a large codebase, and any application where accuracy and sourcing are critical.
1. Ingestion: Load documents, split them into chunks, convert each chunk into a vector embedding, and store the embeddings in a vector database.
2. Retrieval: When a query arrives, embed the query with the same model, search the vector database for the N most similar chunks, and return them.
3. Augmentation: Inject the retrieved chunks into the prompt as context, along with the original query.
4. Generation: Pass the augmented prompt to the LLM and return the response.
Install dependencies: `pip install langchain openai chromadb tiktoken pypdf`. Load PDFs: `from langchain.document_loaders import PyPDFLoader; docs = PyPDFLoader('document.pdf').load()`. Split into chunks: `from langchain.text_splitter import RecursiveCharacterTextSplitter; splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200); chunks = splitter.split_documents(docs)`.
Create embeddings and store in ChromaDB: `from langchain.vectorstores import Chroma; from langchain.embeddings import OpenAIEmbeddings; vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings(), persist_directory='./chroma_db')`. Call `vectorstore.persist()` to save to disk.
Load the persisted vectorstore: `vectorstore = Chroma(persist_directory='./chroma_db', embedding_function=OpenAIEmbeddings())`. Create a retriever: `retriever = vectorstore.as_retriever(search_kwargs={'k': 5})`.
Build the RAG chain with LangChain: `from langchain.chains import RetrievalQA; from langchain.chat_models import ChatOpenAI; qa_chain = RetrievalQA.from_chain_type(llm=ChatOpenAI(model='gpt-4o-mini'), chain_type='stuff', retriever=retriever, return_source_documents=True)`.
Query: `result = qa_chain({'query': 'What is the refund policy?'}); print(result['result']); print(result['source_documents'])`. The response includes citations to the source chunks.
Hybrid search: combine dense vector search with BM25 keyword search. Dense search finds semantically similar content; BM25 finds exact keyword matches. Combining both with a reranker reduces retrieval failures.
Chunk overlap is critical for maintaining context at boundaries — use `chunk_overlap=200` to ensure sentences split at chunk boundaries are included in both chunks. Too little overlap leads to fragmented context; too much wastes tokens.
Add metadata filtering: tag each chunk with its source document, section, and page number. Let users filter by source and include metadata in the context so the model can cite specific pages.
Hermes is NousResearch's open-source model family trained specifically for function calling and agentic workflows. This guide covers running Hermes locally, defining tools in JSON Schema, and building a basic tool-calling loop.
Read guideReAct (Reason + Act) is the foundational pattern for LLM agents. This guide builds a working ReAct agent from scratch using the OpenAI API and function calling, with a web search and calculator tool.
Read guide