How to build a multimodal RAG system for documents with images

Many business documents — annual reports, research papers, technical manuals — contain charts, tables, and diagrams that convey critical information not repeated in the text. Standard RAG pipelines that only process text will miss this information entirely, leading to incomplete or incorrect answers.

Multimodal RAG addresses this by extracting images from documents, generating text descriptions or embeddings of those images, and including them in the retrieval index. When a query requires information from a chart, the system can retrieve and reason over the image directly.

Use `pymupdf` (formerly PyMuPDF) to extract both text and images from PDFs: `import pymupdf; doc = pymupdf.open('report.pdf'); for page in doc: text = page.get_text(); image_list = page.get_images()`. For each image, render it as a PNG: `img_data = doc.extract_image(xref)['image']`.

Alternatively, render each page as a high-resolution image and use a vision model to extract all text and describe all figures — this avoids separate text extraction and catches text embedded in images (scanned documents). Use 200+ DPI for legible text: `pix = page.get_pixmap(dpi=200); pix.save('page.png')`.

For each extracted image, send it to GPT-4o or Gemini with a specific description prompt: 'Describe this chart/diagram in detail. Include: (1) the chart type, (2) what is being measured, (3) all data values, (4) trends or patterns, (5) any labels or annotations. Be comprehensive — your description will be the only representation of this image in a search index.'

Store the description alongside the image and associate it with the page number and document metadata. The description is what gets embedded and indexed. At retrieval time, you can include the image itself in the prompt for visual grounding.

Create two separate collections in your vector database: one for text chunks, one for image descriptions. Embed both with the same text embedding model. At query time, search both collections and merge the results by relevance score.

For the final generation step, include the retrieved images directly in the prompt: retrieve the top-3 text chunks and top-2 image descriptions, then build a multimodal prompt with both the text and the actual images. The vision model can then reason over the visual content directly rather than relying on the text description.