Semantic search finds relevant results based on meaning rather than keywords. This guide builds a complete semantic search system from document ingestion to ranked retrieval, using embeddings and a vector store.
Keyword search (BM25, Elasticsearch) finds documents containing the exact words in the query. A search for 'vehicle maintenance' misses documents that talk about 'car servicing' or 'automobile repair'. Semantic search embeds both the query and documents into a shared vector space, finding matches based on meaning regardless of word choice.
The two approaches are complementary. Hybrid search — combining BM25 scores with semantic similarity scores — outperforms either method alone. For most production systems, implement semantic search first, then add keyword search and blend the scores using Reciprocal Rank Fusion (RRF).
Step 1: Load documents. `from langchain.document_loaders import DirectoryLoader; docs = DirectoryLoader('./docs', glob='**/*.md').load()`.
Step 2: Split into chunks. Use `RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)`. Chunk size of 512 tokens balances specificity (small enough to match a specific answer) and context (large enough to be coherent). The 64-token overlap prevents losing context at chunk boundaries.
Step 3: Embed all chunks. Use `SentenceTransformer('BAAI/bge-m3').encode([c.page_content for c in chunks])`. Store embeddings alongside the chunk text and metadata in a numpy array or vector database.
For small corpora (<100K documents), FAISS is the fastest in-memory vector search: `import faiss; index = faiss.IndexFlatIP(embedding_dim); index.add(embeddings_array); distances, indices = index.search(query_embedding.reshape(1, -1), k=10)`.
Return the top-k chunks sorted by similarity score. Apply a minimum threshold (e.g. 0.75 cosine similarity) to filter irrelevant results. If no results exceed the threshold, fall back to a keyword search or return an 'I don't know' response.
The top-k results from vector search are close in semantic space but not necessarily ranked by relevance to the exact query. Re-ranking with a cross-encoder model significantly improves precision at the top of the results.
Use a cross-encoder: `from sentence_transformers import CrossEncoder; reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L6-v2'); scores = reranker.predict([(query, chunk) for chunk in top_k_chunks]); ranked = sorted(zip(scores, top_k_chunks), reverse=True)`. The re-ranker processes each (query, chunk) pair together and produces a more accurate relevance score.
For a question-answering system, take the top 3–5 re-ranked chunks and inject them into an LLM prompt: 'Use the following passages to answer the question. If the answer is not in the passages, say so. Passages: [chunks]. Question: [query].'
Always include source citations in the response. Surface the document name and page number alongside each cited fact so users can verify the source. This builds trust and makes errors easier to detect.
Text embeddings convert text into dense vector representations that capture semantic meaning. This guide covers generating embeddings with OpenAI and local models, measuring similarity, and practical applications.
Vector databases are purpose-built for storing and querying embeddings at scale. This guide compares the leading options — Pinecone, Weaviate, Qdrant, pgvector, and Chroma — and helps you choose the right one for your use case.
Embedding model choice significantly affects search quality and cost. This guide compares the leading models — OpenAI text-embedding-3, Cohere embed, BAAI/bge, and others — across quality, speed, cost, and language support.
Read guide