Introduction
Introduction
In a previous Community post by Idan Arbiv and Gal Fadlon, we saw how to build a retrieval-augmented generation (RAG) system in the Wolfram Language using CV data. That post walked through chunking resumes, embedding them into a vector database with , retrieving relevant context with , and synthesizing answers with . It worked—and for many questions, it worked well.
CreateSemanticSearchIndex
SemanticSearch
LLMSynthesize
But if you experimented with that pipeline—as I did while building on Idan and Gal's foundation—you likely discovered a frustrating class of failures. Try asking "What tools has Roee Ziv used for agent orchestration?" and the system might confidently return chunks about someone else's engineering work. The embedding model understands the meaning of the query—"a person and their technical tools"—but treats the proper noun "Roee Ziv" as just another token in a semantic cloud. Exact identifiers like names, dates, certification codes, and version numbers get diluted in the high-dimensional embedding space.
This is the keyword gap, and it is one of the most common failure modes in production RAG systems. In this sequel, we close that gap with two powerful techniques: Hybrid Search and Reranking.
Why Pure Semantic Search Fails
Why Pure Semantic Search Fails
Vector embeddings are remarkable at capturing meaning. The query "machine learning experience" will correctly match chunks about "deep learning projects" and "neural network development." That is their strength.
Their weakness is lexical precision. Embedding models compress text into fixed-dimensional vectors, and in doing so, they sacrifice the ability to distinguish between surface-level tokens that carry critical identity information. Consider these two resume chunks:
Chunk A: "Roee Ziv | AI Lead & Co-Creator at Blabber. Engineered a stateful LangChain and LangGraph conversational agent with dynamic transitions and persona adaptation."
Chunk B: "Noa Levi | ML Lead at SynthAI. Built a stateful multi-agent orchestration system with dynamic routing and context-aware persona switching."
Chunk B: "Noa Levi | ML Lead at SynthAI. Built a stateful multi-agent orchestration system with dynamic routing and context-aware persona switching."
Semantically, these are nearly identical—both describe a lead engineer building stateful AI agent systems. Their embedding vectors will be extremely close. But if a recruiter asks specifically about Roee Ziv, only Chunk A is correct. A pure vector search has no reliable mechanism to enforce that distinction.
Even when semantic search happens to return the right person's chunks, it does so by accident—because the correct chunks were marginally closer in embedding space, not because the model understood who the query is about. This fragility becomes catastrophic at scale: with 50 or 500 candidates instead of 4, the odds of a lucky tiebreak collapse.
Keyword search (traditionally implemented via algorithms like BM25) solves exactly this problem. It operates on term frequency: if the query contains "Roee Ziv" and a chunk contains "Roee Ziv," that chunk gets a high score. It does not understand synonyms or meaning—but it is ruthlessly precise about tokens.
The insight behind hybrid search is simple: combine both signal types, letting semantic understanding and lexical precision compensate for each other's blind spots.
Architecture Overview
Architecture Overview
Our upgraded pipeline has three stages:
Query → [1. Hybrid Retriever] → Candidate Set → [2. Reranker] → Top Chunks → [3. LLM Synthesis] → Answer
1
.Hybrid Retriever: Runs both a vector (semantic) search and a keyword (BM25-style) search in parallel, then fuses their ranked results using Reciprocal Rank Fusion (RRF).
2
.Reranker: Takes the fused candidate set and re-scores each chunk against the original query using a cross-encoder–style evaluation (via ), which is far more accurate than either initial retrieval signal alone.
SemanticRanking
3
.LLM Synthesis: The familiar call, now fed with surgically precise context.
LLMSynthesize
Let's build each component.
Step 0: The CV Knowledge Base
Step 0: The CV Knowledge Base
We begin with the same data model from the original post—a list of structured CV records, chunked into passages. For this demonstration I use my own CV alongside realistic fictional candidates to construct a corpus that exposes the keyword gap clearly.
cvChunks={(*--RoeeZiv(real)--*)"Roee Ziv | Professional Summary: AI Agent Engineer and Researcher \focused on building intelligent agent workflows and LLM applications. \Combines research in model safety with practical engineering to create \reliable AI systems for enterprise use. Skilled in designing complex \conversational agents, defining reasoning logic, and managing the full \agent development process.","Roee Ziv | Forward Deployed Engineer (AI Engineer) at Wonderful \(Dec 2025 - Present). Architects reasoning logic and prompt chains \for complex enterprise voice and chat workflows. Programs custom tools \and backend integrations to enable autonomous task execution. \Transforms business requirements into scalable, shipped AI agent \specifications.","Roee Ziv | AI Lead & Co-Creator at Blabber (Jan 2024 - Aug 2025). \Engineered a stateful LangChain and LangGraph conversational agent \with dynamic transitions and persona adaptation. Designed autonomous \evaluation algorithms to analyze conversation flow and trigger \real-time corrective interventions. Architected the full FastAPI and \MongoDB backend to handle complex agent memory structures.","Roee Ziv | Algorithms Engineer at Pulsenmore (July 2023 - Dec 2025). \Fine-tuned and optimized production-grade ML models to enhance \real-time inference performance and accuracy. Built automated image \processing validation suites using C#, .NET, and Python. Led \algorithmic validation and technical documentation, directly \contributing to the company's successful FDA regulatory clearance.","Roee Ziv | Education: M.Sc. in Computer Science (Meitar Honors \Program), Ben-Gurion University (Aug 2024 - Present). Focus on \Generative AI, LLMs, and Multimodal Audio-Text Integration. \Researching security vulnerabilities and adversarial attacks and \defenses on LLMs (Multimodals). Accepted to PhD program commencing \Spring 2026. B.Sc. in Software Engineering, Ben-Gurion University \(Oct 2021 - 2025), Honors Track.",(*--NoaLevi(fictional)--*)"Noa Levi | ML Lead at SynthAI (2023 - Present). Built a stateful \multi-agent orchestration system with dynamic routing and \context-aware persona switching. Developed evaluation harnesses for \conversational quality using custom LLM-as-judge pipelines.","Noa Levi | Data Scientist at MedFlow (2020 - 2023). Deployed \real-time clinical NLP models for radiology report classification. \Fine-tuned BioBERT for named entity recognition on Hebrew medical \texts.","Noa Levi | Education: M.Sc. in Data Science, Technion (2020). \B.Sc. in Computer Science, Technion (2018). Skills: Python, PyTorch, \LangChain, Kubernetes, FastAPI, PostgreSQL. Certifications: \AWS Solutions Architect (SAA-C03).",(*--DanielKramer(fictional)--*)"Daniel Kramer | Senior Backend Engineer at ScaleOps (2022 - Present). \Architected a microservices platform handling 10M daily API requests \on Kubernetes. Implemented GitOps deployment pipelines with ArgoCD.","Daniel Kramer | Software Engineer at Finova Bank (2019 - 2022). \Built fraud detection pipelines processing 5M+ transactions/day \using Apache Kafka and TensorFlow Serving. Achieved 99.95% uptime \on core banking services.","Daniel Kramer | Education: B.Sc. in Software Engineering, \Tel Aviv University (2019). Skills: Go, Java, Kubernetes, Terraform, \Apache Kafka, TensorFlow, Docker. Certifications: CKA (Certified \Kubernetes Administrator), Google Professional Cloud Architect.",(*--YaelShapiro(fictional)--*)"Yael Shapiro | Research Scientist at Ben-Gurion University \(2023 - Present). Investigating adversarial robustness of multimodal \LLMs. Published 4 papers on jailbreak attacks and alignment techniques \at ACL and EMNLP.","Yael Shapiro | AI Engineer at CyberAI (2021 - 2023). Developed \prompt injection detection systems for enterprise LLM deployments. \Designed red-teaming frameworks for evaluating model safety.","Yael Shapiro | Education: Ph.D. candidate in Computer Science, \Ben-Gurion University (2023 - Present). M.Sc. in Computer Science, \Hebrew University (2021). Skills: Python, PyTorch, HuggingFace \Transformers, LangChain, adversarial ML. Certifications: \DeepLearning.AI Generative AI with LLMs."};
Step 1: Build the Semantic Search Index
Step 1: Build the Semantic Search Index
We use to embed all chunks. This is identical to the approach from the original post.
CreateSemanticSearchIndex
cvIndex=CreateSemanticSearchIndex[cvChunks,"CVHybridIndex"];
We can verify it works:
SemanticSearch[cvIndex,"machine learning experience",3]
This returns chunks about ML work—good. But now consider the query we want to stress-test:
SemanticSearch[cvIndex,"What FDA-related work has Roee Ziv done?",3]
As we will see in the results section, semantic search may return Roee's chunks here—but not because it understood the name. It returns them because the Pulsenmore chunk happens to be the closest semantic match for "FDA." The ranking is fragile: the name "Roee Ziv" contributes no discriminative signal. With a larger corpus, this luck evaporates.
Step 2: The BM25 Keyword Scorer
Step 2: The BM25 Keyword Scorer
BM25 (Best Matching 25) is the gold standard of lexical retrieval. It scores a document by how frequently the query terms appear, with diminishing returns for repeated terms and normalization for document length. The Wolfram Language does not include a built-in BM25, so we implement it from its mathematical definition.
(*Tokenizer:lowercase,splitonnon-wordcharacters*)tokenize[text_String]:=DeleteCases[StringSplit[ToLowerCase[text],RegularExpression["\\W+"]],""];(*InverseDocumentFrequencyforatermacrossacorpus*)idf[term_String,corpus_List]:=Module[{n=Length[corpus],df},df=Count[corpus,chunk_/;StringContainsQ[ToLowerCase[chunk],term]];Log[(n-df+0.5)/(df+0.5)+1.0]];(*FullBM25scoreforasingledocumentagainstaquery*)bm25Score[query_String,doc_String,corpus_List,k1_:1.2,b_:0.75]:=Module[{qTerms,docTokens,avgDL,dl,tf,score=0.0},qTerms=tokenize[query];docTokens=tokenize[doc];dl=Length[docTokens];avgDL=Mean[Length[tokenize[#]]&/@corpus]//N;Do[tf=Count[docTokens,t];score+=idf[t,corpus]*(tf*(k1+1))/(tf+k1*(1-b+b*(dl/avgDL))),{t,DeleteDuplicates[qTerms]}];score];(*RankallchunksbyBM25andreturntop-k*)bm25Search[query_String,corpus_List,k_Integer:5]:=Module[{scored},scored={#,bm25Score[query,#,corpus]}&/@corpus;Take[SortBy[scored,-Last[#]&],UpTo[k]]];
Testing on our target query:
Result: The Pulsenmore chunk (the only one containing both "Roee Ziv" and "FDA") scores 3.71, while every non-Roee chunk scores essentially zero. This is the lexical precision that semantic search fundamentally cannot provide.
Step 3: Hybrid Fusion with Reciprocal Rank Fusion (RRF)
Step 3: Hybrid Fusion with Reciprocal Rank Fusion (RRF)
And the full hybrid retriever:
Hybrid fusion gives us a much better candidate set, but the initial retrieval scores are computed independently for query and document. A reranker evaluates each (query, document) pair jointly, allowing it to capture fine-grained relevance signals that independent scoring misses.
Step 5: The Complete Pipeline
Step 5: The Complete Pipeline
We compose the full upgraded RAG pipeline into a single clean function:
Results: Before vs. After
Results: Before vs. After
Test 1 — "What FDA-related work has Roee Ziv done?"
Test 1 — "What FDA-related work has Roee Ziv done?"
Naive Semantic RAG retrieved Roee's Pulsenmore chunk as its top result—but also pulled in chunks from other candidates whose work is semantically adjacent to regulatory/validation topics. The LLM produced a correct but fragile answer: it happened to find the right chunk, but the retrieval set was contaminated with irrelevant context.
Hybrid RAG retrieved only Roee Ziv chunks (Pulsenmore → Professional Summary → AI Lead), with the FDA-specific Pulsenmore chunk at rank 1. The LLM answer was precise:
"Roee Ziv led algorithmic validation and technical documentation at Pulsenmore, directly contributing to the company's successful FDA regulatory clearance. He also built automated image processing validation suites using C#, .NET, and Python to ensure algorithmic robustness."
Test 2 — "Where has Roee Ziv worked and in what roles?"
Test 2 — "Where has Roee Ziv worked and in what roles?"
Naive Semantic RAG returned a mix: Roee's summary chunk alongside Daniel Kramer's and Noa Levi's role descriptions (all are semantically "person describing their job title at a company").
Hybrid RAG returned three Roee-only chunks: Professional Summary, Pulsenmore, and Wonderful—a clean sweep of his work history.
Test 3 — "Who has experience with adversarial attacks on LLMs?"
Test 3 — "Who has experience with adversarial attacks on LLMs?"
This is a conceptual query with no specific name—the kind where semantic search shines. Both naive and hybrid pipelines correctly surfaced Yael Shapiro's and Roee Ziv's research chunks. The hybrid approach matched the semantic result here, confirming it does not degrade performance on thematic queries.
Diagnostic: Pipeline Stages (FDA Query)
Diagnostic: Pipeline Stages (FDA Query)
The key observation: Semantic search ranked Noa Levi's clinical NLP work and Yael Shapiro's AI safety work in its top 5—these are semantically adjacent to "FDA/regulatory" but belong to the wrong candidates. BM25's name-anchoring eliminated all non-Roee chunks. RRF fusion preserved this precision, and reranking promoted the most topically relevant chunks to the final context window.
Visualizations
Visualizations
Visualization 1: BM25 Score Distribution
Visualization 1: BM25 Score Distribution
This chart reveals the core mechanism: BM25 produces a sharp, discriminative signal on the Pulsenmore chunk (score 3.71), with the remaining Roee chunks scoring 1.6–1.9 (matching on name alone), and all non-Roee chunks scoring effectively zero. The name "Roee Ziv" acts as a hard lexical filter that embedding models simply cannot replicate.
Visualization 2: BM25 vs. Semantic Scores (Side-by-Side)
Visualization 2: BM25 vs. Semantic Scores (Side-by-Side)
BM25 vs Semantic Comparison
This is the most pedagogically revealing visualization. On the left, BM25 shows clear discrimination: five blue bars (Roee's chunks) stand out sharply. On the right, the semantic similarity panel shows a nearly uniform distribution—all chunks receive comparable similarity scores regardless of candidate identity. Semantic search sees "engineering roles at companies" everywhere; it fundamentally cannot distinguish whose role it is looking at. When naive RAG returns the correct chunks, it is winning a tiebreak in a flat ranking—not making a confident retrieval decision.
Visualization 3: Retrieval Funnel
Visualization 3: Retrieval Funnel
Retrieval Funnel
The funnel shows how each stage narrows the candidate set: 14 total chunks → 8 BM25 candidates → 5 after RRF fusion → 3 after reranking. Each stage increases precision, discarding irrelevant context before it reaches the LLM.
Visualization 4: Retrieval Accuracy Across Queries
Visualization 4: Retrieval Accuracy Across Queries
We tested four Roee-specific queries and counted how many of the top-3 retrieved chunks actually belonged to Roee Ziv:
Retrieval Precision: Naive vs Hybrid
The pattern is stark: naive semantic RAG returns two wrong-candidate chunks out of three on every single query. The embedding model consistently confuses candidates who share similar roles and skills. Hybrid search with reranking achieves perfect retrieval precision across all four tests—a 3× improvement.
When to Use What
When to Use What
In practice, hybrid retrieval with reranking is the safe default for any production system. The overhead is modest—one additional scoring pass over a small candidate set—and the accuracy gains on entity-specific queries are dramatic. Critically, as shown in Test 3, hybrid search does not degrade performance on purely conceptual queries where semantic search already excels.
A Note on Corpus Scale
A Note on Corpus Scale
Our demonstration uses a small corpus of 14 chunks across 4 candidates. At this scale, naive semantic RAG can sometimes return correct results through fortunate ranking tiebreaks. This is important to acknowledge honestly.
However, the BM25 vs. Semantic side-by-side visualization reveals why this success is fragile: the semantic score distribution is nearly flat, meaning any small perturbation—a new candidate with similar experience, a slightly rephrased query—can flip the ranking. In a real-world deployment with hundreds of resumes, the embedding space becomes far more crowded, and the probability of correct-by-coincidence retrieval drops sharply.
The hybrid approach converts fragile, coincidental correctness into robust, deterministic precision by anchoring retrieval on exact lexical matches. This is the difference between a demo that works and a system you can trust in production.
Summary and Next Steps
Summary and Next Steps
We started with a simple but fragile semantic RAG pipeline and upgraded it to a production-grade architecture:
◼
BM25 keyword scoring provides lexical precision that embedding models lack — in our tests, it produced a score of 3.71 on the correct chunk versus zero on all wrong-candidate chunks.
◼
Reciprocal Rank Fusion elegantly combines heterogeneous ranking signals without score normalization, boosting retrieval precision from 33% to 100% across all test queries.
The full pipeline remains under 100 lines of core logic—a testament to the Wolfram Language's expressiveness.
Where to go from here:
◼
Scale testing: Expand the corpus to 50+ candidates and measure how the precision gap widens.
◼
Chunking strategies: Experiment with overlapping windows, sentence-level splits, or hierarchical (parent-child) chunking.
◼
Evaluation harness: Build a test set of (query, expected_chunks) pairs and measure retrieval precision/recall systematically.
Building on Idan Arbiv and Gal Fadlon's excellent original walkthrough, we now have a pipeline that handles both conceptual and entity-specific queries with confidence. I hope this gives you a solid foundation for building RAG systems that hold up under real-world query diversity. Please share your experiments—I'd love to see what the community builds on top of this.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Leveling up RAG: implementing hybrid search and re-ranking
by Roee Ziv, and Idan Asis
Wolfram Community, STAFF PICKS, February 6, 2026
https://community.wolfram.com/groups/-/m/t/3635386
by Roee Ziv, and Idan Asis
Wolfram Community, STAFF PICKS, February 6, 2026
https://community.wolfram.com/groups/-/m/t/3635386