Rethinking agentic RAG: what if retrieval didn't start with embeddings?

Nearly every RAG walkthrough, reference architecture and production write-up starts the same way: chunk the corpus, embed the chunks, push them into a vector store, embed the question at query time, take the top k by cosine similarity. That sequence is so standard it stopped looking like a design decision. It's just what step one is.

A paper from May 2026 — Rethinking Agentic RAG: Toward LLM-Driven Logical Retrieval Beyond Embeddings — argues that step one is load-bearing in a way we don't examine, and that if your LLM is already strong enough to run a multi-turn retrieval loop, you can delete the embedding layer entirely and hand it a boolean query language over a plain inverted index instead. Their system, LogicalRAG, matches a dense+sparse hybrid baseline on accuracy while building the index 41× faster.

This is my read of it, plus a sketch of the smallest implementation the idea reduces to — written while reading, to force myself to be concrete about what the architecture actually requires.

Why this one caught my attention

Because the RAG failure I keep hitting isn't a ranking failure. It's that semantic similarity and actual information need are two different things, and the gap between them is where the complexity budget goes.

Ask "which of our enterprise customers churned before the pricing change?" and a vector search happily returns passages about enterprise customers, about churn, about pricing changes. All three topics are present. The conjunction — and especially the negation, and especially the ordering — is not something a single 1536-dimensional point can carry. Cosine similarity has no operator for "and also not this."

So the standard fix is to add layers. Hybrid search, because BM25 catches the exact terms dense retrieval smears. A reranker, because the fused top-50 is noisy. A knowledge graph, because multi-hop questions need structure the embeddings threw away. Query decomposition on top, because one question is really three. Each layer is defensible on its own, and the stack as a whole becomes something you can't debug — when an answer is wrong, the culprit could be the chunker, the embedding model, the fusion weights, the reranker, or the prompt.

The uncomfortable part: every one of those layers exists to recover intent that the embedding step discarded. We compress a precise question into a lossy vector, then spend the rest of the pipeline trying to reconstruct what we lost.

What the paper actually proposes

Invert the division of labour. Don't make the retrieval system guess at intent — make the LLM state it, in a language the retrieval system can execute exactly.

The model doesn't emit a string to be embedded. It emits a logical expression: boolean operators with grouping, quoted phrases for exact matching, field targeting, and term boosting.

AND / OR / NOT          boolean composition, parenthesised
"exact phrase"          phrase match, order preserved
title:Vivaldi           field-scoped constraint
opera^2                 term boost
default_operator=OR     tunable per query

The backend underneath is deliberately boring: a standard inverted index — OpenSearch in their implementation — with BM25 ranking at the default parameters (k₁=1.2, b=0.75). The logical expression decides which documents are candidates at all; BM25 only orders the ones that already satisfy the constraints.

That split is the whole argument. Intent is executed faithfully, and ranking becomes a separate, smaller problem. When retrieval goes wrong you can look at the expression and see why, which is not a thing you can do with a vector.

Traditional agentic RAG vs. this

AGENTIC HYBRID (baseline)

  question
     |
     v
  LLM rewrites query
     |
     +--> embed --> ANN search --+
     |                           +--> RRF fuse --> rerank --> top-k
     +--> BM25 ------------------+                             |
                                                               v
                              LLM reads passages, maybe loops, answers


LOGICALRAG

  question
     |
     v
  LLM writes logical expression  <-------------------------+
     |     (title:X AND "exact phrase") NOT Y              |
     v                                                     |
  inverted index: constraint filter --> BM25 rank          |
     |                                                     |
     +--> passages + hit count ----------------------------+
     |
     v
   0 hits    ->  relax a constraint
   too many  ->  add one
   enough    ->  answer, or abstain if evidence absent

Same agentic shape — multiple turns, refinement based on intermediate results. What changes is that the refinement signal is now legible. "Zero documents matched this conjunction" is information. "The top-5 cosine neighbours were 0.71, 0.70, 0.69" is not: you can't tell from it whether the evidence is missing or just worded differently.

The loop, step by step

1. Plan. The model states what evidence would answer the question — not the query yet, the evidence requirement.
2. Formulate. That requirement becomes a logical expression. Entities become field-scoped terms; distinguishing details become quoted phrases; distractors it wants excluded become NOT clauses.
3. Execute. The index returns ranked passages from the candidate set that satisfies the constraints — plus a hit count.
4. Observe. The model reads what came back and checks it against the evidence requirement from step 1.
5. Refine or answer. Zero hits triggers constraint relaxation. A flood of weakly-related hits triggers a tighter conjunction. Enough evidence ends the loop. The paper caps it at eight turns.

Step 5 has a fourth exit that I think is the most interesting result in the paper: abstain. If repeated, progressively relaxed expressions keep coming back empty, that's a far stronger signal that the corpus simply doesn't contain the answer than anything you get from dense retrieval — where every query returns its top k no matter what, with a similarity score that looks about the same whether the evidence exists or not.

A concrete comparison

Take a question with a conjunction and a negation in it. Here's what each path actually does:

Q: "Which Vivaldi operas had an Italian libretto but were
    never staged in Venice?"

embed(Q) --> [0.013, -0.228, 0.091, ... ]   (1536 dims)

ANN search, top 5 by cosine:
  0.84  "Vivaldi's operas premiered largely in Venice..."     <-- contradicts
  0.82  "The libretto tradition of Italian baroque opera..."  <-- topical only
  0.81  "Vivaldi composed over 40 operas, many for Venice"    <-- contradicts
  0.79  "Venetian opera houses of the 1720s..."               <-- topical only
  0.78  "Handel's Italian-libretto works in London..."        <-- wrong composer

The NOT is invisible to cosine similarity. "Never staged in
Venice" and "staged in Venice" sit near each other in the
embedding space — the negation is a word, not a direction.

Result: 3 of 5 passages argue the opposite of the question.
The model either hedges, or confidently gets it wrong.

Q: "Which Vivaldi operas had an Italian libretto but were
    never staged in Venice?"

turn 1 --> (title:Vivaldi AND opera AND "Italian libretto")
               NOT Venice
           hits: 0
           -> constraint too tight; "Italian libretto" is my
              phrasing, not necessarily the corpus's

turn 2 --> (title:Vivaldi AND opera AND (libretto OR librettist))
               NOT (Venice OR Venetian)
           hits: 6
           -> every passage is about a Vivaldi opera, mentions
              its libretto, and does not mention Venice

turn 3 --> answer from those 6, citing them
           or: had turn 2 also returned 0, abstain —
           three failed relaxations is evidence of absence,
           not evidence of bad phrasing

Note what the hit count did in turn 1. 0 was not a failure, it was a measurement: it told the model its own phrasing was the problem, and which clause to loosen. That diagnostic doesn't exist in the vector path.

What the numbers say

All figures below are the paper's, not mine. They evaluate at two scales: medium (HotpotQA ~9K passages, 2WikiMultiHopQA ~6K, MuSiQue ~11K, 1,000 questions each) and large (the KILT Wikipedia corpus). Baselines are Direct Answer with no retrieval, Agentic Hybrid (the same loop but with dense+sparse retrieval fused by RRF), HippoRAG2 (graph-based, medium scale only), and MA-RAG (multi-agent, embedding-based).

On KILT-scale accuracy, judged by LLM-as-judge:

Parity. Pure lexical matching, no embedding model anywhere in the system, same accuracy as the hybrid stack. The accuracy result is the uninteresting half of the finding — it's what makes the other half admissible.

The other half, measured at KILT scale with 16 concurrent clients:

41× on build time, 2.3× on throughput, 3.1× on latency. The build-time gap is the one with real operational consequences: 52 hours of GPU-bound embedding isn't just a cost line, it's the reason chunking strategy becomes a quarterly decision instead of an afternoon experiment.

And the abstention result, from the answer-unavailable evaluation (400 examples per dataset with the gold passages deliberately removed):

Hallucinations roughly a third lower when the answer isn't in the corpus. The paper attributes this to exactly the mechanism above: repeated failure under explicit lexical constraints is a clear signal that the evidence is absent, and the model acts on it.

Where I think this holds, and where it doesn't

It holds when the question's precision lives in its terms. Entities, identifiers, dates, error codes, version numbers, API names, legal clause references, SKUs. Internal documentation search. Log and incident retrieval. Anything where the user already knows the vocabulary of the corpus — which, inside a company's own documents, they usually do. An inverted index is better than an embedding at ERR_CONN_RESET or v2.14.3, not merely cheaper; those tokens are near-noise to a semantic model.

It doesn't hold when the user's words and the corpus's words don't overlap at all. "Why do our users give up partway through?" against a corpus that only ever says "funnel abandonment" and "drop-off" is a vocabulary mismatch, and no boolean expression the model writes can bridge a gap it can't see. The paper is honest about this: embeddings remain better for highly abstract semantic matching and fuzzy association.

Which points at the actual design conclusion, and it isn't "embeddings are over." It's that the default is wrong. Starting with a vector store and bolting BM25 on when precision fails is backwards; starting with an inverted index and adding dense retrieval only for the queries that demonstrably need it gives you a system you can reason about, where the expensive layer has to justify itself. The paper's own stated limitation is the real constraint: the whole thing rests on the LLM's ability to formulate accurate logical queries. That ability is what changed recently — and it's why this is worth revisiting now rather than being a step backwards to 2010.

The smallest version the idea reduces to

The fastest way to tell whether an architecture is genuinely simpler is to write it out and see how much code it takes. So: Python, FastAPI, and no OpenSearch — a hand-rolled inverted index, so there's nothing in the retrieval path that can't be inspected. What follows is the core of it; the fuller version is going into a public repo, and there's a note at the end on what's still to come.

Three pieces: an index that supports boolean expressions, an LLM that writes them, and a loop that uses the hit count as feedback.

1. The index

A posting list per token, plus a doc-length table for BM25. The constraint evaluator returns a set of doc ids; ranking happens only inside that set.

from collections import defaultdict
from dataclasses import dataclass, field
import math, re

TOKEN = re.compile(r"[a-z0-9_.]+")

def tokenize(text: str) -> list[str]:
    return TOKEN.findall(text.lower())

@dataclass
class Index:
    docs: dict[int, str] = field(default_factory=dict)
    titles: dict[int, str] = field(default_factory=dict)
    postings: dict[str, dict[int, int]] = field(default_factory=lambda: defaultdict(dict))
    title_postings: dict[str, set[int]] = field(default_factory=lambda: defaultdict(set))
    lengths: dict[int, int] = field(default_factory=dict)

    def add(self, doc_id: int, title: str, body: str) -> None:
        self.docs[doc_id] = body
        self.titles[doc_id] = title
        toks = tokenize(body)
        self.lengths[doc_id] = len(toks)
        for t in toks:
            self.postings[t][doc_id] = self.postings[t].get(doc_id, 0) + 1
        for t in tokenize(title):
            self.title_postings[t].add(doc_id)

    @property
    def avgdl(self) -> float:
        return sum(self.lengths.values()) / max(len(self.lengths), 1)

    def match_term(self, term: str) -> set[int]:
        return set(self.postings.get(term, {}))

    def match_field(self, fieldname: str, term: str) -> set[int]:
        if fieldname == "title":
            return set(self.title_postings.get(term, set()))
        return self.match_term(term)

    def match_phrase(self, phrase: str) -> set[int]:
        # candidates must contain every token; then verify adjacency
        toks = tokenize(phrase)
        if not toks:
            return set()
        cands = self.match_term(toks[0])
        for t in toks[1:]:
            cands &= self.match_term(t)
        needle = " ".join(toks)
        return {d for d in cands if needle in " ".join(tokenize(self.docs[d]))}

    def bm25(self, query_terms: list[str], candidates: set[int],
             k1: float = 1.2, b: float = 0.75) -> list[tuple[int, float]]:
        N, avgdl = len(self.docs), self.avgdl
        scored: list[tuple[int, float]] = []
        for doc_id in candidates:
            score, dl = 0.0, self.lengths[doc_id]
            for term in query_terms:
                tf = self.postings.get(term, {}).get(doc_id, 0)
                if not tf:
                    continue
                df = len(self.postings.get(term, {}))
                idf = math.log(1 + (N - df + 0.5) / (df + 0.5))
                score += idf * (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl / avgdl))
            scored.append((doc_id, score))
        return sorted(scored, key=lambda p: p[1], reverse=True)

Note match_phrase: intersect posting lists to get candidates, then verify adjacency only on those. Cheap, and it's why an exact phrase constraint is genuinely exact rather than a bag of words that happen to co-occur.

2. Parsing the expression

A recursive-descent parser over the operator set, compiling to set operations. NOT is a difference against the full doc set — which is the operation a vector store structurally cannot perform.

import re as _re

TOKEN_RE = _re.compile(r'\s*(\(|\)|AND|OR|NOT|"[^"]*"|[A-Za-z0-9_.:^]+)')

def parse(expr: str, index: Index, default_or: bool = True) -> tuple[set[int], list[str]]:
    """Returns (matching doc ids, terms used for ranking)."""
    tokens = TOKEN_RE.findall(expr)
    pos = 0
    terms: list[str] = []
    all_docs = set(index.docs)

    def peek() -> str | None:
        return tokens[pos] if pos < len(tokens) else None

    def atom() -> set[int]:
        nonlocal pos
        tok = tokens[pos]; pos += 1
        if tok == "(":
            inner = expression()
            if peek() == ")":
                pos += 1
            return inner
        if tok == "NOT":
            return all_docs - atom()
        if tok.startswith('"'):
            phrase = tok.strip('"')
            terms.extend(tokenize(phrase))
            return index.match_phrase(phrase)
        tok = tok.split("^")[0]           # boosts are a ranking hint, ignored here
        if ":" in tok:
            fieldname, _, raw = tok.partition(":")
            parts = tokenize(raw)
            term = parts[0] if parts else ""
            terms.append(term)
            return index.match_field(fieldname, term)
        parts = tokenize(tok)
        term = parts[0] if parts else ""
        terms.append(term)
        return index.match_term(term)

    def expression() -> set[int]:
        nonlocal pos
        acc = atom()
        while (nxt := peek()) and nxt != ")":
            if nxt in ("AND", "OR"):
                op = nxt; pos += 1
                rhs = atom()
                acc = acc & rhs if op == "AND" else acc | rhs
            else:                         # implicit operator between bare terms
                rhs = atom()
                acc = acc | rhs if default_or else acc & rhs
        return acc

    return (expression() if tokens else set()), terms

3. The agent loop, behind FastAPI

The model gets the operator grammar in its system prompt and the hit count back after every turn. That's the entire control signal.

from fastapi import FastAPI
from pydantic import BaseModel
import anthropic, json

app = FastAPI()
client = anthropic.Anthropic()
index = Index()           # populated at startup
MAX_TURNS = 8

SYSTEM = """You retrieve evidence by writing logical query expressions.

Grammar:
  AND OR NOT      boolean, parenthesisable
  "exact phrase"  phrase match
  title:term      field-scoped
  term^2          boost

After each query you are told the hit count.
  0 hits          -> your phrasing is too tight. Relax ONE clause
                     (widen a phrase to OR'd synonyms, drop a NOT).
  many weak hits  -> add a constraint.
  good evidence   -> answer, citing doc ids.
If three progressive relaxations all return 0 hits, say the corpus
does not contain the answer. Do not answer from prior knowledge.

Reply with JSON only:
  {"action":"search","expr":"..."}
  {"action":"answer","text":"...","cited":[ids]}
  {"action":"abstain","reason":"..."}"""

class Ask(BaseModel):
    question: str
    top_k: int = 5

@app.post("/ask")
def ask(req: Ask):
    messages = [{"role": "user", "content": req.question}]
    trace = []

    for _ in range(MAX_TURNS):
        reply = client.messages.create(
            model="claude-sonnet-5-5",
            max_tokens=1024,
            system=SYSTEM,
            messages=messages,
        )
        raw = reply.content[0].text
        messages.append({"role": "assistant", "content": raw})
        step = json.loads(raw)

        if step["action"] == "answer":
            return {"answer": step["text"], "cited": step["cited"], "trace": trace}
        if step["action"] == "abstain":
            return {"answer": None, "reason": step["reason"], "trace": trace}

        doc_ids, terms = parse(step["expr"], index)
        ranked = index.bm25(terms, doc_ids)[: req.top_k]
        trace.append({"expr": step["expr"], "hits": len(doc_ids)})

        passages = "\n\n".join(
            f"[{d}] {index.titles[d]}: {index.docs[d][:600]}" for d, _ in ranked
        )
        messages.append({
            "role": "user",
            "content": f"hits: {len(doc_ids)}\n\n{passages or '(no documents matched)'}",
        })

    return {"answer": None, "reason": "turn limit reached", "trace": trace}

The trace is the part that makes the argument. Every response carries the exact expressions that produced it and how many documents each one matched, so a wrong answer can be debugged by reading the retrieval reasoning as text — no embedding inspection, no fusion weights to second-guess. That property is the real case for this architecture, more than any of the benchmark numbers.

What breaks in production

A demo that works on a clean corpus with a cooperative model is not a system. The gaps I'd have to close before putting this anywhere real:

• Malformed expressions. Unbalanced parens, invented operators, a field name that doesn't exist. The parser above throws. Production needs it to return a structured parse error back to the model as another feedback turn — a bad expression is recoverable information, not a 500.
• Latency is now LLM-bound. Retrieval dropped to ~75 ms in the paper; a 3-turn loop is three sequential model calls. The index is no longer the bottleneck, the reasoning is. Cache the expression per normalised question — the same question shouldn't pay for formulation twice.
• Degenerate queries. A single common term as the whole expression matches half the corpus and BM25 then ranks noise. Worth rejecting expressions whose hit count exceeds some fraction of the index and asking for a tighter one.
• No fallback. The vocabulary-mismatch case is real, so after N empty relaxations the honest move is a dense-retrieval fallback — if one exists — before abstaining. Which means the embedding layer doesn't vanish; it becomes the escape hatch instead of the front door.
• Observability. The things I'd actually instrument: hit count per turn, turns per question, and abstention rate. If turns-per-question drifts upward, the corpus vocabulary and the users' vocabulary are diverging — and that's measurable here in a way it isn't in a vector pipeline.
• Stemming and synonyms. My tokenize is exact-match, so libretto and librettos are different tokens. An analyser with stemming and a synonym map does most of the semantic work people reach for embeddings to get — unglamorous, and it moves the needle more than a better embedding model would.

What I took from it

Not "embeddings are dead" — the paper doesn't claim that, and its limitations section says the opposite. What I took is narrower and more useful: retrieval architecture doesn't have to grow more complex every time reasoning gets better. It can get simpler, because capability moved up the stack and the layers that existed to compensate for a weak orchestrator are now compensating for nothing.

The default I'm changing: when I start the next RAG system, the first thing I build is a plain inverted index with a good analyser and an honest boolean query path. Then I measure. Then the embedding layer has to earn its place by fixing queries I can show it failing on. That's the opposite of how I've done it every previous time, and I can't find the argument for the old order anymore.

Reference

• Yuqi Zeng, Qixiang Deng, Yulei Wan, Ruiquan Jiang, Xiaoqing Zheng, Xuanjing Huang. Rethinking Agentic RAG: Toward LLM-Driven Logical Retrieval Beyond Embeddings. arXiv:2605.27123, 26 May 2026. arxiv.org/abs/2605.27123 ↗

Every accuracy, latency, index-construction, refusal and hallucination figure quoted above is from the paper, measured by its authors — none of them are mine. The Python and FastAPI code is my own reconstruction of the architecture, not the authors' implementation. I'm building it out into a full project — demo corpus, tests, and a reproducible side-by-side against a vector baseline — and I'll make the repo public and link it here, along with whatever numbers I measure myself.