writing · oct 2026 · 6 min read

Evaluating a RAG system when access control matters

What FinSight taught me about routing questions to the right store, enforcing permissions at retrieval time, and why leakage is a count, not a percentage.

  • RAG
  • Evaluation
  • FinSight

Most RAG evaluations ask one question: are the answers good? Add user roles and a second question appears that matters more: did anyone see something they shouldn't have? A chatbot that answers brilliantly but occasionally quotes a document the user can't access isn't a good chatbot. It's a data leak with a nice interface.

FinSight is a financial question-answering chatbot I built in December 2025. Users ask questions in plain English, and it answers from two kinds of data: structured financial tables in DuckDB and documents in a Chroma vector store. It runs as a FastAPI service using GPT-4o and Cohere, and serves six user roles that are allowed to see different documents.

  • 120question-answer pairs in the eval
  • 94%faithfulness
  • 90%context recall
  • 0unauthorized document retrievals
FinSight request flow A question and the user's role go to a router. Numeric questions go to DuckDB as SQL. Document questions go to Chroma, filtered to documents the role may see. The retrieved results go to GPT-4o, which writes a grounded answer. The eval measures faithfulness, context recall and unauthorized retrievals. Question+ user role Router DuckDBexact SQL for numbers Chromafiltered by role first GPT-4ogrounded answer structured documents
Permissions are applied before retrieval, so the model never sees a document the user can't.

Route each question to the right store

Not every question belongs in a vector search. "What was Q3 revenue?" is a lookup over structured data; answering it from text chunks invites the model to do arithmetic it isn't good at. So FinSight routes structured queries to DuckDB, where SQL returns exact numbers, and document queries to Chroma, where semantic search finds the right passages.

Routing is a quality decision as much as a performance one. The best way to stop a model from making up a number is to not ask it to compute one.

Enforce permissions at retrieval, not in the prompt

The tempting shortcut is to retrieve everything and tell the model "don't reveal documents this user can't see." That's a request, not a guarantee. If restricted text is in the context window, it can leak.

FinSight enforces document-level access across its six roles at retrieval time: the model only ever receives documents the asking user is allowed to see. There's nothing to leak, however the question is phrased.

Simplified sketch of the pattern (not production code)

def retrieve(question: str, role: str) -> list[Doc]:
    allowed = ROLE_ACCESS[role]                      # groups this role may read
    return collection.query(
        query_texts=[question], n_results=5,
        where={"access_group": {"$in": allowed}},   # filter before the model
    )

def unauthorized(docs: list[Doc], role: str) -> int:
    return sum(doc.access_group not in ROLE_ACCESS[role] for doc in docs)

assert sum(unauthorized(r.docs, r.role) for r in eval_runs) == 0    # a hard gate, not an average

Measure quality and leakage separately

I evaluated the system on 120 question-answer pairs and tracked three numbers:

  • Faithfulness: 94%. Is the answer supported by the retrieved context, rather than made up?
  • Context recall: 90%. Did retrieval find the information needed to answer?
  • Unauthorized retrievals: 0. For every question, did any retrieved document fall outside the asking role's permissions?

The first two are percentages because they're quality measures: 94% faithfulness is good, and you keep improving it. The third is a count with a hard requirement of zero. A system that leaks 1% of the time isn't "99% secure"; it leaks.

Report security results as counts with a pass/fail bar, not as averages that can hide a single bad answer.

What I'd tell someone building one

  • Write your eval before you tune anything. 120 well-chosen questions told me more than any amount of trying it by hand.
  • Cover every role. Ask the same question as different users, since the right answer depends on who's asking.
  • Include questions that should be refused. A role asking about documents it can't see is the most important test you have.
  • Keep structured data structured. Don't turn your tables into text chunks just because you're building a RAG system.

I used the same approach for this site's assistant: Guru has its own 52-question eval with separate categories for facts, details it shares only when asked, unknowns, off-topic requests and prompt injection.