Why We Chose Metadata-Filtered Retrieval: A Four-Way Empirical Comparison
As promised in Sprint 3 Week 1, here’s the full breakdown of the retrieval evaluation we ran this week.
The Question
When we designed the RAG pipeline for Granite Lifeline, we chose metadata-filtered retrieval over semantic vector search. The reasoning was simple: the Model Layer already tells us exactly which anomaly type it detected, so an exact metadata match should be more reliable than a fuzzy similarity search. That made sense on paper — but we hadn’t tested it. This week, we ran the experiment.
The Experiment: A 2×2 Design
We compared two variables at once: retrieval method (metadata filter vs. semantic search) and knowledge base chunking strategy (section-level vs. document-level), for four method combinations:
| Method | Retrieval | Chunking | Documents |
|---|---|---|---|
| A | Metadata filter | Section-level | 28 |
| B | Semantic search | Section-level | 28 |
| C | Metadata filter | Document-level | 7 |
| D | Semantic search | Document-level | 7 |
The four methods compared in the experiment.
Both knowledge bases hold the same underlying fault knowledge from grounded_knowledge.yaml — they’re just grouped differently. The section-level collection splits each anomaly type into separate description, causes, and actions documents (28 total across 7 types). The document-level collection merges each anomaly type into one combined document (7 total). Each method was tested across all seven anomaly types, three trials per type.
Results
| Method | Accuracy | Speed |
|---|---|---|
| A — Metadata + Section | 100% | 0.34 ms |
| B — Semantic + Section | 57.1% | 61 ms |
| C — Metadata + Document | 100% | 0.31 ms |
| D — Semantic + Document | 85.7% | 57 ms |
Retrieval accuracy and average latency across all seven anomaly types.
Key Insights
Metadata filter doesn’t care how you chunk. Methods A and C both hit 100% accuracy, whether the knowledge base was split into 28 documents or merged into 7. When you already know exactly what you’re looking for, how the shelf is organised doesn’t matter.
Chunking matters a lot for semantic search. Document-level chunking (D) beat section-level chunking (B) by 28.6 percentage points. Merged documents give the embedding model more surrounding context to work with, which makes similarity matching noticeably more reliable — though still far from perfect.
One anomaly type broke semantic search entirely. electronic_throttle_tracking_fault failed retrieval under both chunking strategies. Its symptom profile turned out to be too semantically similar to other fault types for the embedding model to reliably tell them apart — a real failure mode that metadata filtering simply doesn’t have, because it never has to guess.
Metadata filter was also roughly 180x faster than semantic search across every condition we tested.
Why We Keep Section-Level Chunking
If metadata filter hits 100% either way, why not just use the simpler 7-document collection? Because section-level chunking does something document-level chunking can’t: it lets each prompt layer retrieve only what it needs. Layer 3, which writes the recommended action, retrieves only the risk-level-appropriate actions for the specific risk level detected — not the description, not the causes, and not monitoring guidance meant for other risk levels. A High risk report can’t accidentally include Low risk “just keep an eye on it” language, because that language was never in its context window to begin with. A merged, document-level chunk would put all of that in every retrieval and rely on the model to correctly ignore the irrelevant parts — a harder thing to get right consistently, prompt after prompt.
The Bigger Picture
None of this means semantic search is a bad tool. It’s the right one when a system has to work out what a user means from open-ended natural language, with nothing else to go on. But that’s not our problem: in Granite Lifeline, anomaly_type is already confirmed by the Model Layer before retrieval ever happens. There’s nothing to disambiguate. Metadata filtering isn’t a compromise we settled for — it’s the right tool for the shape of our architecture, and now we have the numbers to back that up.
Full results, including the per-anomaly-type breakdown, are documented in the evaluation folder of the project repository.