Enterprise RAG
2026-09-26 · 7 min read
Key takeaways
- Evaluate retrieval and generation separately.
- Test with real questions from the roles who will use the system.
- Permission leakage is a release blocker, not a quality metric.
ERP, PLM, CRM and documents keep provenance and version.
Why RAG evaluation needs its own method
A fluent answer can still be wrong, uncited or based on a document the user should not see. Evaluation therefore has to follow the whole chain: access, retrieval, generation and outcome.
Build a test set of real questions from buyers, planners, store managers and compliance, with the expected sources for each.
The 10-point checklist
1. Retrieval recall: are the expected documents in the top results? 2. Retrieval precision: are irrelevant sources kept out? 3. Faithfulness: is every claim supported by retrieved evidence? 4. Citation completeness: does each answer name its sources?
5. Permission compliance: no evidence outside the user's scope. 6. Freshness: superseded versions are excluded. 7. Refusal quality: the system says when evidence is missing. 8. Latency within the role's workflow. 9. Cost per answer at expected volume. 10. Operational outcome: did the approved action produce the intended result?
How to run the evaluation
Automate points 1–7 with a fixed test set and rerun it after every index, prompt or model change.
Review a sample manually with domain owners, and track points 8–10 in production dashboards.
Release criteria
Define thresholds before testing. Any permission leak blocks release; faithfulness and citation targets should be agreed with the process owner.
Keep human approval on actions above threshold while evidence accumulates.
FAQ
Frequently asked questions
What metrics should be used to evaluate enterprise RAG?
Retrieval recall and precision, faithfulness, citation completeness, permission compliance, freshness, refusal quality, latency, cost and operational outcome.
How big should a RAG test set be?
Start with a few dozen real questions per role and process, with expected sources, and grow it as new failure cases appear.
How often should RAG be re-evaluated?
After every change to the index, prompts, retrieval settings or model, and periodically on production samples.
Is a demo enough to validate enterprise RAG?
No. A demo shows capability; only evaluation on the organisation's own sources and permissions shows readiness.