Enterprise Knowledge-Base Q&A, Production-Grade: From RAGFlow Deployment to Retrieval Quality

Prerequisite: This article continues the Kubernetes + RAGFlow Quick-Deploy Guide — deployment makes "the system run"; this one makes it "answer correctly, stay usable." In enterprise knowledge-base Q&A, 80% of failures happen in retrieval quality, not model capability.

1. Scenario and Pain Points

The classic RAG (Retrieval-Augmented Generation) chain:

ingest documents ──> retrieve (vector/keyword) ──> assemble prompt ──> LLM generate ──> cite sources

Real production pain points, ranked:

Pain point	Symptom	Root cause
Bad retrieval	irrelevant answers, missing key documents	chunking breaks semantics, embedding domain drift
Hallucination	cites things that don't exist / wrong attribution	retrieval returns wrong hits but LLM answers confidently
Stale updates	answer unchanged after docs change	missing ingest/index update pipeline
Not evaluable	"feels okay" but unquantifiable	no evaluation set and metrics

2. Full-Stack Architecture

Production-grade RAG has four more stages than a demo:

┌────────────────────────────────────────────────────────────┐
│  ① Document processing pipeline                               │
│  multi-format parsing (PDF/Word/tables) → cleaning →           │
│  structural chunking → metadata                               │
├────────────────────────────────────────────────────────────┤
│  ② Index layer (hybrid retrieval)                             │
│  vector index (semantic) + keyword index (BM25/full-text) →    │
│  fusion ranking                                              │
├────────────────────────────────────────────────────────────┤
│  ③ Rerank layer                                               │
│  coarse recall Top50 → cross-encoder rerank Top5 → prompt      │
│  assembly                                                    │
├────────────────────────────────────────────────────────────┤
│  ④ Quality & operations                                       │
│  eval set → offline metrics → online feedback loop →           │
│  update/permissions/observability                             │
└────────────────────────────────────────────────────────────┘

3. Key Implementation

3.1 Document Processing: Chunking Is the First Determinant of Retrieval Quality

Strategy	Approach	Use when
Fixed length	split by token	fallback
Structure-aware	split at heading/paragraph/table boundaries	manuals, policy docs (recommended)
Parent-child chunks	small chunks retrieve + parent chunk as context	long documents, answers spanning sections
Semantic splitting	use embedding similarity to find breaks	mixed-style corpora

Engineering notes:

• Metadata must carry: source, section, page, update time — source citation depends on it;
• Handle tables separately: convert tables to Markdown before ingest; direct vectorization works poorly;
• An overlap window (50–100 tokens) significantly reduces semantics broken at boundaries.

3.2 Hybrid Retrieval: Vectors Are Not Everything

Pure vector retrieval often fails on "proper nouns, model numbers, exact figures" (semantically similar but literally different). The production standard:

Score = α × vector similarity + β × BM25 relevance (α+β=1, tunable)

Scenario	vector weight α	BM25 weight β
general Q&A, many paraphrases	0.7–0.8	0.2–0.3
technical docs, model/code/spec queries	0.4–0.5	0.5–0.6

3.3 Rerank: The Highest ROI Layer

Only a handful of the coarse Top-50 are truly relevant. Use a cross-encoder (e.g., bge-reranker) to rank the 50 one by one, take Top-5 into the prompt:

• Effect: retrieval precision (Recall@5 hit rate) typically improves 15–30 points;
• Cost: 50 rerank calls at tens per second ≈ +100 ms per query;
• Necessity: add it whenever the eval data allows — the highest-ROI optimization in RAG.

3.4 Evaluation System: Make "Answering Correctly" Quantifiable

Layer	Metric	Approach
Retrieval	Recall@k / MRR	build a "question → expected documents" eval set (50–200 items)
Generation	citation accuracy, hallucination rate	sample human labeling + LLM-assisted scoring
Business	adoption rate, follow-up rate	online log instrumentation

Engineering notes: the eval set is a lasting asset — build a 50-item seed set before launch, expand it by refluxing "user-reported-wrong" queries during operation, and run the same eval set for every chunk-policy change / embedding swap / rerank tuning, deciding with numbers rather than feel.

4. Four Productionization Concerns

1. Update mechanism: document changes → trigger incremental parse/index (watch file sources or periodic scan), keep old versions archived for rollback;
2. Permission isolation: isolate knowledge bases by department/classification; both the vector store and retrieval API must pass the permission layer — otherwise the index itself becomes a data-leak surface;
3. Observability: log every query (query, recalled Top-5 docs, rerank scores, generation time) so abnormal queries are replayable;
4. Cost control: compute embeddings offline in batch; online only retrieval + rerank + generation; cache retrieval results for high-frequency queries.

5. Pitfalls and Tuning (Field Experience)

1. Embedding domain drift is the No. 1 killer: general embeddings understand industry jargon poorly (e.g., "10K-GPU cluster", "RoCE"). Mitigation: benchmark with domain corpus first; fine-tune or swap embeddings when needed — quantify drift with the eval set, don't guess.

2. "Not retrieved" — check chunking before blaming embeddings: many "can't answer" cases are actually information broken apart by chunking. Troubleshooting order: chunk boundaries → metadata → hybrid weights → embedding.

3. The engineering defense against hallucination is source citation: enforce "answer must attach sources; refuse to answer when no source" — far more reliable than asking the model to "be more careful." The rule layer guarantees determinism; the model layer covers open-endedness.

4. RAG is not deploy-once, use-forever: documents change, business changes, embedding models upgrade — treat "eval set + feedback reflux + periodic re-benchmarking" as part of operations, not a project wrap-up.

6. Quantified Results and Retrospective

Metric	Baseline (vector-only Top5)	Optimized (hybrid + rerank)
Recall@5 (seed eval set)	~65%	~85%+
Citation accuracy	low (wrong sections often cited)	high (metadata + citation)
Average response latency	~0.5 s	~0.7 s (+100–200 ms rerank, acceptable)

Note: figures are engineering-order-of-magnitude estimates for setting expectations; actual gains follow your own eval set.

Takeaway: retrieval-quality optimization order = chunk strategy → hybrid retrieval → rerank → embedding upgrade; move to the next layer only when the current one has eval numbers to back it. The four productionization concerns (update/permissions/observability/cost) decide whether the system survives long-term.

Engineering Takeaways

• Knowledge-base Q&A is the most mature, most replicable "AI-compute-side application" — from deployment (RAGFlow) to quality (this article) to Agent-ization (planned), it can grow into a complete engineering capability showcase;
• "Rule layer guarantees determinism, model layer covers open-endedness" is the universal craft principle for AI application engineering — it recurs in the RAGFlow deploy guide and future Agent workflows;
• Evaluation-driven iteration (build the eval set before optimizing) deserves promotion across all AI applications — the application-side version of the "reproducibility" principle emphasized by the AI-HPC platform.