Organization-Level RAG over Data-Engineering Knowledge

Studio Knowledge Base is a Databricks-native RAG system deployed at organization level over internal data-engineering documents: field-mapping workbooks, data dictionaries, and design/spec documents. Files are uploaded to a Unity Catalog Volume corpus where the folder path is a contract — it assigns client IDs, document kind (prose / mapping / dictionary), and sharing class, and doc_kind doubles as a processing instruction (only dictionary documents feed the structured dictionary table). A Lakebase (managed Postgres) registry tracks each upload through a Uploaded → Processing → Completed | Failed | Missing-file lifecycle; because only serverless compute reaches Lakebase, the pipeline job brackets its classic-compute stages with serverless status tasks.

The ingestion run is deliberately restricted: it parses exactly the selected files (selection is an instruction — selected files parse unconditionally, even if byte-identical), leaves untouched documents in place via per-document DELETE+APPEND on Delta (never full-table rewrites), withdraws rows for files deleted from the Volume, and refuses cross-run duplicate content (same hash at a new path is quarantined while the incumbent keeps its identity). The Databricks job runs git-sourced notebooks as a task graph: parse documents (docling → typed elements and table cells) and spreadsheets (per-sheet archetype classification) → normalize into a canonical schema (source→silver→gold field mappings, a core+ext data dictionary, canonical sections, and precomputed rollup summaries) → parent/child chunking with content-derived chunk IDs and a provenance prefix on every chunk.

Before anything reaches answers, a checks stage enforces ~27 hard gates and measured baselines — parse fidelity, furniture leaks, chunk size/uniqueness/reproducibility, sheet coverage, mapping-chunk contracts, PHI/identifier patterns, and landing/quarantine reconciliation — so any unexpected gate failure fails the run before the vector index syncs. A vocabulary stage extracts ~18k corpus terms with size-independent rarity and bare forms of qualified names, then the sync stage triggers the Vector Search Delta Sync and waits. The index (chunks_index) is a Databricks Vector Search Delta Sync index over the gold chunks with change-data-feed, using Databricks-managed BGE-large embeddings (1024 dims) applied to both chunk text at sync and query text at query time.

Questions are answered by three composed paths. Cheap deterministic intent routing classifies lookup / aggregate / retrieval and extracts identifiers — aggregate questions are never answered from top-k because counting from a handful of chunks produces confident wrong answers. The exact path resolves prose names to identifiers via the vocabulary and runs plain parameterized SQL over the field-mapping and dictionary tables. The semantic path always runs one SQL statement on a serverless warehouse: hybrid vector search → LEFT JOIN the parent section for surrounding context → ai_query drafts an evidence-only answer with per-claim citations (title + page), emitting a NOT_IN_CORPUS sentinel when nothing supports an answer. Two front ends consume it: a Databricks App (Streamlit) and a Genie space exposing UC functions (ask_knowledge runs all three paths in one call). Deterministic identity — same content yields the same chunk IDs — keeps citations stable and re-embedding minimal, and no PHI or content ever appears in logs or quarantine reasons.