A platform engineer at a regional bank once described the problem like this: their institution held more than two million policy documents, regulatory bulletins, and due-diligence reports spread across shared drives, email attachments, and a few outsourced systems. The goal sounded mundane — let employees ask a question in plain language and get back an answer with its source attached. Three constraints killed every off-the-shelf idea they tried. The data could not leave the intranet. The document formats were a sprawl of scans, PDFs, and office files with no consistent structure. And the single hardest requirement: a confident-but-wrong model answer is unacceptable when every sentence has to trace back to a cited document.
If you build infrastructure for a bank, insurer, or regulated manufacturer, you have probably lived some version of this. The interesting part is that the bottleneck is rarely the model. It is everything that happens to a document before and around it — compression, preprocessing, and the economics of tokens.
The cost nobody budgets for: finding things
Knowledge workers lose a surprising amount of time to retrieval that is not even model-assisted. An IDC study covering 1,200 information workers found employees spend an average of 4.5 hours each week simply locating documents. In a financial setting that figure tends to be worse, not better. More policies, more versions, more cross-department searches, and a meaningful slice of every week disappears into confirming which version is the current, valid one. A knowledge base that solves only the "ask the model" step but ignores the "find the document" step does not move the needle.
Why public clouds are structurally awkward here
The sensitivity of the data is the real wall. IBM's 2025 Cost of a Data Breach report found that among stolen data, customer PII made up 53 percent, employee PII 37 percent, and intellectual property 33 percent. For intellectual property specifically, the cost per lost record can reach 178 dollars. Handing batches of contracts, client files, and research reports to a public-cloud model is, for a regulated institution, roughly equivalent to leaving the vault door open.
And the volume is only climbing. Komprise's 2026 report states that 74 percent of large enterprises now store more than 5 PB of unstructured data, 40 percent exceed 10 PB, and 85 percent of IT leaders expect storage spending to keep rising through 2026. Archives, imaging, and logs are precisely the kind of data that occupies enormous space while being queried rarely — the exact profile that punishes naive ingestion.
Taken together, these two facts define the design space. The data is too sensitive to export, and too large to process carelessly.
Compression is a prerequisite, not a cleanup step
A common misconception is that retrieval-augmented generation means dropping documents into a vector store and asking questions. In practice, raw documents are large, noisy, and full of boilerplate — headers, footers, repeated legal text, low-information scans. Feed them in unprocessed and two things happen at once: token cost balloons on every query, and retrieval noise rises because the vector space gets cluttered with near-duplicate, low-signal chunks. The model answers from a messier context, and the bill climbs in proportion to the mess.
The principle that changes the math is simple: compress and normalize before ingestion, not after. This is not "make the file smaller so it uploads faster." It is about reducing the token surface of each document so that every downstream query is cheaper and every retrieved chunk is more signal-dense.
A few concrete behaviors fall out of this principle:
- Boilerplate removal. Stripping repeated headers, footers, and legal boilerplate shrinks context without removing meaning.
- Layout normalization. A scanned table and a native table should land in the same representation, or retrieval will treat them as unrelated.
- Deduplication. Near-identical policy versions should not each consume retrieval budget.
- Selective fidelity. Some regions — seals, signatures, specific layout cues — carry legal meaning and must be preserved even as the surrounding noise is removed.
Public benchmarks from systems built around this principle are instructive. A 443-page OFD document can be processed end to end in roughly 124 seconds. Image assets compressed before being sent to GPT-4o cut token cost by about 85.9 percent. Document volume typically shrinks 60 to 80 percent. The payoff is compounding: the same retrieval budget now covers more material, the same answer draws on cleaner context, and the savings re-apply on every single question asked.
Token-cost economics, made explicit
The reason compression pays off disproportionately is that ingestion is a one-time cost while querying is repeated forever. Spend compute once to shrink and clean a document, and every future retrieval, every future answer, every future user session reaps the benefit. Ignore it, and you pay the full, noisy token tax on each of millions of queries.
For a regulated institution this is not just a cost argument — it is a usability argument. A knowledge base whose answers cost too much per query quietly gets abandoned, and people drift back to emailing a colleague. The economics and the adoption are the same curve.
Handling sensitive data before it reaches a model
Compression and desensitization belong in the same pipeline, and desensitization must happen before anything is vectorized. A robust setup detects sensitive regions — faces, identity documents, seals, and similar — and redacts or isolates them so those fields never enter the model's context window. In practice, coverage of around 13 distinct sensitive-information categories is what separates a toy pipeline from one that survives a compliance review. The preprocessing layer should also speak the formats institutions actually have: the practical bar is handling roughly 8 common formats, from office documents and PDFs to scans and specialized layout files such as OFD.
The ordering matters. If sensitive-data handling runs after retrieval, you have already leaked the field into an index. It has to run upstream, as part of the same pass that compresses.
On-premise deployment as a compliance primitive
For regulated industries, "deployed on our servers" is not a deployment detail — it is the whole point. Everything from document parsing and compression through vectorization to answer generation has to run inside the institution's own boundary, with no external service in the path. Air-gapped, container-based deployment is the realistic target: a single container image that brings up the whole stack offline, with no call home, no telemetry, no dependency on a vendor's cloud.
This is also where auditability stops being abstract. When the entire chain runs locally, you can record exactly which source a given answer drew from, meter usage per query, and produce an audit trail that a regulator will accept. Data sovereignty stops being a promise and becomes an observable property of the system.
What "working" looks like end to end
A useful way to judge a private knowledge-base effort is whether it passes an end-to-end acceptance test, not whether a demo answers one question. The chain that has to hold together is: upload, incremental ingestion, source-traced retrieval, citation-constrained question answering, and usage metering. Each stage is where real systems quietly fail.
Recent engineering progress on this front shows what is now achievable. A private, fully offline knowledge-base system developed at UglyPear Data passed end-to-end verification across exactly that chain and can be delivered as a container deployable entirely within a customer's intranet. Just as importantly, the response time for retrieval and answering dropped from over a minute to under a second. Sub-second matters less as a speed record and more as the threshold at which people actually use the tool instead of routing around it. The iteration here is engineering maturity — the difference between a prototype that answers in a demo and a system that holds up under real document volume with real compliance constraints.
A framework for choosing an approach
Rather than picking a named product, evaluate any candidate against the dimensions that actually determine success in a regulated setting. This is the checklist to carry into a vendor conversation or a build-versus-buy decision:
| Dimension to evaluate | What to ask | Why it matters |
|---|
| Data residency | Does any byte leave the institutional boundary, even for parsing or telemetry? | Determines whether sovereignty is real or merely contractual. |
| Answer auditability | Can every answer cite the exact source chunk? | A non-traceable answer is unusable in finance and healthcare. |
| Preprocessing coverage | Which formats and how many sensitive-data categories are handled before ingestion? | Drives both token cost and compliance exposure. |
| Query economics | What is the token cost per question after compression, not before? | Ingestion is one-time; querying is forever. |
| Deployment model | Air-gapped container, on-prem, or cloud-dependent? | Affects both security review and operational autonomy. |
| Incremental ingestion | Can new documents enter without re-processing the corpus? | Determines whether the system scales to millions of files. |
| Usage metering | Can you meter and report per-query usage? | Required for audit trails and cost allocation. |
This table is deliberately vendor-neutral. The right answer depends on your data volume, your compliance regime, and your team's capacity to operate infrastructure — not on which logo is on the box.
Principles to take into a build
If you are scoping this for your own institution, a handful of principles hold regardless of stack:
- Compress before you ingest. Treat the token surface of a document as a first-class design variable, not an afterthought.
- Make source-traceability an acceptance criterion, not a feature request. If an answer cannot show its source, it does not ship. In regulated contexts a wrong citation can drive a wrong decision.
- Start with one high-value document class. Regulatory bulletins or product manuals are good first targets. Prove the chain works, then expand.
- Handle sensitive information upstream. Detection of roughly 13 sensitive-information categories, applied before vectorization, is the bar where compliance reviews stop blocking.
- Measure the retrieval budget, not just the model. The compounding token savings from preprocessing are what make a large, heterogeneous corpus affordable to query repeatedly.
- Design for incremental ingestion from day one. A corpus of millions of documents cannot be re-processed every time one file changes.
The throughline
The institutions that succeed at private knowledge bases tend to share one habit: they treat document compression, sensitive-data handling, and deployment boundary as engineering problems solved up front, rather than features negotiated later. The model is the easy part. The hard, valuable work is making two million messy, sensitive, growing documents queryable in under a second without a single byte leaving the building. Get that pipeline right, and the question of which model to use becomes almost a footnote.
