Sizing guide¶
KDBL Context Lake (K-Lake) scales horizontally. Its workers are stateless — crawling, content extraction, and Smart Actions pipelines all run on pools of interchangeable workers that coordinate through a shared work queue and never duplicate effort. Add capacity and throughput goes up; the database is the floor on how fast metadata can be persisted, so size it for your peak ingest rate, not your steady state.
This page gives starting points. Tune from there based on the metrics in Telemetry.
Automatic scaling¶
K-Lake's worker pools scale automatically with the workload. When a backlog builds — a large first crawl, a bulk re-extraction, a burst of pipeline work — capacity is added; when the backlog drains, it's released again. Scaling is proportional: the busier the queue, the more workers are brought online, up to a ceiling you set, and back down to a baseline when things are quiet.
This uses standard Kubernetes autoscaling, so it fits naturally into a cluster you already operate. Out of the box the pools scale on CPU; queue depth — the same backlog you can see in Telemetry — is an opt-in signal that reacts sooner, and Autoscaling covers how to turn it on. The figures below are the per-pod sizing and the floor/ceiling you set for each pool; autoscaling moves the replica count between those bounds for you, so for most deployments you set sensible limits once and let K-Lake track demand on its own.
The crawl, extraction, and Smart Actions pools each scale independently, so a heavy extraction backlog doesn't starve crawling and vice versa.
Choosing a deployment size¶
K-Lake ships four deployment sizes — Small, Medium, Large, and Enterprise — that map one-to-one to the capacity tiers you can license. Each size preset provisions sensible baselines and scaling ceilings for every component, so the fast path is simply: deploy the size that matches the plan you bought. The automatic scaling above then moves the elastic pools between each size's floor and ceiling as your workload demands.
What each size provisions¶
| Small | Medium | Large | Enterprise | |
|---|---|---|---|---|
| Licensed capacity | up to 1 TB | up to 10 TB | up to 50 TB | 100 TB+ |
| Crawl workers (baseline → ceiling) | 1 → 3 | 2 → 10 | 4 → 50 | 8 → 200 |
| Extractor fleet (baseline → ceiling) | 1 → 4 | 1 → 10 | 2 → 25 | 2 → 50 |
| API replicas | 1 | 2 | 3 | 4 |
| Database CPU | 2 → 4 | 4 → 8 | 8 → 16 | 8 → 24 |
| Database memory | 4 → 8 Gi | 16 → 24 Gi | 32 → 48 Gi | 32 → 64 Gi |
| Database storage | 50 Gi | 256 Gi | 1 Ti | 2 Ti |
| Cache / work queue memory | 2 → 4 Gi | 4 → 8 Gi | 6 → 10 Gi | 6 → 12 Gi |
| Connection ceiling | 500 | 2,000 | 4,000 | 8,000 |
Values shown as baseline → ceiling mean the baseline is always running and the
component scales up to the ceiling under load. No pool scales to zero: every
size keeps at least one extractor pod resident, because a CPU-driven autoscaler
has no signal to scale up from an empty deployment.
The licence sets the capacity, not the size preset
The size preset governs the compute footprint (how many workers, how big the database). Your licensed capacity — how much indexed data you may hold — is enforced separately by your licence. Deploying a smaller preset to save resources does not reduce what you're licensed for, and vice versa.
Advanced database tuning per size
Each size also tunes the database's internals (shared buffers, connection limit, work memory, WAL size). Small starts at 1 GB shared buffers / 200 connections; Medium 8 GB / 600; Large and Enterprise 16 GB / 800. These are set for you by the size preset — override them only with a specific reason. Enterprise deployments frequently point at an external, highly-available database instead of the bundled one.
Translating a size to infrastructure¶
Kubernetes schedules pods by their CPU/memory requests, so the requests — not the limits — determine how many fit on your nodes. Two groups matter:
- Fixed baseline (always running): the database, the cache/work queue, the API and console replicas, the connection pooler, the Smart Actions pool, and the metadata workers. Size your cluster to hold these first.
- Elastic pools (scale with load): the crawl workers (~1.25 vCPU / ~0.75 Gi requested each) and the extractor fleet. These grow toward the size's ceiling when there's a backlog and release when it drains.
A quick rule of thumb: usable vCPU per node ≈ node vCPU − ~1 (kept for the system), and crawl-worker pods per node ≈ usable vCPU ÷ 1.25.
Example — three 8‑vCPU / 32 GB nodes (24 vCPU total), Medium. The fixed baseline (database ~4 vCPU, cache ~1, 2× API, 2× console, 2× pooler, metadata + Smart Actions pools) requests roughly 6–7 vCPU / ~21 GB before any crawling. The remaining ~17 vCPU comfortably runs 10+ crawl workers plus a few extractor pods — inside Medium's ceilings (workers → 10, extractor → 10). Fits Medium with headroom.
Example — a single 16‑vCPU / 64 GB node, Small. The database alone requests 2 vCPU / 4 GB; the baseline services add ~1.5 vCPU; that leaves ~12 vCPU for Small's ceilings (workers → 3, extractor → 4) with room to spare. This is the natural "first real node" when graduating from a proof of concept. Fits Small.
For Large and Enterprise, plan on multiple nodes (the crawl and extractor pools alone can reach dozens of pods) and, at Enterprise scale, an external highly-available database rather than a single bundled instance.
Defaults¶
The chart's shipped defaults set conservative requests with headroom in the limits.
Worker¶
| Request | Limit | |
|---|---|---|
| CPU | 1 core | 2 cores |
| Memory | 512 Mi | 2 Gi |
Each worker handles many in-flight crawl tasks concurrently. Increase replicas to increase throughput; the workers coordinate through the work queue and will not duplicate work.
Extractor¶
Content extraction runs as its own scale-out fleet, separate from the workers. It defaults to the highest-throughput engine and scales on CPU — add replicas to extract more files in parallel. Per pod that engine is roughly 5–7× the throughput of the previous-generation one at a fraction of the memory, so many more extractor pods pack onto the same node and the fleet as a whole goes considerably faster than the ratio alone suggests. It runs on both x86-64 and ARM. For scanned or image-heavy documents both the document engine and the high-accuracy vision extractor can run GPU-accelerated — provision a small GPU pool if you enable either, and see OCR is the sizing question worth asking first for whether your estate needs one.
Most estates also run the document engine (kdoc2) as a small companion fleet beside the default engine: it reads the scanned PDFs and images the fast engine cannot, captures structured tables, and costs about the same per pod as the fast engine (roughly 2.25 vCPU / 2.5 GB reserved). One pod covers most estates — large scans are split into page ranges and spread across whatever fleet exists, so a single 400-page scan never pins one pod.
If you index audio or video, transcription runs as a separate, optional media-extractor fleet — budget for it independently of the document extractors, only when A/V sources are in scope:
- Heavier per pod, one file at a time. Each pod holds a resident transcription model (on the order of 1–3 GB of memory) and processes one media file at a time; you get parallelism by adding replicas, not concurrency per pod. Size pods for that memory floor and scale the fleet out.
- Compute-bound — GPU for throughput. Transcription is roughly minutes of compute per file on CPU; that's fine for modest volumes, but for large or continuously-growing A/V corpora provision a GPU pool (as with the vision extractor) for a large throughput gain.
- Video costs more than audio. Video is transcribed and frame-sampled for on-screen text (frame OCR reuses your OCR/vision backend), so a video minute is more work than an audio minute — factor that into the fleet ceiling if your corpus is video-heavy.
- Idle-cheap. Like the other extractors it scales on queue depth and back down when there's no A/V to process, so it adds no steady-state cost on deployments that only occasionally see media.
OCR is the sizing question worth asking first¶
OCR is the most expensive thing an extraction fleet does, and it is priced per page, not per file. Reading the text layer out of a digitally-generated PDF takes milliseconds; interpreting a scanned one page by page takes minutes on CPU, and a GPU is roughly fifty times faster. Two estates of identical size and file count can therefore differ by two orders of magnitude in how long they take to extract, purely on the mix of documents in them.
From 0.1.43, a PDF that already contains selectable text skips OCR automatically. On estates of generated documents — accounts, invoices, reports exported from a system — that removes the single largest cost in extraction, with no configuration and no loss: the text was already in the file. Scanned documents are unaffected and are still read by OCR.
What that leaves is a question worth answering before you size a deployment:
- Mostly born-digital? Extraction is CPU-bound and cheap. Scale the default engine out on ordinary nodes and plan no GPU.
- Substantially scanned? The estate is OCR-bound and the page count, not the file count, is what sets the timeline. Provision a GPU pool for the document or vision engine, or narrow what you index, or accept that scanned files become searchable on a longer schedule than the rest.
A sample of a few hundred PDFs is enough to tell which you have, and the answer often surprises people — archives of signed and executed paperwork skew heavily scanned even in organisations whose day-to-day documents are all generated. KDBL can survey an estate and give you the page counts and the projected extraction time before you commit to a deployment size; ask your KDBL contact.
Extraction plugin memory¶
The default extraction engine, klex, runs beside each extractor pod and is
sized per concurrent extraction slot (extractHost.concurrency): multiply the
figures below by the number of slots. The chart applies them by deployment
size:
| Corpus | Chart tier | Memory request / limit per slot | CPU request / limit per slot | Ephemeral storage request / limit per slot |
|---|---|---|---|---|
| Mostly office documents and born-digital PDFs | eval, small |
768 Mi / 1.5 Gi | 1 / 2 | 1 Gi / 3 Gi |
| PDFs up to 300 MB, CAD/BIM models up to 2 GiB | medium |
1.5 Gi / 2.5 Gi | 1 / 2 | 2.5 Gi / 4 Gi |
| Media-heavy office files, very large text outputs | large, enterprise |
3 Gi / 4 Gi | 1 / 2 | 4 Gi / 6 Gi |
The chart never sets the plugin's limits below 4 Gi of memory and 4 CPU, so with one slot every tier gets those limits; the per-slot limits take over once slots × limit passes that floor. The extractor container beside it reserves another 250m CPU and 512 Mi.
The per-file ceiling. From 0.1.44 the default largest file extracted is
2 GiB (extractHost.maxBytes; previously 50 MiB), so large engineering
models, drawings and media-heavy documents are read instead of being skipped as
oversize. Larger files are listed but not extracted. Ephemeral storage has to
hold the largest file in flight — an Office document with embedded media needs
about three times its own size — and every tier's ephemeral-storage limit holds
a 2 GiB file plus at least 1 GiB of temporary files, so a file the extractor
accepts cannot get the pod evicted by its size alone. A media-heavy Office file
above about 1 GiB needs more; use the medium or large figures if your corpus
has them. If you raise extractHost.maxBytes, raise
tierPresets.<tier>.klexSlot.ephemeralLimitMi with it.
A document that would exceed the plugin's memory fails on its own, recorded as
out_of_memory, and the other documents in flight carry on. Raise the memory
limit if you see that reason on files you need.
Enrichment and classification fleet¶
Entity enrichment and data classification share one enrichment fleet, separate from the extractors. Both read the extracted text through a language model, so their cost is text volume, not file count: per document the work is bounded by a character budget (200,000 characters by default), and a 2 MB spreadsheet costs more than a thousand one-page memos.
Measured throughput (K-Lake 0.1.41, the same corpus of long financial filings, each near the 200,000-character budget, entities + relations + classification):
| Fleet | Per pod | Per hour | Per day |
|---|---|---|---|
| CPU pod, 8–10 vCPU | ~1.2 documents / min | ~70 | ~1,700 |
| GPU pod, one current-generation accelerator | ~8 documents / min | ~490 | ~12,000 |
| GPU pods, classification only, short files (a few KB each) | ~120–150 files / min each | ~8,000 | ~180,000 |
A single GPU pod is roughly 7× a CPU pod on long documents, and pods scale out linearly: the work queue spreads documents across whatever fleet exists, so two GPU pods do twice the work of one.
What that means for an estate (long documents, enrichment + classification, first full pass):
| Documents | One CPU pod | One GPU pod | Four GPU pods |
|---|---|---|---|
| 10,000 | ~6 days | ~20 hours | ~5 hours |
| 100,000 | ~2 months | ~8 days | ~2 days |
| 1,000,000 | over a year | ~80 days | ~20 days |
Short documents (chat logs, tickets, small text files) classify an order of magnitude faster — around a million in a few days on two GPU pods — because the cost is text volume. Use your own average document size against the per-pod rates above.
Do not plan a million-document estate on CPU
CPU enrichment is for proofs of concept and small estates — up to the low
tens of thousands of long documents, or a few hundred thousand short ones.
Beyond that, or wherever the estate grows continuously, provision a GPU
pool for the enrichment fleet (enrich.gpu.enabled) and set its ceiling
to the number of GPUs you can give it. Without one, the backlog simply
never drains: the queue is safe, nothing is lost, but the console's
coverage figure stays partial for as long as it takes.
Per pod, as shipped:
- CPU: 2 vCPU / 4 Gi requested, 8 vCPU / 12 Gi limit, 4 documents in flight. Give it the whole limit — under 8 Gi the pod is killed on its first long document. One pod per 8–10 vCPU; add pods rather than cores.
- GPU: one accelerator per pod with at least 8 Gi of host memory, 4 documents in flight. Node placement is by the usual node selector / tolerations values; the fleet runs on both x86-64 and ARM nodes.
- Autoscaling (
enrich.autoscaling): scales on the number of documents waiting, so the fleet grows for the first pass and shrinks once only new and changed files arrive. Plan GPU time for the first pass; steady state is a fraction of it.
After the first pass, only new and changed files are processed. Two things re-run the whole estate, at the sweep's own bounded pace: a change to what is detected in the classification scheme (not to its rules), and an upgrade that changes the built-in scheme for tenants who never saved one. Both are visible as a backlog in the console; neither floods the queue.
Database. Classification adds a handful of small rows per file (one per label plus the level) and keeps its counts as running tallies, so the Classification page costs the same at a million files as at a thousand. Budget a few million small rows per million classified files.
Embedding service¶
Hybrid search is powered by a dedicated embedding service that scales independently of both the workers and the extractor. It's only needed when hybrid search is enabled; size it to the ingest rate of your hybrid-enabled sources. Lexical-only deployments don't need it at all.
API¶
| Request | Limit | |
|---|---|---|
| CPU | 100 m | 1 core |
| Memory | 128 Mi | 512 Mi |
The API is mostly thin — it reads and writes the database on behalf of the UI and CLI. Two replicas is a sensible default for availability; scale up only if you push it with heavy programmatic API traffic.
UI¶
| Request | Limit | |
|---|---|---|
| CPU | 25 m | 200 m |
| Memory | 32 Mi | 128 Mi |
The UI is static assets served by a lightweight web server. One or two replicas is enough.
Database¶
K-Lake persists everything in a managed database. Recommended starting points:
- CPU: 2 cores, scale up under heavy ingest
- Memory: at least 8 Gi for ingests above a few million files
- Disk: provision for the eventual file count — figure tens of bytes per file record, plus indexes
A connection pooler is recommended in front of the database if you run more than a handful of workers.
When to scale up¶
Worker scaling is automatic (see above) — for the common case you don't intervene. Reach for this table when autoscaling has hit the ceiling you set, or for the parts that don't autoscale (the database, per-pod limits):
| Symptom | Action |
|---|---|
| Backlog sustained high and the worker pool is already at its ceiling | Raise the pool's maximum replica count |
| Worker CPU at limit, queue still growing | Bump the worker CPU limit |
| Database CPU pinned, queue stable | Scale the database vertically |
| API responses slow under heavy CLI/API use | Add API replicas |
| OOMKills on workers during very wide directory listings | Bump the worker memory limit |
| Enrichment / classification coverage stays partial for days | The fleet is CPU-only or too small for the estate — see the fleet section; add GPU pods |
Source-level tuning¶
Two source-level toggles affect throughput. Both are available from the CLI (source bulk-ingest, source meta-caps) and the API; metadata caps are also on the Enrichment card of a source's Overview tab in the console.
- Bulk ingest — defaults to on. Optimizes the write path for first-time crawls and large catch-up runs. Leave on unless you know you're doing many small incremental updates and have benchmarked the alternative.
- Metadata caps — controls which optional enrichments (S3 tags, Azure tags / content-type, NTFS / NFSv4 ACLs, xattrs) are gathered. Enabling more enrichment costs more crawl time and storage. Start narrow, widen as needed.
Estimating headroom¶
A worker pod can sustain steady-state ingest from a single source at network-bound rates for most object stores and NAS protocols. Real throughput depends heavily on:
- Object size distribution — many small files is harder than fewer large ones
- Source latency — same-region S3 is much faster than a remote SMB share
- Whether metadata enrichment is enabled
Plan for capacity using a representative source: run a crawl on it, watch the metrics, and extrapolate.