Sizing guide¶
KDBL Context Lake (K-Lake) scales horizontally. Its workers are stateless — crawling, content extraction, and Smart Actions pipelines all run on pools of interchangeable workers that coordinate through a shared work queue and never duplicate effort. Add capacity and throughput goes up; the database is the floor on how fast metadata can be persisted, so size it for your peak ingest rate, not your steady state.
This page gives starting points. Tune from there based on the metrics in Telemetry.
Automatic scaling¶
K-Lake's worker pools scale automatically with the workload. When a backlog builds — a large first crawl, a bulk re-extraction, a burst of pipeline work — capacity is added; when the backlog drains, it's released again. Scaling is proportional: the busier the queue, the more workers are brought online, up to a ceiling you set, and back down to a baseline when things are quiet.
This uses standard Kubernetes autoscaling, so it fits naturally into a cluster you already operate, and it's driven by the same queue-depth signals you can see in Telemetry. The figures below are the per-pod sizing and the floor/ceiling you set for each pool; autoscaling moves the replica count between those bounds for you, so for most deployments you set sensible limits once and let K-Lake track demand on its own.
The crawl, extraction, and Smart Actions pools each scale independently, so a heavy extraction backlog doesn't starve crawling and vice versa.
Choosing a deployment size¶
K-Lake ships four deployment sizes — Small, Medium, Large, and Enterprise — that map one-to-one to the capacity tiers you can license. Each size preset provisions sensible baselines and scaling ceilings for every component, so the fast path is simply: deploy the size that matches the plan you bought. The automatic scaling above then moves the elastic pools between each size's floor and ceiling as your workload demands.
What each size provisions¶
| Small | Medium | Large | Enterprise | |
|---|---|---|---|---|
| Licensed capacity | up to 1 TB | up to 10 TB | up to 50 TB | 100 TB+ |
| Crawl workers (baseline → ceiling) | 1 → 3 | 2 → 10 | 4 → 50 | 8 → 200 |
| Extractor fleet (baseline → ceiling) | 0 → 4 | 1 → 10 | 2 → 25 | 2 → 50 |
| API replicas | 1 | 2 | 3 | 4 |
| Database CPU | 2 → 4 | 4 → 8 | 8 → 16 | 8 → 24 |
| Database memory | 4 → 8 Gi | 16 → 24 Gi | 32 → 48 Gi | 32 → 64 Gi |
| Database storage | 50 Gi | 256 Gi | 1 Ti | 2 Ti |
| Cache / work queue memory | 2 → 4 Gi | 4 → 8 Gi | 6 → 10 Gi | 6 → 12 Gi |
| Connection ceiling | 500 | 2,000 | 4,000 | 8,000 |
Values shown as baseline → ceiling mean the baseline is always running and the
component scales up to the ceiling under load. The extractor fleet's Small
baseline is 0 — it scales to zero when there's nothing to extract, so an idle
deployment carries no extractor cost.
The licence sets the capacity, not the size preset
The size preset governs the compute footprint (how many workers, how big the database). Your licensed capacity — how much indexed data you may hold — is enforced separately by your licence. Deploying a smaller preset to save resources does not reduce what you're licensed for, and vice versa.
Advanced database tuning per size
Each size also tunes the database's internals (shared buffers, connection limit, work memory, WAL size). Small starts at 1 GB shared buffers / 200 connections; Medium 8 GB / 600; Large and Enterprise 16 GB / 800. These are set for you by the size preset — override them only with a specific reason. Enterprise deployments frequently point at an external, highly-available database instead of the bundled one.
Translating a size to infrastructure¶
Kubernetes schedules pods by their CPU/memory requests, so the requests — not the limits — determine how many fit on your nodes. Two groups matter:
- Fixed baseline (always running): the database, the cache/work queue, the API and console replicas, the connection pooler, the Smart Actions pool, and the metadata workers. Size your cluster to hold these first.
- Elastic pools (scale with load): the crawl workers (~1.25 vCPU / ~0.75 Gi requested each) and the extractor fleet. These grow toward the size's ceiling when there's a backlog and release when it drains.
A quick rule of thumb: usable vCPU per node ≈ node vCPU − ~1 (kept for the system), and crawl-worker pods per node ≈ usable vCPU ÷ 1.25.
Example — three 8‑vCPU / 32 GB nodes (24 vCPU total), Medium. The fixed baseline (database ~4 vCPU, cache ~1, 2× API, 2× console, 2× pooler, metadata + Smart Actions pools) requests roughly 6–7 vCPU / ~26 GB before any crawling. The remaining ~17 vCPU comfortably runs 10+ crawl workers plus a few extractor pods — inside Medium's ceilings (workers → 10, extractor → 10). Fits Medium with headroom.
Example — a single 16‑vCPU / 64 GB node, Small. The database alone requests 2 vCPU / 4 GB; the baseline services add ~1.5 vCPU; that leaves ~12 vCPU for Small's ceilings (workers → 3, extractor → 4) with room to spare. This is the natural "first real node" when graduating from a proof of concept. Fits Small.
For Large and Enterprise, plan on multiple nodes (the crawl and extractor pools alone can reach dozens of pods) and, at Enterprise scale, an external highly-available database rather than a single bundled instance.
Defaults¶
The chart's shipped defaults set conservative requests with headroom in the limits.
Worker¶
| Request | Limit | |
|---|---|---|
| CPU | 1 core | 4 cores |
| Memory | 512 Mi | 2 Gi |
Each worker handles many in-flight crawl tasks concurrently. Increase replicas to increase throughput; the workers coordinate through the work queue and will not duplicate work.
Extractor¶
Content extraction runs as its own scale-out fleet, separate from the workers. It defaults to the highest-throughput engine and scales on queue depth — add replicas to extract more files in parallel (roughly a 30× throughput uplift over the previous default on suitable hardware). The high-performance engine is x86-64 only today; ARM hosts fall back to the portable engine, with native ARM performance planned for a future release. A GPU-accelerated, high-accuracy vision extractor is available for scanned or image-heavy documents — provision a small GPU pool for it if you enable it.
If you index audio or video, transcription runs as a separate, optional media-extractor fleet — budget for it independently of the document extractors, only when A/V sources are in scope:
- Heavier per pod, one file at a time. Each pod holds a resident transcription model (on the order of 1–3 GB of memory) and processes one media file at a time; you get parallelism by adding replicas, not concurrency per pod. Size pods for that memory floor and scale the fleet out.
- Compute-bound — GPU for throughput. Transcription is roughly minutes of compute per file on CPU; that's fine for modest volumes, but for large or continuously-growing A/V corpora provision a GPU pool (as with the vision extractor) for a large throughput gain.
- Video costs more than audio. Video is transcribed and frame-sampled for on-screen text (frame OCR reuses your OCR/vision backend), so a video minute is more work than an audio minute — factor that into the fleet ceiling if your corpus is video-heavy.
- Idle-cheap. Like the other extractors it scales on queue depth and back down when there's no A/V to process, so it adds no steady-state cost on deployments that only occasionally see media.
Embedding service¶
Hybrid search is powered by a dedicated embedding service that scales independently of both the workers and the extractor. It's only needed when hybrid search is enabled; size it to the ingest rate of your hybrid-enabled sources. Lexical-only deployments don't need it at all.
API¶
| Request | Limit | |
|---|---|---|
| CPU | 100 m | 1 core |
| Memory | 128 Mi | 512 Mi |
The API is mostly thin — it reads and writes the database on behalf of the UI and CLI. Two replicas is a sensible default for availability; scale up only if you push it with heavy programmatic API traffic.
UI¶
| Request | Limit | |
|---|---|---|
| CPU | 50 m | 250 m |
| Memory | 32 Mi | 128 Mi |
The UI is static assets served by a lightweight web server. One or two replicas is enough.
Database¶
K-Lake persists everything in a managed database. Recommended starting points:
- CPU: 2 cores, scale up under heavy ingest
- Memory: at least 8 Gi for ingests above a few million files
- Disk: provision for the eventual file count — figure tens of bytes per file record, plus indexes
A connection pooler is recommended in front of the database if you run more than a handful of workers.
When to scale up¶
Worker scaling is automatic (see above) — for the common case you don't intervene. Reach for this table when autoscaling has hit the ceiling you set, or for the parts that don't autoscale (the database, per-pod limits):
| Symptom | Action |
|---|---|
| Backlog sustained high and the worker pool is already at its ceiling | Raise the pool's maximum replica count |
| Worker CPU at limit, queue still growing | Bump the worker CPU limit |
| Database CPU pinned, queue stable | Scale the database vertically |
| API responses slow under heavy CLI/API use | Add API replicas |
| OOMKills on workers during very wide directory listings | Bump the worker memory limit |
Source-level tuning¶
Two source-level toggles affect throughput. Both are exposed via the UI, the CLI (source bulk-ingest, source meta-caps), and the API.
- Bulk ingest — defaults to on. Optimizes the write path for first-time crawls and large catch-up runs. Leave on unless you know you're doing many small incremental updates and have benchmarked the alternative.
- Metadata caps — controls which optional enrichments (S3 tags, Azure tags / content-type, NTFS / NFSv4 ACLs, xattrs) are gathered. Enabling more enrichment costs more crawl time and storage. Start narrow, widen as needed.
Estimating headroom¶
A worker pod can sustain steady-state ingest from a single source at network-bound rates for most object stores and NAS protocols. Real throughput depends heavily on:
- Object size distribution — many small files is harder than fewer large ones
- Source latency — same-region S3 is much faster than a remote SMB share
- Whether metadata enrichment is enabled
Plan for capacity using a representative source: run a crawl on it, watch the metrics, and extrapolate.