Skip to content

Autoscaling

K-Lake autoscales its worker pools out of the box. Nothing to install, nothing to configure — this page is only worth reading if you want it to react faster to a large backlog.

What happens by default

Four pools scale on CPU between each tier's baseline and ceiling, using the metrics-server every managed cluster already ships:

Pool Does
Worker Crawls sources and lists files
Metadata Fetches per-file metadata and permissions
Extraction Turns documents into searchable text
Smart Actions Runs webhooks, redaction and enrichment chains

This is fine for steady workloads and needs no decisions from you.

When CPU isn't enough

Most of these pools spend their time waiting — on a file server, an object store, a webhook endpoint. A worker crawling ten thousand directories sits near idle because it is blocked on the network, not computing. CPU stays flat, so a CPU-based autoscaler sees nothing to react to while the backlog grows.

If your crawls are large or bursty and you want capacity added sooner, scale on queue depth — the number of items actually waiting — instead.

There are two ways to do that. Pick the one that matches what your cluster already runs.

Option A — KEDA (if you have no metrics adapter)

KEDA reads the queue directly from K-Lake's internal Valkey. No Prometheus, no metrics adapter, no scrape configuration.

1. Install KEDA (once per cluster):

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

2. Turn it on:

helm upgrade kdbl oci://ghcr.io/kdbl-consulting/kdbl \
  --namespace kdbl --reset-then-reuse-values \
  --set keda.enabled=true

That's the whole change. K-Lake creates the scalers, wires them to its own Valkey credentials, and takes over autoscaling for the worker, metadata, extraction and Smart Actions pools.

3. Check it took:

kubectl -n kdbl get scaledobject
kubectl -n kdbl get hpa          # KEDA manages these for you now

If KEDA isn't installed, the upgrade stops immediately and tells you so — it will not leave you with a half-configured deployment.

Tuning

Each pool has a target backlog per replica. Lower means capacity is added sooner.

--set keda.scalers[0].maxReplicas=40      # worker ceiling

The shipped defaults are deliberate:

  • Worker and metadata watch two queues and scale generously — this work is cheap per item and highly parallel.
  • Extraction uses a much smaller per-replica backlog, because each item is far heavier.
  • Smart Actions is capped at 4 replicas, well below the others. If your chains include an enrich step, the limit is the enrichment fleet's GPUs, not this pool — and scaling past that point reduces throughput. Raise it only if your chains are webhooks and redaction with no enrichment.

Option B — Prometheus (if you already run prometheus-adapter)

If you have Prometheus and prometheus-adapter, K-Lake can publish queue depth as a standard Kubernetes external metric instead.

1. Let Prometheus scrape the pools. Every pod carries a prometheus.io/scrape annotation, which is enough for annotation-based discovery. Running the Prometheus Operator instead?

--set monitoring.serviceMonitor.enabled=true

2. Add the adapter rules for the metrics you want, under rules.external — see Production deployment for the exact rules.

3. Enable the pool:

--set worker.subtreeAutoscale.enabled=true
--set actionHost.autoscaling.queueDepthAutoscale.enabled=true

Here queue depth is added to CPU rather than replacing it, so if the adapter is missing or a rule is wrong, the pool keeps scaling on CPU rather than stopping.

Choose one — they cannot coexist

KEDA and prometheus-adapter cannot both run in the same cluster. Kubernetes allows exactly one provider for external metrics, and both claim it. If you already run prometheus-adapter — it ships with kube-prometheus-stack, so you may have it without having chosen it — installing KEDA fails outright:

APIService "v1beta1.external.metrics.k8s.io" exists and cannot be imported
into the current release … current value is "prometheus-adapter"

That makes the decision simple:

  • Already running Prometheus and its adapter? Use Option B. KEDA would mean removing the adapter and re-pointing everything else that depends on it.
  • No metrics stack at all? Use Option A. It is one install instead of two, with nothing to wire.

Within a cluster running KEDA, the chart additionally refuses to enable KEDA and a Prometheus queue metric for the same pool — KEDA creates its own autoscaler, so the two would fight over one deployment.

What is deliberately not autoscaled

  • Audio/video transcription saturates its CPU cores for the whole file, so ordinary CPU scaling already reacts correctly. Add replicas for more throughput.
  • The GPU transcription fleet is never autoscaled: each pod claims one GPU, so an extra replica would wait forever for hardware that doesn't exist. Size it to your GPU count.
  • Enrichment is bounded by available GPUs for the same reason.

Scaling in

Pools shed capacity slowly and finish in-flight work first. Removing a pod mid-document leaves that file waiting for its lease to expire before another pod picks it up, so K-Lake favours a calm retreat over a fast one.