Diagnostic packs¶

K-Lake has no call-home. It does not report crashes, does not send usage data, and has no channel to the vendor at all — that property is what makes air-gapped and regulated deployments possible, and nothing about support changes it.

The consequence is that when you open a support ticket, we cannot see your deployment. A diagnostic pack closes that gap on your terms: one command collects everything we would otherwise ask for over a week of email into a single archive that you generate, inspect, and send.

kdbl-control support pack \
  --api-url https://kdbl.internal --api-token "$KDBL_API_TOKEN" \
  --namespace kdbl

✓ diagnostic pack written
  path     kdbl-diag-20260811T183203Z.tar.gz
  size     4.2 MiB
  sha256   7626128ad87794a29caf181c5d8c8c4519486dff3c9c6c16c40a3e4d890a447c

Send that file to your KDBL support contact through whatever channel your policy allows. Nothing leaves your network until you do.

You can also generate a pack from the web console (the Support page in the sidebar, under Operations; or press ⌘K / Ctrl+K and type support) or the REST API — the same collector backs all three.

On an evaluation install: `poc-up.sh support`¶

If you installed K-Lake with the evaluation kit's poc-up.sh (see Proof of concept), you do not need kdbl-control, a kubeconfig, an API token or the namespace. Run this in the kit's directory on the evaluation host:

./scripts/poc-up.sh support            # Docker host
sudo ./scripts/poc-up.sh support       # Podman host, as for the install

✓ support pack written
  path   /home/you/kdbl-poc/kdbl-poc-support-kdbl-poc-20261014T080036Z.tar.gz
  size   288 KiB

Send this file to support.

It takes no arguments and asks no questions. It is read-only: it changes nothing in the cluster and does not switch your kubectl context, even if that context points at a different cluster. The one file it writes to the current directory holds:

Part	Contents
`summary.txt`, `MANIFEST.json`, `in-band/`	What K-Lake knows about itself, exactly as described below.
`cluster/`	Pods, events, nodes, Helm values and history, `describe` output per pod, and each pod's logs, including the previous container's logs after a crash.
`host/`	The container runtime's own status, containers and disk use, free space per mount, the kernel's out-of-memory messages, and the cluster node's own log.
`POC-SUPPORT.txt`	What was collected, how, and anything that could not be.

K-Lake redacts the host-collected files itself. Every file in cluster/ and host/ is passed through kdbl-control support scrub inside the running deployment: the same credential scrubber and leak check as the rest of the pack. A file that still matches the leak check is withheld and named in POC-SUPPORT.txt. If K-Lake is not running, or the evaluation runs a release that cannot redact them (0.1.51 and earlier), the pod logs, describe output, Helm values and node logs are withheld rather than sent unredacted, and POC-SUPPORT.txt says so; upgrade the evaluation to include them.

kdbl-control support scrub can be used on its own as well: it reads text on standard input and writes the redacted text to standard output. If the leak check still matches, it writes nothing and exits with status 3.

The kernel log is often readable only by root. On a Docker host where it is not, POC-SUPPORT.txt says so; re-run with sudo to include it.

If the API is down, the pack still carries the logs and events that explain why, and POC-SUPPORT.txt records that K-Lake's own half could not be collected.

Flag	Default	Notes
`--level`	`operational`	`metadata`, `operational` or `full`. See Redaction levels.
`--acknowledge-full`	off	Required with `--level full`.
`--runtime-tests`	off	Also exercise the dependencies. See Runtime tests.
`--cluster`	`kdbl-poc`	The evaluation cluster's name, if you gave it one at install.
`--context NAME`	—	Collect from another kubeconfig context instead (K-Lake in namespace `kdbl`).

With --context, every call is pinned to that context, so you can collect the same pack from a K-Lake that poc-up.sh did not install. host/ then describes the machine you ran the command on, not the cluster's nodes, and POC-SUPPORT.txt says so.

What it contains¶

Directory	Contents
`summary.txt`	Triage view — versions, licence, and every anomaly the collector noticed. Start here.
`MANIFEST.json`	What was collected, what failed, what was truncated, and a SHA-256 per file.
`in-band/`	What K-Lake knows about itself: configuration, sources and their settings, queue depth, crawls, grouped errors, per-pod self-reports, database health, and AI assistant (MCP) connections.
`cluster/`	What your orchestrator knows: pod state, events, node capacity, logs, Helm values.

AI assistant connections. When Microsoft 365 Copilot, Claude or another MCP client reports an expired session or stops returning results, the pack answers why on its own. It lists:

the AI clients registered on the deployment, and how each one signs in;
each client's sign-in sessions and how each one ended — still active, expired after 30 days unused, revoked by an administrator (or by removing the client), or revoked because a refresh token was replayed;
the MCP endpoint's calls and refusals over the last seven days, grouped by client and reason;
why each client's latest sign-in or session refresh was refused.

From these the pack writes findings in plain language — for example, that a client has no active session and why — and they lead summary.txt. The section carries no tokens, secrets or search queries at any level, and sessions are identified only by a short digest.

The two halves are collected differently, and deliberately so. K-Lake holds no Kubernetes credentials — granting the API permission to read pod logs would make an internet-facing component able to export your logs, just to save you one command. So the cluster/ half is collected by the CLI using your kubectl credentials, from your workstation.

The cluster/ half is deployment-wide. K-Lake's own data in in-band/ is scoped to the token you generate with, but pod logs interleave every tenant a pod serves — so on a multi-tenant deployment cluster/ can name sources and paths belonging to other tenants. Nobody gains access they did not already have (reading those logs needs cluster credentials), but the archive carries them, which matters when deciding who may receive it. Use --no-kubectl for a pack whose contents match your token's scope exactly.

Per-pod self-reports need cluster scope, or a single-customer deployment. A pod's self-report carries its effective configuration, mount paths and recent errors naming files for every tenant it serves, and cannot be cut down to one. A tenant admin's pack therefore withholds it, and says so. On a deployment that serves one customer, set the Helm value support.singleTenant: true (default false) and a tenant admin's pack — including one generated in the console — carries the self-reports too. Evaluation installs set it for you. The value is ignored in MSP mode, so a deployment hosting several customers keeps withholding them. The pack records whether the self-reports are included and which rule decided it.

On a deployment hosting many customers

When the deployment is in MSP mode, a tenant admin's pack describes only that tenant. Its queue counts are the tenant's own work rather than the whole deployment's, the database section — whose settings, activity and largest tables would describe other customers' workloads — is left out, and the licence is shown as the tenant's own quota and usage rather than the deployment's figures. A cluster admin's pack still covers the whole deployment. On a deployment serving a single customer, a tenant admin's pack is unchanged.

Either half is optional. No kubectl, or --no-kubectl, gives an in-band-only pack. An unreachable API gives a cluster-only pack — which matters, because "the API won't start" is a common reason to need one. Whichever half is missing is stated explicitly in MANIFEST.json, so nobody mistakes a partial pack for a clean bill of health.

Every pack carries each component's recent warnings and errors — including an in-band-only pack generated from the console, with no kubectl involved. Each K-Lake component keeps the warnings and errors it logged in the last 24 hours, grouped by message with a count and when each was first and last seen, and reports them with its health status, so the pack includes the line that names the problem even if the pod has since restarted. The list is bounded (200 distinct messages, about 20 KB per component), informational and debug lines are never kept, credential-named fields are masked, and search queries and document text are never recorded: anything logged while handling a search, an embedding or an enrichment has its text replaced with a placeholder before it is kept. At the metadata level each line is reduced to its count, timestamps, the part of K-Lake that logged it, and a pseudonym of the message. This is not a substitute for the full logs in cluster/, which carry informational lines and complete history. The most recent of these lines also appear among the findings in summary.txt and, for cluster administrators, under Recent warnings and errors on the System health page.

Runtime tests¶

Everything above describes how your deployment is configured. Add --runtime-tests and the pack also records what actually works:

kdbl-control support pack --runtime-tests

• running runtime tests…
  8 pass · 1 warn · 0 fail · 1 skipped

Check	What it catches
`database.round_trip`	A database that reads but cannot write — disk full, write-ahead log wedged, a replica promoted by mistake — answers `SELECT 1` perfectly and fails everything the product does. Also reports median and p95 latency.
`queue.dispatch`	A reachable queue that cannot hold an entry, and an eviction policy that silently discards queued work under memory pressure.
`search.embeddings`	An embedder that answers but returns a degenerate or wrong-sized vector, which makes hybrid search silently wrong rather than failing. Also flags CPU-speed embedding.
`search.rerank`	A reranker running on CPU, which leaves search correct and unusably slow. Projects the cost over your configured pool.
`fleet.clock_skew`	Clock drift between pods, which breaks token validity and licence evaluation in ways that never point at the clock.
`licence.enforcement`	Enforcement that is actively blocking operations, and capacity nearing its limit.
`storage.headroom`	Network mounts that have stopped responding, and volumes running out of space.
`extraction.plugin`	An extractor that heartbeats healthily but fails every document — an OCR model missing from an offline image, weights that never loaded. Pushes a test document through and checks the text comes back.
`enrichment.plugin`	An enricher that has stopped answering, so entity extraction silently produces nothing.
`sources.connectivity`	Expired credentials, a rotated key, a firewall change — all of which present as "the crawl finds nothing" and are otherwise only reported after the next crawl fails.
`mcp.configuration`	MCP enabled without a resource URI, which answers happily and then fails every client authorization.

Every check is bounded by its own timeout, so a wedged dependency is reported as wedged rather than hanging the pack. Nothing mutates your data: the two checks that write at all use a rolled-back transaction and a throwaway key.

A check that could not run reports skipped, never pass — "we didn't look" and "we looked and it's fine" are different answers.

Redaction levels¶

kdbl-control support pack --level metadata     # most private
kdbl-control support pack --level operational  # default
kdbl-control support pack --level full --acknowledge-full

Level	Includes	Use when
`metadata`	Counts, configuration, versions, error signatures. Every file path, source name, user and tenant — including source paths in component metrics and the deployment's public addresses in its configuration (0.1.52 and later) — replaced by a pseudonym generated for this pack alone.	The pack will be seen by more people than your named support engineer.
`operational` (default)	Adds file paths, source identifiers and verbatim error messages. No document content, no chunk text, no search queries.	Almost every ticket.
`full`	Adds audit-trail file-key samples and search query strings — what your users searched for and which files came back.	Retrieval, extraction or redaction faults, where the query is the bug.

full requires a cluster-admin token and an explicit acknowledgement. A tenant-admin requesting it gets an error rather than a quietly downgraded pack, because a pack that silently came back at a lower level would be read as evidence the data was not there.

The metadata pseudonyms are salted per pack and the salt is never stored. The same path appears identically throughout one pack — so we can still tell that two errors concern the same file — and differently in every other pack.

Pseudonymisation covers in-band/ only. The cluster/ half (pod logs, events, describe output) goes through the credential scrubber and the leak gate, so it carries no secrets — but file paths, share names and pod names appear verbatim, because they cannot be reliably identified inside free-form log text. If you need a pack with no verbatim identifiers anywhere, use --no-kubectl, which omits the cluster/ half entirely.

What is never included¶

Document content, extracted text, or chunk text
Source credentials in any form. The encrypted secret blobs are never read.
The master key, licence signing material, or download signing secret
API tokens, personal access tokens, or session cookies

Every text file in a pack passes through a credential scrubber, and the result is then checked by a separately written detector. If that detector matches anything, the pack is not written at all — there is no "generated with warnings" outcome.

Before you send it¶

Open it. Everything is plain text or JSON, and it is meant to be read.

tar xzf kdbl-diag-*.tar.gz
less kdbl-diag-*/summary.txt

The sha256 printed on generation is over the archive itself, so you and we can confirm we are looking at the same file.

Sending it somewhere else¶

If attaching the file isn't convenient, --upload-url PUTs it to a presigned URL you supply — your storage, your egress rule, your audit trail:

kdbl-control support pack --upload-url "https://your-bucket.example/…?X-Amz-Signature=…"

This is never a KDBL endpoint. There is no address in this product that uploads to us.

--encrypt-to-vendor writes an additional copy sealed to KDBL, so the file can travel by email or as a ticket attachment without its contents being readable in transit or at rest by anyone but us:

kdbl-control support pack --encrypt-to-vendor

The plaintext pack is always written too — read that one, send the sealed one. Encrypting in place would take away the "inspect it before you send it" property this whole feature is built on.

Each sealed pack names the key it was sealed to, so KDBL can tell a pack that needs an older key from one that arrived damaged. To print the key your build seals to — the id to quote on the ticket:

kdbl-control support key-id

Comparing two packs¶

"It worked last week" is a common way for a problem to be described. If you have a pack from before and one from after, the difference is usually the answer:

kdbl-control support diff last-week.tar.gz today.tar.gz

kdbl pack diff
  before  last-week.tar.gz
  after   today.tar.gz

config
  ~ config.sources[source_id=smbfs://nas/finance].bulk_ingest: false → true
state
  ~ state.queue.failed: 3 → 4102 (+4099)

2 difference(s).

Timestamps, counters, latencies, memory readings and pod names are excluded — they differ between any two packs of the same untouched deployment, and including them would bury the two lines above in thousands. Named counters like failed are reported as deltas instead of being dropped, because the magnitude is the finding.

Works offline on packs someone sent you: it needs no cluster and no API.

Auditing¶

Generating a pack is a bulk export of operational data, and your own security team should be able to see that it happened:

kdbl-control support history

{"packs": [{"at": "2026-08-11T18:32:03Z", "principal": "alice@example.com",
            "level": "operational", "scope": "tenant",
            "digest": "7626128ad877…", "bytes": 184320}]}

K-Lake records that a pack was generated, by whom, at what level, and its digest. It does not keep a copy — a cache of bulk exports sitting in your cluster would be a standing liability with no diagnostic value.

Seeing what a pack would contain¶

To review the scope before generating anything:

kdbl-control support preview --level operational

This lists every section, what it carries, and what is excluded, without collecting a single row. It is the same description the web console shows on its consent screen.

Useful flags¶

Flag	Default	Notes
`--namespace`	`kdbl`	Namespace holding the deployment.
`--log-lines`	`2000`	Per-container tail. Truncation is always recorded in the manifest.
`--since`	`24h`	Only collect logs newer than this.
`--no-kubectl`	off	Skip cluster collection entirely.
`--kube-context`	current context	Which cluster to collect from. Worth setting explicitly if your kubeconfig holds more than one.
`--runtime-tests`	off	Also exercise the dependencies (see above). Adds a few seconds.
`--encrypt-to-vendor`	off	Also write a copy sealed to KDBL, alongside the readable one.
`--upload-url`	—	PUT the pack to a presigned URL you supply.
`--out`	`kdbl-diag-<utc>.tar.gz`	Output path.

If your deployment is in licence lockdown¶

The support endpoints stay reachable when an expired licence has blocked everything else. A deployment in lockdown is exactly the one you are most likely to need a pack from.