Telemetry

KDBL Context Lake (K-Lake) exposes Prometheus-format metrics, structured JSON logs, and standard Kubernetes health endpoints. Together they give you everything you need to alert on, dashboard, and debug a running deployment.

All of it is local. "Telemetry" here means what K-Lake exposes for you to scrape and read inside your own environment — it is not reported to KDBL. K-Lake has no call-home: no metrics, no logs, no usage counts and no crash reports are sent anywhere, and there is no address in the product that uploads to us. The only way anything reaches KDBL support is a diagnostic pack you generate, inspect, and send yourself.

Health endpoints

Every K-Lake service exposes:

Path	Purpose
/healthz	Liveness. 200 if the process is up. Wire to your liveness probe.
/readyz	Readiness. 200 only if dependencies (database, mounts) are reachable. Wire to your readiness probe.

Use /readyz for traffic gating; use /healthz only to restart wedged processes.

Metrics endpoint

Each service exposes Prometheus text-format metrics at /metrics on a dedicated port:

Service	Port	Notes
API	9100
Worker	9200	crawl / listing
Worker (stats sidecar)	9201	statistics rollup
Metadata enrichment	9102	optional metadata enrichment
Extractor	9200	content extraction (kdbl_extract_*)
Extractor (engine)	9101	content-extraction engine metrics

All metrics share the kdbl_ prefix.

Wiring it to Prometheus

There is nothing to "turn on" inside K-Lake — every service always exposes /metrics. You just need Prometheus to scrape it. How depends on your setup:

• Plain Prometheus (annotation-based discovery): the deployments already carry prometheus.io/scrape: "true" / prometheus.io/port / prometheus.io/path pod annotations, so a Prometheus configured with the standard kubernetes-pods scrape job picks them up automatically — no extra config.

• Prometheus Operator / kube-prometheus-stack (the common case): the pod annotations are ignored. Scraping is driven by ServiceMonitor objects, so each component needs one pointing at a Service with a named telemetry port. Ready-made ServiceMonitors for every component (API, workers, metadata enrichment, and the extractor deployments) ship with the deployment manifests — apply them into the same namespace. A minimal one looks like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kdbl-extractor
  namespace: kdbl
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kdbl-extractor
  endpoints:
    - port: telemetry      # the named Service port, not the number
      path: /metrics
      interval: 10s

If your kube-prometheus-stack restricts ServiceMonitor discovery by label (serviceMonitorSelector), add the release label it expects — e.g. release: <your-stack-release> — to each ServiceMonitor's metadata.labels.

Key metrics

Throughput

Metric	Type	Labels	What it tells you
kdbl_files_written_total	counter	backend, source_id	Files persisted to the metadata store
kdbl_files_inserted_total	counter	backend, source_id	New files added
kdbl_files_updated_total	counter	backend, source_id	Existing files with changed metadata
kdbl_files_unchanged_total	counter	backend, source_id	Files seen but already up-to-date
kdbl_bytes_indexed_total	counter	backend, source_id	Total bytes covered by indexed files

Queue depth

Metric	Type	Labels	What it tells you
kdbl_queue_depth	gauge	state (pending, running, done, failed)	Current size of each queue partition. Watch pending to know when to add workers.
kdbl_subtree_backlog	gauge	state (pending, running)	Backlog of in-progress crawl work as a deep share fans out. pending is the signal for queue-depth autoscaling — it grows when a large crawl outpaces the current workers.
kdbl_tenant_queue_depth	gauge	tenant (tenant id), state (pending, running)	The busiest tenants' queued and in-flight work across every queue, refreshed every 30 seconds. Only the top KDBL_TENANT_QUEUE_GAUGE_TOP tenants (default 20; 0 turns it off) get a series, so a deployment with thousands of tenants stays cheap to scrape. On a deployment hosting many customers, this answers "whose crawl is filling the queue?"
kdbl_inflight_tasks	gauge	worker_id	Per-worker concurrency utilization
kdbl_tasks_total	counter	protocol, outcome	Tasks completed, broken down by result

Latency

Metric	Type	Labels	What it tells you
kdbl_list_seconds	histogram	protocol	Time to list one directory / prefix from a source
kdbl_sink_write_seconds	histogram	backend	Time to persist one batch to the metadata store
kdbl_meta_fetch_seconds	histogram	protocol	Time to gather optional enrichments per file

Metadata enrichment

Metric	Type	Labels	What it tells you
kdbl_meta_files_written	counter	backend, source_id	Files with enrichment recorded
kdbl_meta_tasks_total	counter	protocol, result (ok, failed, skipped, parked)	Enrichment task outcomes
kdbl_meta_queue_depth	gauge	state	Enrichment queue size

Content extraction

Emitted by the extractor deployments (scraped via their ServiceMonitors — see above). The compute metrics are how you size GPU vs CPU extractor pools. Track throughput as pages per second — rate(kdbl_extract_pages_total[5m]) — which is a truer measure of extractor work than files/second (a 400-page report and a one-page memo are very different jobs) and is what the console's extraction view shows.

Metric	Type	Labels	What it tells you
kdbl_extract_tasks_total	counter	protocol, outcome (ok, failed, skipped, unchanged), plugin_version	Extraction tasks processed, by result. unchanged = the extractor's per-task dedup guard skipped an already-extracted file.
kdbl_extract_skipped_unchanged_total	counter	source_id	Files dropped from the extract enqueue because they were already extracted at their current version. This is the recrawl-dedup payoff — rises on a recrawl of an unchanged source; near-zero means files are changing (or the dedup isn't matching).
kdbl_extract_seconds	histogram	protocol	End-to-end extraction latency per file
kdbl_extract_plugin_seconds	histogram	plugin_version	Plugin-measured compute time per file (the GPU/CPU work itself)
kdbl_extract_pages_total	counter	protocol, plugin_version	Pages/segments the plugin processed
kdbl_extract_ocr_total	counter	plugin_version	Extractions where the plugin ran OCR (the heavy GPU path)
kdbl_extract_chunks_written_total	counter	protocol	Searchable content chunks persisted
kdbl_extract_bytes_total	counter	protocol	Source bytes streamed to the extractor
kdbl_extract_queue_depth	gauge	state (pending, running)	Content-extraction queue depth. The extractor HPA scales off pending.

Retention

Metric	Type	Labels	What it tells you
kdbl_retention_runs_total	counter	outcome	Retention sweeps completed
kdbl_retention_rows_deleted_total	counter	tenant	Rows removed by retention

Per-tenant limits

Emitted by the API on a deployment hosting many customers, where each tenant has its own request rate and concurrency shares (see Hosting many customers).

Metric	Type	Labels	What it tells you
kdbl_tenant_limit_total	counter	limit (rate, mcp, search, rerank), outcome (admitted, rejected, degraded)	Every per-tenant limit decision. rejected is a 429 (REST) or a tenant-limit error (MCP); degraded is a search served without reranking because no rerank slot came free in time.
kdbl_tenant_limit_rejected_total	counter	tenant, limit	The same refusals by tenant id — which customer is hitting its limit. A series exists only for a tenant that has been refused.

Background index builds and the console certificate

Metric	Type	Labels	What it tells you
kdbl_deferred_indexes_incomplete	gauge	—	Indexes an upgrade added that are still being built in the background. 0 once every one is complete. The worker re-checks every few minutes and finishes them without a restart; kdbl-control status --include-cluster lists any still building.
kdbl_deferred_index_leaves	gauge	index, state	Per index, how many of its parts are built, so a slow build shows its progress.
kdbl_console_tls_expires_timestamp_seconds	gauge	—	When the console's HTTPS certificate expires (Unix time); 0 when none is installed.

The chart's alert rules include KdblConsoleCertificateExpiring (warning, under 30 days left) and KdblConsoleCertificateExpiringSoon (critical, under 7 days). Renew under Settings › Security › HTTPS certificate; while it is close to expiry the console's command palette lists it under Needs attention (type !).

MCP metrics and alerts

Emitted by the API on its telemetry port (9100).

Metric	What it tells you
kdbl_mcp_tool_calls_total{tool,status}	Tool calls (and resource reads) by outcome: ok, error, not_found, denied
kdbl_mcp_tool_seconds{tool}	Tool latency
kdbl_mcp_auth_failures_total{reason}	Requests refused before authentication completed: no_token (the normal discovery probe every client opens with), invalid_token, origin_forbidden, tenant_suspended, server_error, cluster_admin_refused, scope_refused, throttled
kdbl_mcp_rate_limited_total{limit}	Refused by the per-user rate (principal) or by the failed sign-in throttle, per address (auth_ip) or per token (auth_token)
kdbl_mcp_scope_grace_total	Requests accepted on an unscoped personal access token during this release's grace period — re-mint those tokens with the mcp scope
kdbl_mcp_shed_total	Tool calls shed at the MCP concurrency limit
kdbl_mcp_oauth_tokens_issued_total, kdbl_mcp_oauth_errors_total{error}	The built-in sign-in server's token and client-registration endpoints
kdbl_mcp_oauth_refresh_reuse_total	A rotated refresh token was replayed — the token-theft signal; each one revokes that token family
kdbl_mcp_cimd_fetch_total{outcome}	Client metadata document fetches, including refused_address (a client id whose host resolves to a private address)
cloudflared_tunnel_ha_connections	Live edge connections per remote-access connector (tunnel on)

With monitoring.rules.enabled: true (needs the Prometheus Operator), the chart ships these alerts in the kdbl.mcp group:

Alert	Fires when	Severity
KdblMcpAuthFailureSpike	More failed sign-ins a minute than one address's budget (mcp.abuse.authFailPerMinute, 300) for 10 minutes	warning
KdblMcpAddressThrottled	Any address used up its failed sign-in budget	info
KdblMcpTokenThrottled	A client keeps retrying a token that is throttled on its own budget, for 30 minutes	info
KdblMcpScopeGraceInUse	Unscoped personal access tokens were used on MCP in the last hour (re-mint them before the grace ends)	info
KdblMcpRefreshTokenReuse	A rotated refresh token was replayed	warning
KdblMcpCallerRateLimited	A caller is being held to its request rate for 15 minutes	warning
KdblMcpToolErrorRate	More than 20% of tool calls fail for 15 minutes	warning
KdblMcpShedding	Tool calls are shed at the concurrency limit for 10 minutes	warning
KdblMcpSlowTools	Tool p95 over 10 s for 15 minutes	info
KdblMcpTunnelDown	A running connector holds no edge connection for 5 minutes (tunnel on)	critical

The limits these watch are described in Exposing MCP to the internet. Per-caller detail — who called which tool, from where, returning how much — is in the MCP audit trail.

Logs

All services log JSON to stdout. Configure the verbosity with the RUST_LOG environment variable, set in each component's ConfigMap:

RUST_LOG=info       # default
RUST_LOG=debug      # verbose, per-task detail
RUST_LOG=warn,error # quiet

Each log line carries (where applicable):

• tenant_id — owning tenant
• source_id — source the work was for
• task_id — work-unit identifier
• worker_id — emitting worker pod
• outcome — final result tag

Ship logs into whatever aggregation stack you use (Loki, Elasticsearch, CloudWatch). Filtering by tenant_id or source_id is the fastest way to scope an investigation.

Suggested alerts

Starting points for production alerting:

Alert	Condition
Queue building	kdbl_queue_depth{state="pending"} rising for >15 minutes and worker CPU at limit
Failed tasks rising	rate(kdbl_tasks_total{outcome="failed"}[5m]) > 0.1
Sink writes slow	histogram_quantile(0.95, rate(kdbl_sink_write_seconds_bucket[5m])) > 1
Readiness flapping	kube_pod_status_ready{condition="false"} for any K-Lake pod
Source unhealthy	last_error field returned by /api/sources/:id/health is non-null for >1 hour
A tenant is being limited	sum by (tenant, limit) (rate(kdbl_tenant_limit_rejected_total[5m])) > 0 for >15 minutes — raise that tenant's limit, or add API replicas

Dashboarding

The basic operator dashboard is just three panels:

1. kdbl_queue_depth per state, stacked
2. rate(kdbl_files_written_total[1m]) per source_id, top 10
3. histogram_quantile(0.95, rate(kdbl_sink_write_seconds_bucket[5m])) per backend

Build out from there based on the sources and SLOs that matter to your tenants.