Discover — "What data do we have?"¶

Discover is a data-estate overview: a single tenant-admin dashboard that turns everything K-Lake has crawled and extracted into an answer to the governance question every organisation eventually asks — what data do we actually have, where does it live, how old is it, how much is searchable, and how exposed is it?

It reads the metadata and extraction results K-Lake already holds — no re-crawl, no extra configuration — and aggregates them live into tiles, composition breakdowns, and a governance view.

The Discover dashboard: estate tiles (files, total size, sources, last indexed), a searchability bar, composition breakdowns by file type / content type / language / size / age / owner / storage tier / source, and a governance section with exposure signals.

Access. Discover is tenant-admin only — its aggregates span every source in the tenant, so it isn't per-user access-trimmed the way search is. A non-admin token is refused. Cluster admins can get a cross-tenant roll-up (see Cross-tenant).

What it shows¶

Overview tiles¶

Top-line totals for the whole tenant (or one source, via the source filter): files, total size, number of sources, and last indexed time.

Searchability¶

How much of the estate is actually searchable: the share of files extracted OK, plus pending / skipped / failed counts and the number of embedded chunks (hybrid-ready). This is the fastest way to spot a source that was crawled but never extracted.

Composition¶

Each dimension is a breakdown bar + top-N table. Toggle the whole page between ranking by file count and by total size (the Count / Size switch).

Dimension	Answers
File type	What file extensions dominate (derived from the object key).
Content type	MIME composition (from captured metadata).
Language	Detected language of the extracted text.
File size	Distribution across size bands — find the big-file hotspots.
Age	Distribution by last-modified time — how fresh (or stale) the estate is.
Owner	Who owns the data (where the source captures ownership).
Storage tier	Storage class — surfaces cold / archive spend.
Source	Which sources dominate the estate.

Governance — exposure signals¶

A first governance view, computed from metadata that already exists — the bridge to the upcoming data-classification release. It surfaces potentially risky or redundant data:

World-readable — files whose captured ACLs grant everyone read.
Stale (> 3 years) — not modified in over three years (ROT candidates).
No owner — no ownership captured.
Duplicate copies — files that appear to be duplicated (matching content hash).

Coverage is reported honestly. Content-type, language, and the exposure signals depend on metadata the source actually captured. Where a facet is based on a subset of files, the card says so ("based on N% of files with captured metadata") rather than presenting a partial number as ground truth. To raise coverage, enable the relevant metadata caps (e.g. NTFS/NFSv4 ACLs for exposure) and re-crawl.

Drill down to the files¶

Click a bar in an equality-shaped breakdown — file type, content type, owner, storage tier, or language — to list the actual files behind it (scoped to a source). This is the payoff: go from "12% of the estate is PDFs" straight to the PDFs themselves. (Size, age, source, and exposure are summaries only, not drill-through.)

Beyond the console — CLI & AI¶

Discover is available on three surfaces, all from the same aggregates.

CLI¶

kdbl-control discover prints the raw JSON (ideal for scripting or piping to jq). Requires API mode (--api-url / --api-token, or KDBL_API_URL / KDBL_API_TOKEN).

# Estate overview
kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" discover overview

# A breakdown by a dimension (rank by files [default] or bytes)
kdbl-control ... discover breakdown extension
kdbl-control ... discover breakdown size --metric bytes --limit 10

# The files behind one bucket (source, dimension, value)
kdbl-control ... discover files 's3://docs-bucket' extension pdf

breakdown takes a dimension (extension | mime | language | size | age | owner | storage_class | source | exposure) plus optional --source, --metric {files|bytes}, and --limit (max 100). files takes <source> <dimension> <value> and pages with --limit (max 1000) / --after <cursor>.

AI assistants (MCP)¶

An MCP tool — data_estate_summary — gives an AI assistant the same whole-estate overview so it can orient before searching ("what kinds of documents exist, how much, how fresh, how much is searchable, which sources dominate, what languages are present"), then target search_content intelligently. It's tenant-admin scoped like the dashboard. See MCP skills.

REST API¶

GET /api/discover/overview, GET /api/discover/breakdown?dimension=…, and GET /api/discover/files?… — see the API reference.

Cross-tenant (cluster admin)¶

A cluster-admin token can aggregate across every tenant by adding ?include_cluster=true (or --include-cluster on the CLI). Cluster scope supports the estate overview and the source breakdown; the per-file dimensions remain tenant-scoped.

Performance¶

Discover is built for scale: breakdowns are served from a maintained rollup with a statement-timeout guard and short-lived caching, so even a very large tenant answers quickly and a repeated view is served from cache. Figures labelled estimated (e.g. per-tenant storage) are prorated; the file and byte counts per source are exact.