Sources

A source is one location KDBL Context Lake (K-Lake) indexes: an S3 bucket, an Azure Blob container, an SMB share, an NFS export, a SharePoint document library, or a user's OneDrive. This page covers the full lifecycle — adding, listing, configuring, and removing sources — across the UI, CLI, and API.

The source model

Every source has:

Field	Description
source_id	Stable, unique identifier within your tenant. Use a URI-style name such as s3://my-bucket, az://myaccount/documents, smb://nas.corp/finance, or odsp://contoso.onmicrosoft.com/site/<site-id>.
protocol	One of s3, azblob, smb, smbfs, nfs, onedrive.
config	Protocol-specific connection settings. See the protocol sections below.
Credentials	Provided once at creation. Stored encrypted at rest.
enabled	When false, workers stop crawling this source. Defaults to true.
bulk_ingest	When true, uses the optimized first-crawl write path. Defaults to true.
meta_caps	Set of optional enrichments to gather (S3 tags, Azure tags / content-type, NTFS / NFSv4 ACLs, xattrs).

Sources are tenant-scoped. Users in other tenants cannot see or address them.

Adding sources

S3

S3-compatible object stores including AWS S3, MinIO, Wasabi, and on-prem gateways.

Required: bucket name. Optional: endpoint_url, region, force_path_style (for MinIO-style gateways).

Credentials: access key ID + secret access key, or leave blank to use ambient credentials (IRSA, environment variables on the worker pod).

CLI:

echo "<secret-access-key>" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-s3 \
  --source-id 's3://my-bucket' \
  --bucket my-bucket \
  --region us-east-1 \
  --access-key-id AKIA... \
  --secret-access-key-stdin

API:

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources" \
     -d '{
       "source_id": "s3://my-bucket",
       "protocol": "s3",
       "config": { "bucket": "my-bucket", "region": "us-east-1" },
       "secret": { "access_key_id": "AKIA...", "secret_access_key": "..." }
     }'

Azure Blob

Azure Blob Storage containers, including ADLS Gen2 (hierarchical-namespace) accounts and the Azurite emulator. Source IDs use the az://account/container scheme.

Required: account (storage account name), container. Optional: endpoint_url — set this for sovereign clouds (*.core.usgovcloudapi.net, *.core.chinacloudapi.cn) or Azurite; it defaults to https://{account}.blob.core.windows.net.

Credentials — choose whichever your account uses:

• Microsoft Entra ID (recommended): leave credentials blank to use the worker's ambient identity (AKS Workload Identity or Managed Identity). Grant that identity the Storage Blob Data Reader role on the account or container — note that subscription Owner alone does not include data-plane read access.
• SAS token: a shared access signature scoped to the container with read + list permissions (sp=rl).
• Account key: a storage account access key.
• Connection string (CLI only): an AccountKey=… or SharedAccessSignature=… connection string — KDBL parses out the account, endpoint, and credential for you.

CLI (account key):

echo "<account-key>" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-azblob \
  --source-id 'az://myaccount/documents' \
  --account myaccount --container documents \
  --account-key-stdin

For other auth, swap --account-key-stdin for --sas-token-stdin or --connection-string-stdin, or omit all three to use ambient Entra ID.

API (SAS token):

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources" \
     -d '{
       "source_id": "az://myaccount/documents",
       "protocol": "azblob",
       "config": { "account": "myaccount", "container": "documents" },
       "secret": { "auth": "sas", "sas_token": "sv=2026-04-06&ss=b&srt=co&sp=rl&sig=..." }
     }'

For an account key use "secret": { "auth": "account_key", "account_key": "..." }; for ambient Entra ID omit secret entirely.

Archive tier is skipped. Blobs in the Archive access tier are listed but not extracted — reading them would require an explicit rehydration, with its cost and delay. Hot, Cool, and Cold (online) tiers are indexed normally. If a blob is archived after it was queued, the extractor skips it rather than triggering a rehydration.

Blob metadata and tags are mapped through to the index as searchable file metadata (captured during listing, at no extra request cost).

Preserving on-prem permissions on backed-up data. When files are tiered or backed up into Azure Blob by a file-gateway or backup product that stamps the file's original Windows security descriptor into blob metadata, K-Lake reads that descriptor back and applies the source's per-file security trimming as if the data were still on the file server. NTFS ACL-based access stays intact for cloud-tiered copies — a caller sees only the blobs their on-prem identity was entitled to. If the descriptor can't be parsed, the file fails closed (hidden) rather than being exposed.

Recovered original filenames (backup-gateway blobs). File-gateway/backup products often store each file under an opaque GUID blob name while keeping the real filename, extension, and path in blob custom metadata. K-Lake recovers those: the file-detail page shows the real filename as the title (with the GUID key kept below), plus an Original file card with the name, extension, source path, and the NTFS security descriptor — and the real name flows into search results and download filenames. Control this with the metadata profile on the source: Auto (default — detects backup-gateway blobs, a no-op for plain ones), FCG (force the mapping), or None (ignore custom metadata). Requires the Azure custom metadata cap.

SMB (userspace)

For SMB / CIFS shares accessed without a kernel mount.

Required: server, share. Optional: domain.

Credentials: username + password.

CLI:

echo "<password>" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-smb \
  --source-id 'smb://nas.corp/finance' \
  --server nas.corp --share finance \
  --domain CORP \
  --username svc-indexer \
  --password-stdin

SMBFS (kernel mount)

For SMB / CIFS shares mounted via the kernel CIFS client. Higher throughput than userspace SMB for large shares.

Required: server, share. Optional: domain, vers (defaults to 3.1.1), max_channels, extra_opts, backup_intent (defaults to false).

CLI:

echo "<password>" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-smbfs \
  --source-id 'smbfs://nas.corp/finance' \
  --server nas.corp --share finance \
  --username svc-indexer \
  --password-stdin

Backup-operator access (bypass file ACLs)

By default the extractor only sees files the configured account is granted by the share's per-file ACLs — indexing everything otherwise means re-permissioning the data. Instead, enable backup-operator intent and grant the service account the backup privilege on the NAS:

• What it does: adds backupuid=/backupgid= to the CIFS mount, so every file open carries FILE_OPEN_FOR_BACKUP_INTENT. The server honours it (bypassing per-file ACLs) iff the account holds SeBackupPrivilege — i.e. is a member of the server's Backup Operators group (or the vendor equivalent on NetApp / EMC Isilon / HPE). No file ACL changes are required.
• How to enable: pass --backup-intent to source add-smbfs, tick "Backup operator access" in the New Source form, or set "backup_intent": true in the config JSON. To flip it on an existing source without re-adding it, use the toggle (API mode), which re-mounts within ~30 s:

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source set-backup-intent --source-id 'smbfs://nas.corp/finance' --enabled true

- Failback (never-regress): if the worker's kernel doesn't understand the option it logs a warning and remounts without it, so a source that mounts today never breaks. A server that lacks the privilege for the account simply applies the normal ACL check (ACL-readable files stay readable). - Not on userspace SMB: the userspace smb backend cannot send backup intent; the API rejects backup_intent there with a pointer to smbfs. If a NAS hard-rejects backup intent for non-privileged accounts, leave the flag off.

echo "<password>" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-smbfs \
  --source-id 'smbfs://nas.corp/finance' \
  --server nas.corp --share finance \
  --username svc-backup \
  --backup-intent \
  --password-stdin

NFS

NFSv3 and NFSv4 exports mounted into the worker.

Required: server, export (must start with /). Optional: vers (defaults to 4.2), sec (defaults to sys), nconnect (defaults to 16), extra_opts.

CLI:

kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-nfs \
  --source-id 'nfs://nas.corp/export/data' \
  --server nas.corp \
  --export /export/data

OneDrive / SharePoint

Microsoft 365 content via the Microsoft Graph API — a SharePoint site's document library, or a user's OneDrive for Business. Source IDs use the odsp:// scheme.

Listing is incremental: the first crawl enumerates everything, and every crawl after that uses a Graph delta token to fetch only what changed — unchanged files are never re-listed or re-extracted.

Authentication is app-only (client credentials): one Microsoft Entra app registration serves the whole tenant, with no per-user sign-in and no stored refresh tokens. Each source targets one drive (one SharePoint library, or one OneDrive).

1. Register an Entra app (one-time)

In the Azure portal → Microsoft Entra ID → App registrations:

1. New registration — give it a name (e.g. "K-Lake indexer"); single-tenant is fine. Note the Directory (tenant) ID and Application (client) ID.
2. API permissions → Add a permission → Microsoft Graph → Application permissions → add Sites.Read.All and Files.Read.All → then Grant admin consent (a tenant admin must click this — an un-consented permission is the most common cause of a 403 at crawl time).
3. Certificates & secrets → New client secret → copy the secret's Value (not its ID — the value is shown only once).

The app needs only read permissions. It never needs User.Read.All unless you want the console wizard to enumerate users by directory — you can always target a OneDrive by its owner's email directly.

2. Add the source

Console (recommended): on the Sources page, Add source → OneDrive, enter the tenant / client ID / secret, then use Search to pick a SharePoint site by name (and its library), or switch to OneDrive (user) and resolve a drive by the owner's email. You never type a raw Graph id — the wizard discovers them from the credentials you entered.

CLI: the client secret is read from stdin, so it never lands in your shell history. Supply exactly one of --site-id or --drive-id.

# SharePoint site (its default document library):
printf '%s' "$ENTRA_CLIENT_SECRET" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-onedrive \
  --source-id 'odsp://contoso.onmicrosoft.com/site/<site-id>' \
  --entra-tenant contoso.onmicrosoft.com \
  --client-id <application-client-id> \
  --site-id '<hostname>,<siteGuid>,<webGuid>'

# A user's OneDrive (address the drive directly):
printf '%s' "$ENTRA_CLIENT_SECRET" | kdbl-control \
  --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source add-onedrive \
  --source-id 'odsp://contoso.onmicrosoft.com/drive/<drive-id>' \
  --entra-tenant contoso.onmicrosoft.com \
  --client-id <application-client-id> \
  --drive-id '<drive-id>'

--site-id and --drive-id are mutually exclusive. Add --site-drive-id to pin a specific library of a site (otherwise the site's primary document library is used). The connection is probed automatically before the source is saved; add --no-test to skip that check, or --force to save even if it fails.

API:

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources" \
     -d '{
       "source_id": "odsp://contoso.onmicrosoft.com/site/<site-id>",
       "protocol": "onedrive",
       "config": {
         "tenant_id": "contoso.onmicrosoft.com",
         "target": { "kind": "site", "site_id": "<hostname>,<siteGuid>,<webGuid>" }
       },
       "secret": {
         "auth": "client_secret",
         "client_id": "<application-client-id>",
         "client_secret": "<client-secret-value>"
       }
     }'

For a user's OneDrive, use "target": { "kind": "drive", "drive_id": "<drive-id>" }.

Discovering targets by name

You rarely know a SharePoint site or drive's raw Graph id. The console wizard's pickers are backed by POST /api/sources/onedrive/discover, which enumerates targets by friendly name straight from the app-only credentials — before the source is saved:

# Search sites by name:
curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources/onedrive/discover" \
     -d '{ "tenant_id": "contoso.onmicrosoft.com",
           "client_id": "<application-client-id>",
           "client_secret": "<client-secret-value>",
           "kind": "sites", "query": "finance" }'
# → { "items": [ { "id": "<site-id>", "name": "Finance Team", "detail": "https://…" } ] }

kind may be sites (search by query), drives (a site's libraries, by site_id), or user_drive (a user's OneDrive, by upn).

Notes

• Sovereign clouds: set --graph-base-url / --authority-host (CLI) or graph_base_url / authority_host (config) to the US Gov or China Graph and login endpoints. The defaults are the public cloud (https://graph.microsoft.com/v1.0, https://login.microsoftonline.com).
• Network egress: the deployment needs outbound HTTPS to graph.microsoft.com and login.microsoftonline.com (or the sovereign equivalents) — from the API (for discovery) and from the workers (for crawling).
• Throttling: Graph rate-limits app-only workloads; the source honours Retry-After and backs off automatically, so large libraries crawl steadily without manual tuning.
• Exclusions: set an exclude_extensions list (console or API config) to skip matching files entirely.
• Subtree work units don't apply. Like Azure Blob, OneDrive/SharePoint uses Graph's own delta listing rather than a directory walk, so the subtree fan-out knobs are not used for odsp:// sources.

Listing sources

UI: the Sources page lists every source in your tenant with file count, bytes, and last crawl time.

CLI:

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" source list

API:

curl -H "Authorization: Bearer $KDBL_TOKEN" "$KDBL_URL/api/sources"

Enabling and disabling

A disabled source stays in the registry but workers stop crawling it. Re-enabling resumes from the next crawl trigger.

UI: toggle the Enabled switch on the source detail page.

CLI:

kdbl-control source disable --source-id 's3://my-bucket'
kdbl-control source enable  --source-id 's3://my-bucket'

API:

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources/<urlencoded-source-id>/enabled" \
     -d '{"enabled": false}'

TLS trust anchors

Many on-premises endpoints — object stores, SMB gateways, webhook receivers — present certificates signed by an internal certificate authority that no public trust store knows about. Rather than turning verification off, install your CA as a trust anchor and keep certificates verified.

Anchors are managed per tenant:

kdbl-control tenant ca-cert add --name "Acme Internal Root" --file acme-root.pem
kdbl-control tenant ca-cert list
kdbl-control tenant ca-cert remove <cert-id>

list reports what each anchor actually is — subject, issuer and validity — parsed from the certificate rather than echoing back the name you gave it, so an anchor that is about to expire is visible before it starts failing crawls. Trust anchors are also manageable from the web console.

Each source then chooses how it uses them:

Mode	Behaviour
inherit (default)	Trust every anchor the tenant has installed.
custom	Trust only a named subset of the tenant's anchors.
none	Platform roots only — ignore the tenant's anchors for this source.

custom selects anchors by reference, not by holding its own copy. That means one place to rotate a certificate, and no way for a source to keep trusting one the tenant believes it removed.

Trust applies across the board, not just to crawling: identity lookups, object-store and Microsoft 365 connectors, and outbound webhook deliveries all honour the tenant's bundle. Operators can additionally install deployment-wide anchors that apply to every tenant.

Removing an anchor takes effect on the next configuration refresh — a few moments, not a restart.

Access grants

Whether a caller can see a source at all is controlled by its access grants. A caller sees a source if they're a tenant admin or hold a grant on it — viewer (read/search), editor (read + manage content), or owner (full control) — whose principal matches one of theirs (user:<id> or a group:<id> from directory enrichment). Grant to groups where you can: membership then drives access and survives identity churn.

Manage grants three ways, at full parity:

• Web console — the Access grants card on the source detail page: add a principal + role, or revoke.
• CLI:

kdbl-control acl list   's3://my-bucket'
kdbl-control acl grant  's3://my-bucket' 'group:sales' viewer
kdbl-control acl revoke 's3://my-bucket' 'group:sales'

• REST API — GET / POST / DELETE on /api/sources/<id>/acl.

Per-file security trimming then applies within a visible source — the two gates are complementary.

Hybrid search (lexical vs hybrid)

Search has two stacks: lexical (full-text — always on) and hybrid, which adds a dense embedding arm + reranking on top. Hybrid is more accurate but costs more: embedding compute at ingest and vector index storage. You can run a tenant or an individual source lexical-only to keep that cost off, and turn hybrid on later.

Hybrid enrichment is powered by a dedicated embedding service that scales independently of the ingest workers — so turning hybrid on for a large corpus adds embedding capacity without slowing extraction or the write path. (Earlier releases embedded inline in the workers.)

The effective setting is source override ?? tenant default ?? off, then ANDed with whether an embedding service is deployed. New tenants default to lexical-only.

# Tenant default (tenant-admin / cluster-admin):
curl -X PATCH -H "Authorization: Bearer $KDBL_TOKEN" -H "Content-Type: application/json" \
     "$KDBL_URL/api/tenants/<slug>/hybrid" -d '{"enable_hybrid": true}'

# Per-source override (true/false to set, null to inherit the tenant default):
kdbl-control source search enable  --source-id 's3://my-bucket'
kdbl-control source search disable --source-id 's3://my-bucket'
kdbl-control source search inherit --source-id 's3://my-bucket'
kdbl-control source search show    --source-id 's3://my-bucket'

Timing — the toggle is not instantaneous for ingest. Queries honour the setting immediately. New ingest, however, only picks up the change within ~30 s — workers read the resolved setting per task when work is dispatched. So if you flip a source and immediately force a re-extract, the in-flight task can still use the previous setting. Wait ~30 s after toggling before (re-)extracting if you need the new setting to apply to that ingest.

Disabling hybrid is non-destructive: existing embeddings are retained (and harmless to lexical results) until you explicitly reclaim them (kdbl-control source search reclaim --source-id …), which NULLs them in the background. Re-enabling later needs a re-crawl/backfill to rebuild vectors — see the capacity view below for the retained-vector bytes and the per-stack storage split.

Capacity & cost

Hybrid search is more accurate but not free — it adds embedding compute at ingest and a dense-vector index on disk. The Capacity & cost panel (on the console Dashboard, and cluster-wide on the Licence page) makes that cost visible so you can decide, per tenant or per source, whether hybrid is worth it.

It breaks down:

• Storage split — how many bytes each search stack uses: base (the raw content + metadata), lexical (the always-on full-text index), and hybrid (the dense-vector index). Per-tenant figures are prorated estimates; cluster-scope figures are exact.
• Counts — files, total chunks, and how many chunks are embedded (the share that's hybrid-ready).
• Retained vectors — vector bytes still held after hybrid was disabled but not yet reclaimed (and whether they're reclaimable).
• Query latency — average time and query count per mode (hybrid vs lexical), so you can weigh the accuracy/latency trade-off on real traffic.

The same numbers are available headless:

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" capacity
curl -H "Authorization: Bearer $KDBL_TOKEN" "$KDBL_URL/api/capacity"

Add --include-cluster (CLI) or ?include_cluster=true (API) for a cluster-wide view — that requires a cluster-admin token.

Content extraction

Every source runs its files through a content-extraction pipeline that pulls text (and structure — tables, headings) out of PDFs, Office documents, images, and more — and transcribes audio and video — ready for search.

Per-source extraction settings. Open a source's detail page (or use kdbl-control source extract …) to tune extraction for that source:

• On/off and a file-size ceiling — cap the largest file the extractor will open, to bound memory.
• Include / exclude paths and extensions (source exclude-extensions) — skip folders or file types you don't want indexed (build artefacts, media, and so on).
• Extractor choice — pick the extraction engine per source where more than one is available (see below).

K-Lake ships four extraction engines. You choose per source, and most estates never need to change the default:

Engine	Use it for
klex (default)	Standard text extraction. Fast and light, so many pods pack onto a node and extraction scales out cheaply. Lexical (keyword) search.
kdoc	Structured documents — tables and layout in complex PDFs and Office files. Also produces the embeddings that make hybrid (semantic) search work. Heavier per pod.
kvision	Scanned and image-heavy documents, using self-hosted vision models.
kmedia	Audio and video transcription.

Highest performance by default. The extractor defaults to its highest-throughput engine and scales out as a fleet — add replicas to extract many files in parallel. On suitable hardware this is roughly a 30× throughput uplift over the previous default. The high-performance engine is currently x86-64 only; ARM hosts fall back to the portable engine, with native ARM performance planned for a future release.

High-accuracy visual extraction (optional). For scanned documents, complex layouts, and image-heavy PDFs, an optional vision-model (VLM) extractor delivers higher-accuracy OCR and layout understanding using self-hosted, GPU-accelerated models — no document content leaves your environment. Enable it per source where the extra accuracy is worth the compute.

Audio & video transcription (optional). K-Lake can make audio and video files searchable by their content, not just their filename and metadata. A dedicated media extractor transcribes the spoken words and, for video, also reads the text shown on screen (frame OCR — slides, captions, titles). Every segment is timecoded, so a search hit deep-links to the exact moment the term is spoken or shown, and the media player opens there. Common audio and video formats are supported; transcription runs on self-hosted models (CPU, or GPU-accelerated for throughput) so no media ever leaves your environment. Like the other extractors it runs as a scale-out fleet — add replicas to transcribe more files in parallel — and is enabled where you want it. (Audio and video also carry their container metadata — duration, tags — into the index regardless.)

Watching progress. The source detail page shows live extraction progress in pages per second — a truer measure of extractor work than files/second, since a 400-page report and a one-page memo are very different jobs — alongside per-file status and any quarantined failures.

Global content search

By default a search spans every source the caller is allowed to see: results are drawn from all readable volumes at once and ranked together, so you don't have to know which share or bucket a document lives in. Access is still enforced per source and per file (see security trimming) — the global scope only ever includes sources the caller holds a grant on. You can still scope a search to a single source when you want to.

Adjusting metadata enrichment

Use source meta-caps (CLI) or the source detail page (UI) to choose which optional enrichments are gathered: S3 tags, Azure blob tags / content-type, NTFS ACLs, NFSv4 ACLs, extended attributes. Start narrow — every additional cap adds work per file. (Azure tags and content-type are captured during listing at no extra request cost.)

kdbl-control source meta-caps --source-id 's3://my-bucket' --caps s3_tags

You can also enqueue a backfill to retroactively enrich files that were indexed before a cap was added:

kdbl-control source backfill-meta --source-id 's3://my-bucket'

Triggering crawls

UI: click Crawl on the source detail page. Optionally narrow with a path prefix.

CLI:

kdbl-control crawl --source-id 's3://my-bucket'
kdbl-control crawl --source-id 's3://my-bucket' --prefix 'reports/2026/'

API:

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     "$KDBL_URL/api/sources/<urlencoded-source-id>/crawl"

Subtree work units

By default K-Lake crawls a source with a single walk that descends the tree one directory at a time. That is fine for most shares, but on a very large or very wide file system a single walk is slow to finish — one crawler works through millions of paths while the rest of your extraction fleet sits idle.

Subtree work units break a big tree into many independent crawl units that K-Lake hands out across all of your workers at once, so a huge share is crawled in parallel instead of end-to-end. You turn this on per source, from the Subtree work units card on the source's detail page (the Overview tab), and it is controlled by just two values.

The two knobs

Knob	What it does	Range
Depth	The directory level at which K-Lake stops walking and instead hands out each subtree below it as its own independent crawl unit. 0 (the default) keeps the classic single walk; 1–8 fans out. Deeper = more, smaller units = more parallelism.	0–8
Split threshold	A safety valve for a unit that turns out to be huge. Once a single unit has listed this many files, K-Lake re-hands-out its remaining, un-walked contents as fresh units so one runaway folder can't stall the crawl.	file count

Think of Depth as planned parallelism (you know the tree is wide, so split it up front) and Split threshold as adaptive parallelism (a unit turns out bigger than expected, so split it on the fly).

How Depth works

K-Lake walks down to the depth you set, and every directory at or below that level becomes an independent crawl unit; anything shallower is walked normally to reach it. Depth is counted in slash-separated path components — so for a bucket laid out as customers/<id>/<year>/<file>:

• Depth 1 → one unit per top-level folder (customers/, plus any siblings).
• Depth 2 → one unit per customer (customers/acme/, customers/globex/, …).
• Depth 3 → one unit per customer-year (customers/acme/2026/, …).

Pick the level where the number of folders is large enough to keep every worker busy but not so large that you create hundreds of thousands of tiny units — a target of tens to a few hundred units is a good rule of thumb. The maximum depth is 8.

How Split threshold works

Depth splits the tree by shape; it can't know that one particular folder holds 20 million files while its siblings hold a few thousand. The split threshold catches that case: while a unit is being crawled, once it crosses the threshold K-Lake stops, re-queues everything it hasn't reached yet as new units, and lets other workers pick them up.

• Leave it blank to use the system default (10,000,000 files), which bounds any single unit to a manageable size. This suits almost everyone.
• Set a lower number (e.g. 5000000) when you have one enormous, deep folder that a single worker would otherwise grind through alone — a smaller threshold makes it self-split sooner and spread across the fleet.
• Set 0 to disable splitting entirely: each unit runs to completion no matter how big it gets. Only do this if you specifically want whole subtrees handled by a single worker.

0 means "never split", not "use the default". To go back to the system default after setting a custom threshold, clear the field rather than entering 0.

Worked examples

Wide, shallow layoutDeep, dated hierarchyOne enormous folderTurn it off

An S3 bucket with thousands of top-level customer prefixes, each only a few levels deep. Fan out one unit per top level (or per customer) and let the default split threshold guard the outliers.

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source subtree --source-id 's3://acme-bucket' --depth 2

A NAS archive organised as archive/<year>/<month>/<day>/…. Descend a few levels so each month (or day) becomes its own unit.

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source subtree --source-id 'smbfs://nas.corp/archive' --depth 3

A single directory holding tens of millions of files, with little natural branching to split on. Depth alone won't help — rely on adaptive splitting by lowering the threshold, with a shallow depth to seed the fan-out.

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source subtree --source-id 'nfs://nas.corp/export/dump' \
  --depth 1 --split-threshold 5000000

Revert a source to the classic single walk.

kdbl-control --api-url "$KDBL_URL" --api-token "$KDBL_TOKEN" \
  source subtree --source-id 's3://acme-bucket' --depth 0

Setting the values

All three surfaces are at parity. At least one of the two values must be provided; the one you omit keeps its stored value.

• Web console — the Subtree work units card on the source detail page (Overview tab). Type a Depth and/or Split threshold and click Save subtree config. Leaving a box blank leaves that value untouched; to reset a value use the documented 0 semantics above.
• CLI — kdbl-control source subtree --source-id <id> [--depth N] [--split-threshold N] (see the examples above; requires API mode via --api-url/--api-token).
• REST API — POST /api/sources/<urlencoded-source-id>/subtree with a body of {"depth": N}, {"split_threshold": N}, or both. The response echoes the stored values.

curl -X POST -H "Authorization: Bearer $KDBL_TOKEN" \
     -H "Content-Type: application/json" \
     "$KDBL_URL/api/sources/s3%3A%2F%2Facme-bucket/subtree" \
     -d '{"depth": 2, "split_threshold": 5000000}'

Changes take effect for newly started crawl units within about 30 seconds; units already in flight finish with the settings they started with. Managing this requires tenant-admin, or an owner/editor grant on the source.

Watching it work

When a source is running in subtree mode, a Subtree-claim history card appears on its detail page with a live rollup of every unit K-Lake has processed, categorised by outcome:

Outcome	Meaning
drained	The unit was crawled all the way to completion.
split	The unit hit the split threshold and handed its remainder off to new units.
failed	The unit hit an error while listing or writing; it is retried automatically.
skipped	The unit couldn't be processed on the worker that picked it up and was re-queued.

The card also shows the largest crawl trees by file count and the most recent adaptive splits, and lets you search failed units by error message — a quick way to confirm your depth and threshold are producing the parallelism you expect.

Not available for Azure Blob or OneDrive/SharePoint. Subtree work units require the crawler to walk a hierarchical (folder-style) listing itself. Azure Blob containers are listed as a flat namespace, and OneDrive/SharePoint is crawled through Graph's own delta feed — so depth-based fan-out doesn't apply and is rejected for az:// and odsp:// sources. It is supported for S3, SMB, SMBFS, and NFS sources.

Enabling parallel crawling on the workers. For the depth knob to actually spread work across the fleet, subtree crawling must be enabled in your deployment's worker configuration. Large-deployment sizing profiles turn this on; if you set a depth but see no activity in the Subtree-claim history card, check with your operator.

Removing sources

Removing a source deletes its registry entry and its indexed files from the catalog. This is not recoverable — re-add and re-crawl to get back to a populated state.

UI: Delete action on the source detail page (requires confirmation).

CLI:

kdbl-control source remove --source-id 's3://my-bucket'

API:

curl -X DELETE -H "Authorization: Bearer $KDBL_TOKEN" \
     "$KDBL_URL/api/sources/<urlencoded-source-id>"