Skip to content

Introduction

KDBL Context Lake (K-Lake) turns the files you already have into a searchable, governed body of knowledge. It indexes your object stores and filesystems — both what each file is and what each file says — so an estate spread across buckets, shares and document libraries can be searched, understood, and answered against as one thing.

It runs as a small set of services in your cluster and reads your sources read-only. Nothing is moved, renamed, or rewritten: the originals stay exactly where they are, in the systems that already own them. Extraction is self-hosted, so no file content leaves your environment — unless an administrator switches on cloud extraction for a source, which sends only the pages the standard extractor cannot read to a document service in your own cloud account, with per-source consent and an egress audit of every file sent. What K-Lake builds is available through a web console, a command-line tool, a REST API, and an MCP endpoint that AI assistants connect to directly.

What you get

  • Search across the content, not just the file names. Hybrid lexical + semantic search over the text inside documents, spreadsheets, PDFs, scanned images, and audio and video transcripts — so a search finds the paragraph, not just the path. See Sources → Content extraction.
  • Grounded AI answers over your own data. Assistants connect to the MCP server and answer from what your files actually say, citing the file each fact came from with a link that opens the original.
  • Per-file security trimming. Every search, every API call and every AI answer is trimmed to what that person is already authorised to open, using the permissions your file systems already carry. Connecting an assistant does not widen anyone's access. See Security trimming.
  • Knowledge you can act on. Enrichment records the people, organisations, locations and agreements your documents name and how they connect — see the knowledge graph — and Smart Actions let you redact sensitive text, notify your own systems, and more, per source. All of it derived alongside your files, never written back into them.
  • A full inventory and an audit trail. The catalogue of what exists, where, how large and how old it is remains — with every access recorded.

What K-Lake indexes

A source is one location K-Lake is told to track. Today K-Lake supports:

Protocol Typical use
s3 S3-compatible object stores (AWS S3, MinIO, Wasabi, on-prem gateways)
azblob Azure Blob Storage containers (incl. ADLS Gen2 and the Azurite emulator)
smb SMB / CIFS shares accessed over userspace SMB
smbfs SMB / CIFS shares mounted into the worker via the kernel CIFS client
nfs NFSv3 / NFSv4 exports mounted into the worker
onedrive Microsoft 365 — SharePoint document libraries and OneDrive for Business, via Microsoft Graph

Each source has its own credentials (encrypted at rest), its own enable/disable flag, and its own metadata-enrichment policy.

Core concepts

Tenant. Every source, user, and indexed file belongs to a tenant. Tenants are fully isolated from each other — a user in one tenant cannot see another tenant's sources or files.

Crawl. A crawl walks a source from a root prefix and records what it finds. Crawls run on demand, on a recurring schedule, or through the API.

File record. For each file K-Lake discovers, it stores a record with path, size, timestamps, content hash (when available), and optional protocol-specific metadata (S3 tags, Azure blob tags / content-type, NTFS ACLs, NFSv4 ACLs, extended attributes).

Metadata caps. Each source declares which kinds of enrichment it wants (tags, ACLs, xattrs). Cheap fields are recorded inline during the crawl; heavier fields are enriched in the background. You can adjust caps per source at any time.

Content extraction. K-Lake reads each file's content so it becomes full-text (and semantically) searchable — text and structure (tables, headings) from PDFs and Office documents, OCR from images and scanned pages, and transcription of audio and video: the words spoken and the text shown on screen, with timecodes so a search hit deep-links to the exact moment. All extraction is self-hosted — no document content leaves your environment — and configurable per source. See Sources → Content extraction.

Token. Calls to the API and the CLI are authenticated with a personal access token (PAT) or an OIDC bearer token. PATs are minted from the UI or API and are tied to the user that created them.

What you'll see running

A typical K-Lake deployment includes:

  • API — serves the console and answers REST and MCP requests
  • Web console — the operator UI
  • Workers — stateless pods that crawl sources and record what they find
  • Extraction fleet — pods that read file content and turn it into searchable text; sized separately from the crawl workers, because extraction is the heavier job (see the sizing guide)
  • Database — the managed datastore holding metadata, extracted content, and the audit trail
  • Work queue — coordinates crawl and extraction work across the fleet

You manage sources, users, and tokens through any of the interfaces — console, CLI, or API. They all act on the same data.

Next steps

  • Quick start — sign in, add your first source, extract and search its content
  • Sources — protocol-specific configuration
  • Connect your AI — put an assistant on top of your own content
  • Sizing guide — plan resources for your fleet size