Cloud extraction¶

Most of your files are read quickly on ordinary CPUs by K-Lake's standard extractor. Some are not: scanned PDFs, photographed pages, images, and tables the standard extractor can't structure. Reading those well takes OCR and layout models, which are slow without GPUs.

Cloud extraction sends only those pages to a document service in your own cloud account, so you get high-quality OCR and tables without running any GPUs or inference servers yourself:

Provider	Service used
Azure AI Document Intelligence	The `prebuilt-layout` model: text in reading order, and tables
Amazon Textract	`AnalyzeDocument`, with tables and layout
Google Document AI	A Layout Parser or OCR processor in your Google Cloud project

The pages are processed under your own contract with your provider, in the region you choose, and every file that is sent is recorded in the source's egress audit.

flowchart LR
  F[File in a source] --> S[Standard extractor]
  S -- pages it reads well --> L[Indexed locally<br/>nothing leaves]
  S -- scanned pages and<br/>tables it can't structure --> C[Your own provider account<br/>Azure / AWS / Google]
  C --> I[Text and tables<br/>indexed with the rest]
  C -.-> A[Egress audit:<br/>provider, region, pages, bytes]

What leaves your environment¶

Only files from sources you switch on, and only after an administrator has confirmed that pages from that source may be sent.
Only the pages the standard extractor can't read properly from those files:
- pages with no usable text layer (scans and images), even where the standard extractor's own OCR managed some text, because that OCR can misread figures and loses tables;
- pages whose tables the standard extractor couldn't structure.
Pages with a real text layer stay local, and only those pages are sent. (You can instead route a whole source to the cloud service; see Routing a whole source.) - Every file sent is recorded in the source's egress audit, with the provider, region, number of pages and bytes.

Your provider's terms apply

Your provider's own terms and data-handling settings apply to what it receives. Check them for the service you choose, and take questions about your obligations to your own legal and compliance teams. KDBL can describe exactly what K-Lake sends.

Requirements¶

Licence: both the VLM extraction and Cloud extraction features. Cloud extraction doesn't work without VLM extraction. The console's Admin › Licence page shows both; see Licensing.
Deployment: your operator switches cloud extraction on once for the deployment (see For operators). It needs outbound HTTPS to your provider.
An account with one provider, set up as below.

Setting up your provider¶

In the console, open Settings › Search & content › Cloud extraction (or press ⌘K / Ctrl+K and type cloud extraction). You need to be a tenant administrator, and the card appears only once the licence includes Cloud extraction and your operator has switched it on. The card holds everything in one place:

Provider: choose Azure AI Document Intelligence, Amazon Textract or Google Document AI, then fill in that provider's fields (below).
Monthly page budget (optional): the most pages sent in a calendar month. Leave it empty for no limit. See Budgets and usage.
Save cloud extraction settings, then Test connection.
While cloud extraction is set up and on, the card's badge names your provider. Switch cloud extraction on for the sources you choose (see Turning it on for a source).

Azure AI Document Intelligence¶

In the Azure portal, create a Document Intelligence resource (or use an existing one) in the region where you want your pages processed.
From Keys and Endpoint, copy the endpoint (https://<name>.cognitiveservices.azure.com) and a key.
In K-Lake, choose Azure AI Document Intelligence and enter the endpoint, the key, and the resource's region (for example uksouth), so the egress audit records where pages were processed. Leave the model as prebuilt-layout unless you have trained a custom model.

Use the Standard (S0) tier

Use the Standard (S0) pricing tier for production. The free tier (F0) reads only the first two pages of each document it's sent, and allows only about 20 requests a minute. K-Lake copes with both limits, but a large backlog is then slow.

If you reach Document Intelligence through a private endpoint with its own domain, your operator must allow that domain first (see For operators).

Without a key (Microsoft Entra ID). Instead of a key, K-Lake can sign in as a managed identity your operator has set up. Grant that identity the Cognitive Services User role on the Document Intelligence resource, then tick Use this deployment's own cloud identity and leave the key empty. The resource needs a custom subdomain; resources created in the portal have one.

Amazon Textract¶

Create an IAM user or role with this policy:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "textract:AnalyzeDocument", "Resource": "*" }
  ]
}

Create an access key for it.
In K-Lake, choose Amazon Textract and enter the region (for example eu-west-2), the access key ID and the secret access key. A session token is needed only for temporary credentials. To keep the traffic on AWS's private network, create a Textract VPC interface endpoint and enter its URL as the VPC endpoint.

Google Document AI¶

In your Google Cloud project, enable the Document AI API and create a Layout Parser processor (or an OCR processor) in the eu or us location.
Create a service account with the Document AI API User role (roles/documentai.apiUser) and download a JSON key.
In K-Lake, choose Google Document AI and enter the location, the processor ID and the key file's contents. The project ID is taken from the key file unless you enter one.

Without a stored credential¶

If your operator allows it, you can use the deployment's own cloud identity (Azure workload identity, AWS IAM roles for service accounts, or Google workload identity) instead of storing a credential. Tick Use this deployment's own cloud identity and leave the credential fields empty.

Test the connection¶

After saving, press Test connection. K-Lake sends one sample page (a sentence and a small table) through the same path real files take, and shows you the text your provider read. If the test fails, the provider's reason is shown, for example a rejected key or a missing permission.

Turning it on for a source¶

Extraction must already be on for the source. Then open the source (Data › Sources, or press ⌘K / Ctrl+K and type # and part of its ID), go to its Configure tab, Extractor card, and tick Send pages this extractor cannot read to your cloud document service. You're asked to confirm, because pages from this source will leave your environment.

New and changed files are then handled automatically. Files that were extracted before you switched it on are picked up too: their pages that still need OCR are offered to your cloud service in the background, within a few minutes, with no re-extraction needed. To redo a whole source at once instead, force a re-extraction, for example of its PDFs only:

kdbl-control crawl --source-id '<source-id>' --force-reextract --only-ext pdf

The standard extractor reads each file again, and only the pages it can't read go to your cloud service.

Routing a whole source¶

To have the cloud service read every file of a source, choose Cloud document service as the source's extractor under Configure › Extractor (the Per-source extractor list). This sends much more, so use it only where the standard extractor adds little, for example a source of scanned archives. It reads PDFs and PNG, JPEG, TIFF (every page), WebP and BMP images.

Under Settings › Search & content › Cloud extraction you can also tick Send every page of a document routed here to send every page of a document routed to the cloud service, not just the pages without readable text. That gives better tables in born-digital PDFs, and more pages are billed.

Budgets and usage¶

Set a monthly page budget to cap what your provider can bill. The budget is a hard cap: pages are counted against it before they are sent, so several documents processed at once can't take you past it. The Settings › Search & content › Cloud extraction card shows the pages and files sent this month and in previous months.

When the budget is used up, the card shows a warning and nothing more is sent until next month or until you raise the budget. Meanwhile, pages that need the cloud service keep the standard extractor's text. They are offered again automatically, about once a day, and read as soon as the budget allows, with no re-extraction needed. The same catch-up applies to pages held back while cloud extraction was switched off.

To stop sending pages without losing your settings, switch cloud extraction off on the same card.

Audit¶

Every file sent is recorded in the source's egress audit (the Cloud egress log card on its Security tab), with the provider, region, pages sent and bytes.
The settings card shows who last changed the settings and when, and every change is written to the admin audit log.
The credential is never shown again once saved.

Troubleshooting¶

A file in a source routed wholly to the cloud service shows why it wasn't read. When only some pages are sent, a page the cloud service couldn't take keeps the standard extractor's text, and your operator can find the reason in the cloud extraction service's log.

Message	What to do
cloud extraction is not set up for this organisation	Choose and save a provider in Settings › Search & content › Cloud extraction.
cloud extraction is not included in this organisation's licence	You need both the VLM extraction and Cloud extraction features. See Licensing.
cloud extraction is switched off for this organisation	Switch it back on in Settings › Search & content › Cloud extraction.
cloud extraction has no credential, and this deployment does not allow the cluster's own cloud identity	Enter a credential in Settings › Search & content › Cloud extraction, or ask your operator to allow the deployment's own identity.
cloud page budget exhausted (also shown on the settings card)	Nothing to do: the pages are offered again automatically about once a day. Raise the budget to have them read sooner.
…rejected the key (401)…	Check the credential and endpoint, then press Test connection.
…not allowed (403)…	Grant the permission listed for your provider under Setting up your provider.

A provider that is throttling or briefly unavailable is retried automatically, with backoff.

For operators¶

Cloud extraction is off by default. Switch it on in the Helm values:

extract:
  cloud:
    enabled: true
    replicas: 1
    maxReplicas: 4
    concurrency: 8              # requests mostly wait on the provider
    ambientCredentials: false   # let tenants use the deployment's own cloud identity
    endpointAllowSuffixes: ""   # e.g. ".docai.corp.example" for private-link domains
    workloadIdentity:
      azureClientId: ""
      awsRoleArn: ""
      gcpServiceAccount: ""

Value	Meaning
`enabled`	Deploy the cloud extraction service. It is CPU only, needs no GPU or inference server, and runs on both x86-64 and Arm nodes (for example Graviton or Ampere).
`replicas`, `maxReplicas`	How many replicas run, and how far they scale out under load.
`concurrency`	Files each replica works on at once.
`ambientCredentials`	Allow tenants to leave the credential empty and use the deployment's own cloud identity. This lets every tenant on the deployment use that identity, so use it only where the deployment serves one organisation.
`endpointAllowSuffixes`	Extra Azure endpoint domains tenants may use (comma-separated), for private-link custom domains. Microsoft's own Cognitive Services domains are always allowed.
`workloadIdentity.azureClientId`	Client ID of a user-assigned managed identity, for Azure workload identity. Grant it Cognitive Services User on the Document Intelligence resource.
`workloadIdentity.awsRoleArn`	An IAM role (IAM roles for service accounts) allowed `textract:AnalyzeDocument`.
`workloadIdentity.gcpServiceAccount`	A Google service account (GKE workload identity) with `roles/documentai.apiUser`.

The workloadIdentity settings need ambientCredentials: true. Provider credentials are never set here: each tenant enters its own in the console, and they are stored encrypted, like source credentials.

Throughput and throttling. Each replica sends at most 8 requests at a time to one provider endpoint. When the provider throttles (HTTP 429), K-Lake waits for the time the provider asks for, and sends fewer requests at once until requests succeed again. A burst larger than your provider quota therefore slows down rather than failing. Azure Document Intelligence S0 allows 15 analyse requests a second by default; keep maxReplicas × 8 near that, or ask Microsoft to raise your quota.

Network. If your namespace denies egress by default, allow outbound HTTPS from the cloud extraction pods to your provider. The chart doesn't add an egress policy for them.