Sizing for ingest time¶
The first thing a new deployment does is crawl a source and extract its content into the index. How long that takes is set almost entirely by how many extractors run, and extractors are CPU-bound. This page turns "how many documents, indexed by when" into an extractor count, and that count into vCPU.
For per-component requests and limits see the sizing guide.
What sets the speed¶
An ingest has three stages, and only one of them is usually slow:
| Stage | What it does | Measured cost |
|---|---|---|
| Crawl (worker) | Lists every file and records its metadata | 2,947 files listed from a Google Cloud Storage bucket in under 35 s. Rarely the limit. |
| Extract (klex) | Reads each file's text and chunks it for search | About 1.5 s of extractor time per file on a mixed estate. This is the stage to size. |
| Re-read (kdoc2) | Re-reads scanned pages and unread tables in PDFs, and the pictures inside Office files | Runs after extraction, on the documents that need it. Paced, see kdoc2. |
Each klex replica works on one file at a time and reserves 1.25 vCPU (it can burst to 6). Throughput rises with the number of replicas until the nodes run out of CPU.
The planning formula¶
- seconds_per_file: 1.5 for a mixed business estate. Use the per-type table below when you know the mix.
- 0.75: the share of extractor time that does useful work over a whole run. It allows for pods starting and for the long tail of a few heavy files at the end. Measured: 8.1 effective replicas out of 11.
Solved for replicas:
Worked examples¶
| Estate | Target | klex replicas | klex vCPU reserved |
|---|---|---|---|
| 100,000 files | 8 hours | 7 | 9 |
| 500,000 files | 24 hours | 12 | 15 |
| 1,000,000 files | 48 hours | 12 | 15 |
| 1,000,000 files | 24 hours | 24 | 30 |
Add the rest of the deployment on top: about 7.5 vCPU for the database,
workers, API, console, one kdoc2 replica and the embedding service at the
small tier.
Measured cost by file type¶
Measured on chart 0.1.52, GKE n2-standard-8 nodes, against 2,947 real
documents from the public GovDocs1 corpus (000–002, 1.8 GB). Times are
extractor time per file, the figure the formula uses.
| Type | Files | Average | Median | 95th percentile |
|---|---|---|---|---|
PowerPoint (.ppt) |
207 | 5.8 s | 1.8 s | 23 s |
| 726 | 2.3 s | 0.4 s | 4.1 s | |
Excel (.xls) |
137 | 1.8 s | 0.2 s | 9.8 s |
Word (.doc) |
254 | 1.7 s | 0.2 s | 5.9 s |
Images (.jpg, .gif), OCR'd |
269 | 0.4–0.6 s | 0.3 s | 1 s |
| HTML | 681 | 0.35 s | 0.15 s | 0.3 s |
| Plain text, CSV, logs | 532 | 0.06 s | 0.05 s | 0.1 s |
| Whole corpus | 2,947 | 1.4–1.6 s |
The averages are pulled up by a few large files: a 132-page PDF took 3.5 minutes and a 27-page PowerPoint 36 seconds. An estate of presentations and long reports runs closer to 3–4 s per file. One of mostly web pages and text runs well under 0.5 s.
Measured end to end: 2,947 files extracted in 9.6 minutes with 11 klex replicas on 3 × n2-standard-8, about 18,000 files an hour. That is the most klex those three nodes hold beside the rest of the stack.
Size from your own data¶
The crawl finishes long before extraction, so you can size against the real estate rather than a guess:
- Add the source and start the crawl.
-
As soon as it has listed, break the estate down by file type:
-
Multiply each type's file count by its time in the table above and add them up. Feed that total into the formula in place of
files × seconds_per_file. -
Set the extractor count (below) and watch the live rate and ETA:
Setting the extractor count¶
The tier's autoscaler starts small and scales on CPU: small runs 1 to 4 klex.
That suits steady state, after the first index. For a first index against a
deadline, pin the count you calculated (0.1.53 and later):
helm upgrade kdbl oci://ghcr.io/kdbl-consulting/kdbl --version <X.Y.Z> \
--namespace kdbl --reset-then-reuse-values \
--set extract.replicas=12
extract.replicas on its own fixes the count. Add extract.maxReplicas to let
the autoscaler run between the two (--set extract.replicas=4 --set
extract.maxReplicas=12), or set only extract.maxReplicas to keep the tier's
floor and raise its ceiling. The chart refuses a maxReplicas below
replicas. Once the first index is done, unset both
(--set extract.replicas=null --set extract.maxReplicas=null) to go back to
the tier's autoscaler. Measured with the default small ceiling of 4, about
80% of a 24 vCPU cluster sat idle during the first index.
The older tierPresets.<tier>.extract.min / .max keys still work;
extract.replicas and extract.maxReplicas take precedence over them.
Fitting replicas onto nodes¶
A pod cannot span nodes, and nodes keep some CPU back for the system. An
n2-standard-8 node has 7.9 vCPU allocatable. Three of them held the base stack
plus 11 klex; a twelfth stayed Pending. When nodes are fully packed, heavy
files run slower (the same PowerPoint took 36 s on a quiet node and 3.8 minutes
on a saturated one), so leave some headroom rather than filling every core.
A klex parse is not single-threaded: it used about 1.5 cores on average and about 2 on Word files. Pinned to one core, the same sample took 1.4 times as long. Fewer klex per node, each with room to burst, gets more done per vCPU than packing the node full.
Office files: pictures inside documents¶
Most of klex's time on Word, PowerPoint and Excel files goes on running OCR over the pictures embedded in them: logos, photos and pasted screenshots. Measured on 608 GovDocs1 Office files:
| Extraction time | Text | |
|---|---|---|
| OCR on embedded pictures | 1,291 s | baseline |
| Off | 211 s (6 times faster) | 0.2% fewer characters |
The words that OCR added were mostly the menus and address bars of pasted screenshots; some were genuine labels in diagrams. Office files were nearly half of all klex time on this corpus, so turning it off cuts total extraction time by about 40% there.
KDBL_KLEX_EMBEDDED_IMAGE_OCR on the klex container chooses what happens to
those pictures:
| Value | What it does |
|---|---|
on |
klex reads the pictures itself, inline. The default without a kdoc2 fleet. |
off |
Office files are read from their text only. Standalone images and the scanned pages of PDFs are still OCR'd. Files extracted while it is off are not re-read when you turn it back on. |
offload |
Office files are read at text-only speed, and the pictures that could hold text are handed to kdoc2, which reads them in the background. The default whenever a kdoc2 fleet is deployed (0.1.53 and later). |
To read Office files from their text only, for example for a first index on a deadline without kdoc2:
helm upgrade kdbl oci://ghcr.io/kdbl-consulting/kdbl --version <X.Y.Z> \
--namespace kdbl --reset-then-reuse-values \
--set-json 'extractHost.pluginEnv=[{"name":"KDBL_KLEX_EMBEDDED_IMAGE_OCR","value":"off"}]'
Offload: read the pictures with kdoc2 instead¶
offload keeps the speed of off without losing the text in pictures. klex
reads a Word, PowerPoint or Excel file (the current formats and the legacy
.doc, .ppt and .xls) from its text only, then sends the pictures that
could contain text to kdoc2. kdoc2 reads them in the background second pass,
with more accurate OCR, and adds their text to the document, usually within a
minute or two. A picture that appears in many documents (a logo on every
slide, say) is read once per tenant.
With a kdoc2 fleet, the chart does this for you. When extract.plugin or
an extract.additionalFleets entry is kdoc2, the chart sets
KDBL_KLEX_EMBEDDED_IMAGE_OCR=offload on klex. A value you set in
extractHost.pluginEnv wins: set on there to keep pictures read inline.
Pictures waiting for kdoc2 are held for up to 7 days and dropped if no kdoc2
replica reads them in that time. Use offload only where kdoc2 is running:
without kdoc2, the pictures are never read.
Under offload, klex applies these rules to the pictures in each document:
- Only PNG, JPEG, GIF, BMP, TIFF and WebP pictures are sent. Vector drawings (EMF, WMF) are skipped, because the text inside them is already extracted.
- Small pictures are dropped. A picture is sent only if both its short side and its long side are big enough (see the table below). For example, a 600 × 80 banner is sent, but a 48 × 48 icon is not.
- A picture that appears several times in the document is sent once.
- A PowerPoint picture is tagged with the number of the slide it is on. Pictures in Word and Excel files are not tagged with a page.
- Measured on 599 GovDocs1 Office files, listing the pictures took 2.5 s in total. klex found 2,435 pictures and sent 1,856 of them. It dropped 557 as too small and 19 as repeats, and dropped 3 more because their document was over the 64-picture limit.
Other documents are read as under on. This covers archives, e-mails,
OpenDocument and RTF files, whose pictures cannot be sent to kdoc2.
| Variable (klex container) | Default | What it does |
|---|---|---|
KDBL_KLEX_EMBEDDED_IMAGE_OCR |
on (offload with a kdoc2 fleet) |
on, off or offload |
KDBL_KLEX_OFFLOAD_MIN_SIDE |
64 |
Drop a picture whose shorter side is under this many pixels |
KDBL_KLEX_OFFLOAD_MIN_LONG_SIDE |
200 |
Drop a picture whose longer side is under this many pixels |
KDBL_KLEX_OFFLOAD_MAX_BYTES |
8Mi |
Drop a picture bigger than this (at most 15 MiB) |
KDBL_KLEX_OFFLOAD_MAX_IMAGES |
64 |
Most pictures sent per document; the largest are kept |
KDBL_KLEX_OFFLOAD_MAX_TOTAL_BYTES |
128Mi |
Most bytes of pictures sent per document |
kdoc2: scanned pages and tables¶
klex reads the text layer of a PDF and runs OCR on standalone images itself. A PDF page with no text layer (a scan), or a table klex cannot rebuild, is flagged and re-read by kdoc2 in a second pass:
- It runs after extraction, sized to the kdoc2 capacity that is running. Work is handed to kdoc2 in batches in proportion to the fleet: a GPU pod, which reads far faster, is given several times the work of a CPU pod. Documents with scanned pages go first, then documents with only unread tables, and the pictures from Office files alongside them.
- It holds back when kdoc2 is behind, so kdoc2's queue never grows without bound. Only kdoc2's own queue counts, so a busy first pass does not hold back the second.
- It keeps kdoc2 fed. While there is work to hand over, the second pass starts within about 30 seconds of the first flagged document, and the next batch arrives before any pod runs out. A kdoc2 fleet scaled up from zero gets work within about 30 seconds.
- Measured: 349 of the 2,947 documents qualified (12%), nearly all for tables. A table-heavy estate, such as financial filings, can qualify almost entirely.
- One kdoc2 replica on CPU is CPU-bound at about 3.3 vCPU. Adding replicas speeds up the second pass.
For a proof of concept, expect tables and scans to fill in after the main index. To finish the second pass sooner, add kdoc2 replicas, or give kdoc2 a GPU.
kdoc2 on a GPU¶
kdoc2's OCR is almost all neural-network inference, so a GPU speeds it up a lot.
Turn it on with gpu: {enabled: true} on the kdoc2 fleet (see the
GKE guide for a node pool). The chart then runs the
GPU build of kdoc2 with several OCR pipelines in each pod.
Why several pipelines. One pipeline keeps the GPU waiting: around each inference it prepares and post-processes the image on one CPU core, one image at a time. Pipelines side by side keep the GPU busy. Measured on an NVIDIA L4 (GKE g2-standard-8) with the 1,740 distinct pictures from the GovDocs1 Office files:
gpu.ocrPipelines |
Pictures/s | GPU busy | GPU memory | Pod CPU | Pod memory |
|---|---|---|---|---|---|
| 1 | 2.9 | ~40% | 4.4 GB | 1.0 core | 1.1 GiB |
| 2 | 4.7 | 45% | 1.6 cores | ||
| 4 (default) | 6.6 | 78% | 15.9 GB | 2.5 cores | 2.0 GiB |
| 6 | 7.2 | 77% | 22.1 GB | 3.3 cores | 2.6 GiB |
- Every setting produced byte-identical text.
- Each pipeline holds about 3.7 GB of GPU memory. A pipeline count that does
not fit fails extractions with
Failed to allocate memory. The default of 4 fits a 16 GB GPU and a 24 GB one. Use 6 only on a 24 GB GPU, and only if nothing else shares it.gpu.ocrPipelinestakes 1 to 16. - The chart runs at least as many extractions at once as there are pipelines, and sizes the pod's CPU and memory to the pipeline count.
- Cost. The L4 node costs $0.97 an hour on demand in London ($0.58 on Spot). At 6.6 pictures a second that is about $0.04 per 1,000 pictures, or $0.025 on Spot. kdoc2 on a CPU node costs about $0.18 per 1,000 pictures.
- One GPU pod in place of several. Before 0.1.53, kdoc2 read one image at a time, and four pods sharing one L4 through GKE time-sharing read about 6 pictures a second. One pod with 4 pipelines now does the same.
The whole index, measured. These are the 2,947 GovDocs1 files on 24 vCPU (two n2-standard-8 nodes for the stack and klex, plus one g2-standard-8 with an L4), with 5 klex replicas and one GPU kdoc2 pod at the defaults. The second pass had 338 documents with scans or tables and 1,740 pictures:
| Setup | First pass and second pass complete |
|---|---|
| kdoc2 on CPU (3 × n2-standard-8), pictures offloaded | 32 min |
| Before 0.1.53: 4 kdoc2 pods sharing the L4 | 17.4 min |
| 0.1.53: 1 GPU pod, 4 pipelines | 10.5 min |
The second pass finished within a minute of klex's last file, so klex was the limit. Run on its own after klex, the same second pass took 5.5 minutes, against 35 minutes for 3 CPU kdoc2 replicas.
What these numbers do not cover¶
- Audio and video: transcription holds 4 vCPU per file and runs at about real time on CPU. See the sizing guide.
- More than 11 concurrent klex replicas. The database was nowhere near busy at 11, but higher counts have not been measured on this setup.