Document dates and metadata fields

A file's modified time is when the source system last touched it. On a migrated or bulk-imported archive that says nothing about the document: an archive imported in one afternoon gives every note the same date. The real publication date is usually in the folder (26/04/15/…) or on the first page ("15 April 2026"), and the facts people filter by, such as ticker, sector, analyst and note type, are in the file name, the first page, or the line repeated at the top of every page.

Metadata rules read these back. Each source can have an ordered list of rules. Each rule produces a field from the file's path, the start of its extracted text, or its running page header.

Field	What it holds	Notes
doc_date	the document date	used by every "when" in search
ticker	upper-case	can hold several values: a sector note names every company it covers
sector	upper-case	typically read from the running page header
analyst	upper-case	can hold several values (co-authored notes)
note_type	lower-case	initiation, results, sector note, …

A file that no rule dates falls back to its modified time, so a source without rules behaves exactly as before.

Example: a broker-research archive

{
  "day_first": true,
  "rules": [
    { "field": "doc_date", "from": "path",
      "pattern": "^(?P<yy>\\d{2})/(?P<mm>\\d{2})/(?P<dd>\\d{2})/" },
    { "field": "doc_date", "from": "content", "detector": "first_date" },
    { "field": "ticker", "from": "content", "case_insensitive": false,
      "max_chars": 20000, "pattern": "\\((?P<value>[A-Z]{2,5})\\):" },
    { "field": "ticker", "from": "path", "case_insensitive": false,
      "pattern": "_(?P<value>[A-Z]{2,5})_[A-Z]{2,3}(?:_v\\d+)?\\.pdf$" },
    { "field": "sector", "from": "header", "case_insensitive": false,
      "pattern": "^(?P<value>[A-Z][A-Z &]+[A-Z]) \\| .*Research" },
    { "field": "analyst", "from": "content", "case_insensitive": false,
      "max_chars": 1500, "pattern": "Analyst: [^(\\n]+\\((?P<value>[A-Z]{2,3})\\)" },
    { "field": "note_type", "from": "content", "max_chars": 1500,
      "pattern": "Initiation of coverage", "value": "initiation" },
    { "field": "note_type", "from": "content", "max_chars": 1500,
      "pattern": "(?m)^(?P<value>[A-Za-z ]+?) \\| Analyst:" }
  ]
}

• Order is precedence. For each field, the first rule that produces a value wins. Above, the date folder decides and the first date on page 1 is the fallback.
• from: "path" matches the file's path. from: "content" reads the first max_chars characters of the extracted text (default 3,000, at most 20,000). from: "header" reads the line that repeats at the top of the document's pages.
• A rule has either a pattern (a regular expression; a keyword value is the group named value, a date is yy/yyyy, mm, dd or date) or the first_date detector, which finds the first date in a common English format (25 September 2026, 25 Sep 26, 2026-09-25, 25/09/2026). A rule can also give a constant value, used whenever its pattern matches.
• day_first (default true) reads 05/09/2026 as 5 September.
• Values are stored normalised: whitespace collapsed, punctuation trimmed and the field's case applied. A filter value is normalised the same way, so "Investment Companies" matches INVESTMENT COMPANIES.

Rules are checked when you save them. A pattern that does not compile, or an unknown field, is refused with the rule's position and the reason.

Setting the rules

In the console: open the source (Data › Sources, or press ⌘K / Ctrl+K and type # and part of its ID), then its Configure tab, Metadata fields card. It has a Simple form for the common cases, with Presets such as Ticker from file name and Sector from running header; Edit as JSON for everything else; and Preview, which shows what the rules read from the source's first files. Re-apply to all files reads the existing files again with the current rules.

From the CLI:

# Try rules on a sample path before saving.
kdbl-control source metadata-rules preview --source-id 's3://research' \
    --key '26/04/15/note.pdf'

kdbl-control source metadata-rules set  --source-id 's3://research' --file rules.json
kdbl-control source metadata-rules show --source-id 's3://research'

Saving the rules re-reads the source's existing files in the background, with no re-crawl. metadata-rules show and the console card report its progress. New and changed files are read as they are indexed.

Using the fields in search

• search_content (MCP) and the REST content search take published_after / published_before (YYYY-MM-DD, inclusive) on the document date, and fields, for example {"ticker": ["CORD"], "sector": "INVESTMENT COMPANIES", "note_type": "initiation"}. Every field named must hold one of its values.
• The filters are applied before ranking and under the caller's access rights, so every matching document the caller may read is reachable.
• Every hit carries its document_date and fields, and the first page's summary lists the most common values of each field among the matches.
• list_files takes the same filters.
• A ticker value may also be a company name that the tenant has aliased.

With these rules in place, an assistant can also ask in plain English: "all real estate initiations from 2025" becomes a sector filter, a note-type filter and a date window.