An internal app can assume things a product cannot. It can assume one tenant, a caller who will read a release note, a field that means whatever the last person to touch it decided, and a bill nobody itemises. Each of those assumptions is cheap to make and expensive to remove, and removing them later is a breaking change to somebody else’s integration.
This cartridge was built with the opposite defaults. The distinction is concrete, not positioning:
| Concern | Internal app default | What this does instead |
|---|---|---|
| Identity | The signed-in operator | tenant_id partitions every read and write. A partner presents <key_id>.<secret>; the key resolves to its own tenant, which then partitions everything downstream. Only the SHA-256 hash is stored — the secret is shown once. |
| Output shape | Whatever the extractor returns | A declared contract with a schema_version. Every key is always present; a value may be null, the key may not be missing. Tenants extend only inside extensions. |
| Change | Edit in place | An approved record is superseded by a new version, never edited. The prior version stays retrievable, so “what did we pay against?” survives a later correction. |
| Cost | An aggregate cloud bill | Every model call emits a completion line carrying tenant, model, tokens and dollars — including the calls that failed, which is where the accounting most recently broke. |
| Prompts | Constants in the deploy | Served from the Console Content CMS with the bundled constant as an unconditional floor. Tuning is no longer a Terraform apply. |
Does any of our code branch on this field? If yes, it is schema. If no, it is metadata. The question is deliberately about our behaviour rather than the field’s importance, because promoting a metadata field into the schema later is a breaking change for every integrator — and demoting one is worse.
The cartridge went from scaffold to product surface in 16 days. The order matters more than the speed, because each stage was only safe once the previous one could be measured.
| Stage | What it established |
|---|---|
| Scaffold | The _template shape: API Gateway HTTP API behind CloudFront with X-Origin-Secret, Cognito JWT plus requireCartridgeGroup, a permission boundary on every role, customer-managed CMKs. None of this is cartridge-specific; it is inherited, which is the point of the substrate. |
| One extraction | A single unified pass. Deliberately the crudest approach, because it establishes the measurement harness — canonical shape, cost accounting, completion logging — before there is anything to compare. |
| Four approaches | Competing topologies for the same job, raced rather than argued about (§4.2). This is the stage that makes the cartridge a laboratory instead of an opinion. |
| Comparison surface | Four models × four approaches, with per-cell cost and duration. The point is not to pick a winner once but to keep re-deciding as models and prices move. |
| The contract | 29 document types over one field registry (§4.3), emitted alongside the existing canonical shape rather than replacing it — so the mapping is proven against live traffic before any consumer depends on it. |
| Learning + audit | Pins (§5) let a correction teach the next document. Provenance (§6) records why a run produced what it did, captured at the time rather than recomputed. |
| Audit rounds | Four review passes from different lenses — doctrine drift, emergent abstractions, testability, and the operator’s view. Each shipped fixes and, more importantly, converted findings into invariants (§7). |
Across every audit round the defects were silent wrong numbers, not crashes. A guard that only catches the variant someone thought to write down is the same failure it was meant to prevent. Two examples from the most recent round: a repair pass whose every fault was swallowed by a catch and looked identical to “nothing to repair”; and the two most expensive extraction paths logging their spend under names the cost dashboard filters out — so the priciest documents were the ones missing from the bill. In the second case the platform had been emitting a logComplete-missing-field warning on every occurrence the whole time. The guard worked. Nobody read it.
These are load-bearing. Each is written so that it is obvious what evidence would overturn it.
| Assumption | Why we hold it — and what would break it |
|---|---|
| The model cannot certify itself. | On our own sample, mean self-reported confidence was 0.940 when correct and 0.949 when wrong. Confidence is therefore not evidence, and no record becomes authoritative on the model’s say-so. Overturned by a calibration measurement showing separation. |
| A document type is a closed vocabulary; tenant variation is an extension. | 29 types cover the freight corpus we have. A tenant that needs a 30th gets it through mapsTo, not by inventing a private type — otherwise no two tenants’ data can be compared. Overturned if extension usage concentrates on one shape, which would mean the base is wrong rather than incomplete. |
| Most document types legitimately carry very little. | Schema membership was derived by reading the consumers. A bill of lading carries identity and nothing more because nothing branches on more. Inventing cargo structure nobody reads is a promise to integrators we have not earned. Overturned the moment a consumer appears — then the group is added as a versioned change. |
| A field can be required only if we generate it. | All 9 required fields are platform-generated (doc_id, tenant_id, created_at, schema_version, source, doc_type, version, status, extensions). Nothing read from a document is required, because the document is the one thing we do not control. |
| Prompt caching makes carrying unused guidance cheaper than fetching the right guidance. | Cached prefix tokens bill at 10%, so carrying the full 29-type feature dictionary in one cached prompt costs less than a tool call to select the right entry — and stay a single round trip. This is why unified-typed is viable at all. Overturned by a pricing change; the risk it carries is dilution, which is why it is raced rather than assumed (§4.2). |
| Correction geometry transfers between documents from the same vendor. | Pins are stored as fractions of page width and height, never pixels, and as regions rather than points. Overturned per-vendor by the miss counter, which retires a pin whose layout has changed. |
Three layers, each with a decision that was contested and is worth recording.
Classification answers what is this document against a closed vocabulary of 29 types — 14 that carry money (invoices, credit and debit notes, statements, per-diem) and 15 that do not (bills of lading, airway bills, arrival notices, customs releases, delivery orders, manifests, packing lists).
Two decisions define this layer:
other, never null. other carries identity — number, date, issuer, references — and nothing else. Discarding those would make an unclassified document unfindable afterwards, which is worse than an imperfect label; but inferring a total for a document we could not classify is how wrong data acquires an authoritative face.{{doc-type-guide}} and is resolved through one substitution.The four approaches differ only in how classification relates to extraction. Each has a real failure mode, which is why the cartridge runs them side by side instead of picking one on reasoning alone.
| Approach | Calls | Duration | Trade-off it makes |
|---|---|---|---|
unified | 1 | t | One pass, no type hint. The baseline and the measurement harness. |
unified-typed | 1 | t | One pass carrying per-type field guidance; the model selects the section itself. Cheapest targeting via caching. Risk: dilution — a 29-type dictionary in one prompt may blur the entry that matters. |
sequential-typed | 2 | t₁ + t₂ | Classify, then extract with the type as a hint. Highest targeting. Risk: cascade — a wrong type mis-hints extraction, and the second pass cannot recover it. |
parallel-raw | 2 | max(t₁, t₂) | Classify and extract concurrently; a deterministic step folds the type into the raw fields. No cascade — extraction never waits on or is starved by classification. Pays for two calls to buy independence. |
All four run against four models — claude-haiku-4-5, claude-sonnet-4-5, nova-lite, nova-pro — with per-cell cost and duration. Two rules keep that comparison honest:
Nova and Anthropic use different request and response schemas: inputTokens versus input_tokens, inferenceConfig versus max_tokens, output.message.content versus content[].text. Read the Nova response with the Anthropic reader and usage comes back undefined → zero tokens → $0.00 → Nova wins on cost, every time, for free. The translation is explicit in the shared call path for exactly this reason. The same shape recurs anywhere a metric can be absent rather than wrong: the dangerous number is not the large one, it is the zero.
The second rule: a truncated or unparseable response still spent money, so it is billed as a completion carrying an outcome field rather than logged as an error and lost. The truncation path is the most expensive in the cartridge — two calls, the second at the larger output ceiling, triggered by the largest documents — and it was, until the fourth audit round, entirely absent from the cost dashboard.
A document type does not define fields. It declares which groups it composes from a single field registry. That is the whole difference between one library and 29 hand-written schemas that drift apart — and drift is not hypothetical: the configuration this replaces carries 250 distinct field names with no field common to all its document types.
Three tiers:
| Tier | Contents | Rule |
|---|---|---|
| Required | 9 fields: doc_id, tenant_id, created_at, schema_version, source, doc_type, version, status, extensions | Identical for every type and every tenant. All platform-generated, so a null here is a bug in us rather than a gap in the document. |
| Groups | identity (on all 29), money and terms (on the 14 that carry money) | Present-nullable: the key always exists, the value may be null. A nullable total on a bill of lading would teach integrators to distrust the field everywhere. |
| Extensions | Tenant fields, lower_snake_case, inside extensions | Additive only. A key that shadows a registry field is rejected rather than silently accepted. |
The system this replaces has an ignore_fields mechanism — 153 deletions across six live tenant configurations. That single capability is what makes a schema unusable as a contract: an integrator cannot rely on a field existing if any tenant can remove it. So applyExtensions has no counterpart, and the guard enforcing that is structural rather than behavioural — it reads the module source and fails if a deletion mechanism appears. A behavioural test could not catch it: adding an ignoreFields parameter left all 30 existing tests passing with the capability fully present.
The freeze. A record is a proposal until something independent corroborates it — a reconciliation against an accrual on the happy path, or a person. Never the model (§3). Approval names what certified it as a typed value, {type: 'rule'|'user', id}, so “how many invoices were approved by a machine rather than a person” is answered by filtering rather than by parsing prefixes out of strings. After approval a change produces a new version; the previous one stays retrievable.
Migration is dual-write. Every consumer today reads the older canonical shape and every stored extraction is written in it, so switching would break the readers and orphan the archive in one commit. Instead both shapes are emitted, the mapping is validated on every real extraction, and what the mapping drops is reported rather than silently discarded — because a translation layer that quietly loses data is indistinguishable from one that has a bug. Consumers migrate one at a time; the day the last one moves, the old shape is deleted rather than deprecated.
When a clerk corrects a missing field they already know something the system does not: where on the page it was. A pin captures that, so the next document from the same vendor can be told where to look.
Repair runs only for fields that are missing. Re-reading a field already extracted spends money to risk a worse answer.
The run answers what. Provenance answers why, and it is read at a completely different time, by a different person, for a different reason: an auditor months later asking why a particular invoice was coded one way and posted to a particular account. Nobody is standing at the screen and the run is long gone.
It is captured, not recomputed. Re-running the extraction to explain it would call the model again — and the model is the one step that need not return the same answer twice. An explanation produced that way would describe a run that never happened. So the record is written once, at the time, and stored as cold-path data with a different lifetime from the result itself.
A repair is recorded as its own section rather than appended as a step, because splicing a probabilistic step onto the end of a completed chain would make the record’s own replay claim false.
Findings become invariants. A fix repairs one occurrence; an invariant prevents the class — and specifically prevents it in the cartridges that have not had the bug yet, which a per-cartridge test never does.
| Invariant | What it prevents |
|---|---|
no-unguarded-invoke | A module invoking Bedrock without a guardrail. The prior rule caught only a conditional guardrail, so the one compliant module was flagged while ten unguarded call sites passed. |
no-guardrail-fallback | A conditional guardrail — an empty environment variable shipping an unguarded model. |
no-invisible-completion | A model call logged under a name the cost dashboard filters out. Repo-wide, so spend cannot go missing in a cartridge that has not made this mistake yet. |
| Import-shape test | A wrong import path in a Lambda entrypoint. This is not theoretical: one took every route to a 500 because the module never initialised. The test reads exports textually, so it never constructs an AWS client. |
| Model-list drift test | The allowed-model list exists in three places and only one decides — the backend environment variable. The original guard compared the two that do not. |
| Structural schema guard | A field-deletion mechanism reappearing in the contract (§4.3). |
Every guard here was verified by reintroducing the regression it claims to catch and confirming it fails. A guard that has never been seen to fail is a guard nobody has tested.
Recorded plainly, because a document that only lists what works is a sales sheet.
| Path | Role |
|---|---|
lambda/handler.js | 16 exact-path routes plus parameterised sub-resources (keys, jobs, runs, benchmark), across two audiences: the Console surface and the tenant-scoped /api/v1/* partner API. |
lambda/worker.js | The second entrypoint — batch and enrichment processing off the queue. |
lambda/lib/extract-unified.js | The one-pass approaches and the canonical normalizer. |
lambda/lib/extract-approaches.js | sequential-typed and parallel-raw. |
lambda/lib/extract-shared.js | The shared Bedrock call: guardrail application, model-family translation, cost accounting. |
lambda/lib/field-registry.js | The three tiers — required fields, groups, status. |
lambda/lib/doc-schema.js | 29 type compositions, validation, extensions, approval, versioning. |
lambda/lib/contract-map.js | Canonical → contract translation, with what it drops reported. |
lambda/lib/pin-map.js · pin-store.js · pin-repair.js · pin-repair-run.js | The learning layer: geometry, persistence, planning, and the orchestrated repair pass. |
lambda/lib/provenance.js | The captured explanation. |
lambda/lib/prompt-cms.js · prompts.js | CMS-served prompts with the bundled constant as an unconditional floor. |
lambda/lib/apikeys.js | Tenant-scoped M2M keys, hashed at rest. |
scripts/arch-lint.mjs | The 10 deterministic invariants (§7). Repo-wide, not cartridge-local. |