Most of what we know about AI's effect on the economy is measured from the outside. Population-scale efforts — the Anthropic Economic Index is the best example — read aggregate usage and infer where AI is touching work. That's the right altitude for the macro question, and it's genuinely hard, careful work.
But there's a layer it can't see directly: the firm. The decision to automate a task, to not backfill a role, to create net-new work that didn't exist before — those happen inside a company, in its operational systems, not in the aggregate. If you want to measure AI's economic impact at the level where the decisions are actually made, you have to instrument the firm itself.
For the past two years I've been doing exactly that, by accident first and then on purpose. I run a venture studio whose agent systems operate live functions — finance, compliance, reconciliation, reporting — inside real small businesses. Running those systems means I generate a continuous, structured record of which tasks an AI now performs, how well, and at what cost. This post is the architecture of the pipeline that turns that exhaust into clean, analyzable, firm-level data.
It's three layers: ingest and reconcile the messy operational streams, extract and classify with LLMs, and — the part most people skip — evaluate the output so the data is trustworthy enough to actually use.
The shape of the problem
A small business does not emit clean data. It emits a point-of-sale export with one schema, an accounting system with another, a payments processor with a third, and a membership platform with a fourth — none of which agree on what a "transaction" is, what period it belongs to, or how a refund is represented. Before you can measure anything, you have to make these streams agree.
POS export ─┐
Accounting ─┤
Payments ──┼─► normalize ──► reconcile ──► canonical ledger
Membership ─┘ (cross-source (one row =
matching) one economic event)
The reconciliation step is where most of the real engineering lives, and it's where the first interesting measurement signal shows up. A concrete example: a revenue stream was being systematically misreported because two source systems disagreed on the service period a payment belonged to — one booked on charge date, the other on the period the membership covered. The aggregate looked plausible, so nothing flagged it. The fix was a filtering layer that derived the correct period boundary and recovered the true signal from imperfect data.
That's the unglamorous heart of firm-level measurement: the truth is recoverable, but only after you reconcile sources that were never designed to be compared.
Layer 2 — LLM extraction and classification, in batch
Once events are reconciled, much of the interesting structure is still locked in unstructured fields — line-item descriptions, document images, free-text notes. This is where LLMs earn their place: extracting structured records from heterogeneous inputs (invoices, bank statements, POS and membership logs) and classifying each into an economic taxonomy.
Two engineering decisions matter here:
- Batch, not synchronous. Measurement is not latency-sensitive — nobody needs the classification in 200ms. Running extraction through a frontier provider's batch API costs roughly half the synchronous cost, with the tradeoff of asynchronous completion. For a measurement pipeline that's a free lunch.
- Typed outputs. Every extraction returns a strict schema, not prose.
from dataclasses import dataclass
from datetime import date
@dataclass(frozen=True)
class EconomicEvent:
source: str # which operational system emitted it
event_type: str # 'sale' | 'refund' | 'subscription' | ...
amount_cents: int # integers only; never float money
service_period: tuple[date, date]
task_class: str # the automation taxonomy label
confidence: float # model's self-reported extraction confidenceIntegers for money, explicit service periods, and a task_class that ties each event back to the function that produced it. That task_class is the bridge to the actual research question.
Layer 3 — the part everyone skips: evaluation
Here's the claim that makes or breaks a measurement pipeline: LLM-extracted data is only as trustworthy as your ability to catch the model when it's wrong. A classifier that's 90% right and silently 10% wrong will quietly poison any downstream economic estimate. So before any of this output becomes "data," it passes through a closed-loop evaluation harness I built for exactly this failure mode.
The loop is: generate → execute → judge → converge.
NLSpec ──► generate ──► execute on held-out scenarios ──► judge ──► converge?
▲ │ │
└──────────────── feedback ◄────────────────┘ └─► ship
The judging is a three-layer stack, cheapest-first:
def evaluate(output, scenario) -> Verdict:
# 1) deterministic assertions — structural truths, free to check
if not deterministic_checks(output, scenario):
return Verdict(passed=False, layer="assertion")
# 2) behavioral scenario suites — does it do the right thing on held-out cases
if not scenario_suite(output, scenario):
return Verdict(passed=False, layer="scenario")
# 3) LLM-as-judge — graded satisfaction vs the spec, last + most expensive
score = satisfaction_judge(output, scenario) # 0.0–1.0, model-graded
return Verdict(passed=score >= scenario.threshold, layer="judge", score=score)Deterministic assertions run first because they're free and catch the dumb failures (wrong types, impossible values, dropped fields). Behavioral scenario suites check that the output does the right thing on held-out cases. Only what survives both reaches the LLM judge, which grades satisfaction against the specification. Each domain carries its own critical-correctness threshold — the bar below which output is rejected outright — and per-run cost tracking so evaluation never silently outspends the work it's checking.
Across five distinct business domains, this harness reached 89–97% validated satisfaction convergence on held-out suites, typically in one to two iterations. The number that matters isn't the headline accuracy — it's that the system knows when it's below the bar and refuses to ship, which is the only thing that makes the resulting dataset usable for measurement rather than anecdote.
What this produces: an operator-side capability metric
Stack the three layers and you get something more than clean books. Every reconciled, classified, evaluated event is tagged with the task that produced it and how reliably the AI handled it. Aggregate that over time and you have a task-level automation-maturity metric over real firm activity — a measure of which functions an AI has actually taken over inside a specific business, observed rather than surveyed.
This is what I mean by firm-level observed exposure. Population-level indices tell you AI usage is rising in, say, bookkeeping. A firm-level pipeline tells you which specific bookkeeping tasks crossed the reliability threshold to be left to the machine, when, and what happened to the work around them — backfilled, restructured, or newly created. It's the same observed-exposure idea the macro work uses, pushed down to the level where employment decisions are actually made.
Limitations (so nobody mistakes this for more than it is)
- n is small. This instruments a handful of small firms, not a representative sample. It's a method and an existence proof, not a population estimate.
- Selection is severe. These are firms that already adopted heavy automation — by construction, the leading edge, not the median.
- The metric is young. "Automation maturity" is a candidate construct that needs validation against outcomes I can't yet observe at scale.
What it is: a working demonstration that the firm-level layer can be instrumented, that the data can be made trustworthy through evaluation rather than hope, and that the architecture generalizes across domains. The interesting frontier is connecting this bottom-up signal to the top-down indices — using firm-level mechanism to explain population-level movement.
That's the measurement infrastructure I want to help build. If you're working on the same problem from either end, I'd like to hear from you.