The hardest part of putting AI into a business is rarely the model. The model is a commodity now, and a good one. The hard part is everything around it: knowing what the business actually does, deciding what's worth automating, proving the automation is right, and making it survive the person who built it leaving the room.
I run a venture studio where agent systems operate live functions, finance, compliance, reconciliation, reporting, inside real small businesses. Not pilots. The same close, every month, and it has to survive an audit, not just a demo. Doing that repeatedly, across companies, forced the method into a shape I can hand to someone else. That shape is this playbook.
It's five phases. Each phase produces a specific artifact, has specific ways it goes wrong, and ends in a gate: a condition you either meet or you don't, and if you don't, you stop.
The gates are the playbook. Everything else is technique.
An undisciplined team with great tools will blow through every one of them and end up with an impressive system that's confidently wrong. A disciplined team gets something rarer: an operation where the right outcome happens by default.
One note on scope before the phases. This is the same method whether you run it once inside one company or repeatedly across a portfolio. What changes at portfolio scale isn't the method, it's the marginal cost, and I'll come back to that at the end.
Phase 1 — Diagnose: establish what actually happened
What happens. Before anything gets automated, you find out what the business actually does, as opposed to what the org chart, the process docs, or the loudest person in the room says it does. Two moves, in parallel:
First, the human layer. Sit with the people who've lived each process longest, not the most senior, the most tenured in the pain. Map what's actually done versus what's perceived as done. In every operational turnaround I've run, the gap between those two is where the money is.
Second, the data layer. Pull the actual exports from the actual systems. A small business does not emit clean data. It emits a point-of-sale export with one schema, an accounting system with another, a payments processor with a third, and none of them agree on what a transaction is or what period it belongs to. You reconcile those streams until they agree on what happened last period. That reconciled record is your baseline, and it is the single most valuable thing this phase produces, because every claim you make later ("the AI saved X hours," "error rate fell") is measured against it.
Artifact. A map of what actually happens: every in-scope process, who owns it, which system holds its record, and where each number is derived versus merely asserted. Plus the reconciled baseline for the most recent complete period.
What goes wrong. The classic failure is that the aggregate looks plausible, so nobody digs. I've watched a revenue stream get systematically misreported because two source systems disagreed on which service period a payment belonged to, one booked on charge date, the other on the period the payment covered. Totals looked fine. The books were wrong every single month. Nothing flagged it because plausible is not the same as reconciled. The other failure is auditing the org chart instead of the operation: interviewing managers about a process their team quietly abandoned a year ago.
Gate. The baseline reconciles. You can state, with independent sources agreeing, what the business actually did last period. If the sources don't agree, you have not finished diagnosing, and nothing downstream is worth building yet. Automating on top of an unreconciled baseline just makes the wrong numbers arrive faster.
Phase 2 — Roadmap: rank by value, gate by verifiability
What happens. From that map, you now have a list of candidate processes. Rank them on two axes: value and feasibility. Value in operator terms first, hours recovered, error rates, days-to-close, cash visibility, because those are the terms the people inside the business live in, and they translate cleanly upward into the terms an investment committee lives in: margin, operating leverage, what the company is worth with this cost structure versus without it.
Feasibility has one test that outranks all others: can the output be verified against the reconciled baseline? A process with a checkable right answer (does the ledger tie out, did the filing go in on time, does the extracted invoice match the source document) can be automated safely. A process where correctness is a matter of taste cannot be gated, and what cannot be gated should not run unattended. This is why the boring, frequent, verifiable work goes first, and the impressive demo goes last or never.
And rigor gets right-sized to the org. A ten-person company does not need the control framework of a money-center bank; it needs the two or three controls that actually catch its failure modes. Bringing bank-grade ceremony to a small firm is driving a Ferrari in a school zone. Bringing the discipline without the ceremony is the actual skill.
Artifact. A sequenced roadmap where every item carries, on one page: the baseline (from Phase 1), the target, the verification method, and the kill criteria, the observable condition under which you stop and roll back.
What goes wrong. Three ways. Picking the demo-friendly project over the frequent one, so nothing compounds. Roadmapping a task with no verifiable right answer, which guarantees a Phase 4 you can't pass. And scoping around a specific person's job instead of the process, so the plan quietly depends on that person's undocumented judgment, which is the exact disease you were hired to cure.
Gate. Every roadmap item has a written definition of correct before anything is built. If you can't write down how you'd know the output is right, the item doesn't advance. No exceptions for exciting ideas.
Phase 3 — Deploy: embed in the live process, supervised
What happens. The agents go into the real process, not a sandbox next to it. Real system access through scoped credentials, real data pipelines, outputs in strict typed schemas rather than prose, integers for money, never floats. Where the work isn't latency-sensitive, and measurement and back-office work almost never is, it runs in batch, which cuts cost roughly in half for the same result. Every action the system takes is logged and attributable, because in Phase 4 you'll need to audit not just what it concluded but what it did.
Deployment is supervised by default. The system does the work; a human watches the work, on real periods, until the evidence says otherwise.
Artifact. The working system, plus its paper trail: a complete, queryable record of every action taken, by what, on whose authority, against which source data.
What goes wrong. The deployment failures are less exotic than people expect. Two writers who each believe they own the same record or file, and silently overwrite each other, is a real and recurring one; the fix is boring and absolute: one owner per artifact, everyone else reads. Credential scope creep is another: give a system broad permissions for convenience and you've built a loaded gun into your back office. The discipline is minimal scopes, set so the destructive action is structurally impossible, not merely forbidden by a prompt. And the subtlest failure: replacing a person's judgment before you've captured it, so the knowledge walks out the door while the automation is still learning what the job was.
Gate. One full cycle of the real work, on a real period, end-to-end, watched. The first run is a supervised dry run, never an auto-commit. A system that hasn't survived a watched live cycle has no business running an unwatched one.
Phase 4 — Evaluate: catch the model when it's wrong
What happens. This is the phase everyone skips, and it's the one that separates a system you can bet operations on from a demo. The claim I build everything around: AI-produced output is only as trustworthy as your ability to catch the model when it's wrong. A classifier that's 90% right and silently 10% wrong will quietly poison every downstream number, and the aggregate will look plausible the whole time.
So output passes through a closed evaluation loop, generate, execute, judge, converge, before it counts as done. The judging is a three-layer stack, cheapest first:
- Deterministic assertions. Structural truths that are free to check: types, impossible values, dropped fields, sums that must tie. These catch the dumb failures instantly.
- Behavioral scenario suites. Does the output do the right thing on held-out cases it hasn't seen, including the ugly ones: refunds, partial periods, duplicates.
- Model-graded judgment. Last and most expensive, a judge grades the output against the specification, with a critical-correctness threshold per domain, the bar below which output is rejected outright, and per-run cost tracking so evaluation never silently outspends the work it's checking.
In my own systems this harness has reached 89 to 97 percent validated satisfaction on held-out suites across five distinct business domains, typically converging in one to two iterations. But the headline number isn't the point. The point is that the system knows when it's below the bar and refuses to ship. Every number defaults to untrustworthy until verification flips it, and the burden of proof sits on the machine, not the reviewer.
Artifact. The evaluation harness itself, plus the convergence record: which suites, what thresholds, where it failed, what changed, where it landed.
What goes wrong. Evals that only test the happy path, so the system aces the demo and dies on the first refund. Grading against cases the builder cherry-picked instead of held-out scenarios. And skipping the layered structure so every check runs at the expensive layer, at which point evaluation gets cut for cost reasons, which is how the silent 10% gets back in.
Gate. Documented convergence above the domain's threshold on held-out cases, and a demonstrated refusal: you have seen the system decline to ship substandard output on its own. Until you've watched it say no, you don't know that it can.
Phase 5 — Operate: make it survive people
What happens. A system that works is not yet a system that lasts. Operating means runbooks a non-builder can execute, scheduled runs that don't need a human to remember them, and exception-first reporting: the operator's attention goes to what broke or got skipped, not to a wall of green checkmarks.
Two operating rules carry most of the weight. Loud blockers: the system never declares work complete with an unverified stream inside it; an unfinished close announces itself. No silent truncation: anything skipped is reported as skipped, with a reason, so a partial run can never impersonate a complete one. Coverage stays honest even when honesty is unflattering.
Then there's the ladder, which is where this stops being a project and becomes an operating capability. The same engine climbs four rungs: run it by hand for one company; parameterize it so any company is one command; schedule it so it runs without you; then run the whole registry, every company, in one pass, exceptions first. Each rung drops the marginal cost of the next company. By the top rung, adding a business to the operation is a registry entry, not a rebuild. For a portfolio operator, that ladder is the thesis: the method costs real effort once and near-zero the Nth time.
Artifact. The runbook, the schedule, and the exception report that lands on a cadence, readable by someone who has never seen the code.
What goes wrong. Silent degradation: a source system changes an export format and accuracy erodes for months, which is why the Phase 4 gates keep running in production, forever, not just at launch. And the failure I watch for hardest: the handoff that never happens. If only the builder can run it, you haven't removed the exceptional employee, you've just changed which one the company depends on. That's not an upgrade. That's the same disease with better tooling.
Gate. Someone who didn't build the system runs a full cycle from the runbook, and the system reports its own gaps loudly while they do it. Pass that, and you have an operation. Fail it, and you have a very sophisticated dependency.
Why the gates matter more than the tools
Every phase of this playbook is aimed at the same enemy, and it isn't bad technology. It's dependence on the exceptional: the business that runs because one founder holds it in their head, one bookkeeper knows where the bodies are buried, one engineer is the only person who can restart the pipeline. The exceptional person is inconsistent, gets tired, and leaves. And the risk never sends you a bill with its own name on it; it invoices you as a bad hire, a rough quarter, books nobody can read because three bookkeepers in three years each did it differently.
AI does not fix that. AI amplifies whatever you've already got, your discipline, or your drift. Drop capable agents into an undisciplined operation and you get bad decisions at machine speed. Run them through gates like these and you get the other thing, the thing this whole playbook exists to produce:
Operational discipline is making the right outcome the default, so it doesn't depend on heroics, memory, or whoever happens to be in the room.
Don't build a company that depends on exceptional people. Build one where the right thing is the default. The playbook is just that sentence, made procedural.
If you're running this problem from the portfolio side or from inside the firm, I'd like to compare notes.