In our last post, Finance as Code, we made a case: represent a financial model the way software represents a program — structure, validation, evaluation loops, a harness of agents that find, fix, and verify — and everything that makes AI exceptional at coding carries over to finance. We backed it with a benchmark. Holding the model fixed and varying the regime, the full system lifted quality far above a frontier model working on a spreadsheet alone.
That raises the obvious next question, and it's the one we get asked most: which model should build the model? Frontier labs ship a new state-of-the-art every few weeks. If the generation model is what matters, we should be chasing the leaderboard. If the system is what matters, the model underneath is a swappable part. So we ran the experiment the other way around: hold the whole system fixed, and change only the model that writes the first draft.
The setup: one variable
We took three frontier models from three different labs — Google's Gemini 3.1 Pro (what we run
in production today), Anthropic's Claude Opus 4.8, and OpenAI's GPT-5.2 — and pointed each of
them at the identical pipeline. Same three companies as before: a luxury fashion house, a global
healthcare company, an EV-charging network. Same source documents, same extraction, same
validators, same refine → analyst → remediate → verify harness, same definition of
done. We reuse the exact same benchmark, rubric, and independent judge from the first post, so
the numbers are directly comparable. The only thing that changes between the three runs
is which model authors the DSL spec.
And, as before, we look at two points in each build: the single pass — the spec the model hands over before the agent loops touch it — and the full harness — the delivered model after refinement, analysis, remediation, and independent verification. That pairing is what isolates the model's raw output from the system's contribution, per model.
What we found
Three things, and none of them is "chase the leaderboard."
Gemini 3.1 Pro (our production default)
Claude Opus 4.8
GPT-5.2
Methodology: identical to the "Finance as Code" benchmark, with one variable — the model authoring the DSL spec. Three companies, same serialized source data and same pipeline across all three models; each artifact is rendered through our calculation engine into computed values plus formulas — the same form for every model — and scored by a single independent frontier judge (OpenAI's GPT-5.2, from a different lab than either the generation model or the Claude-based harness) against 21 binary criteria spanning historical fidelity, completeness, internal consistency, driver logic, and usability/traceability, majority vote over three samples, one deterministic (the model executes in a calculation engine). Criteria are severity-weighted by decision-grade importance, applied uniformly across all three models. The independent judge shows no home-lab preference — it rates Gemini's output highest and, on its own lab's model, GPT's lowest. As a robustness check we re-scored everything with a three-model judge panel (one from each lab); the ranking is unchanged. Judge-based scores on three companies are directional evidence, not a leaderboard.
First: the model matters, but the system matters more. After the harness, the three frontier models sit in a thirteen-point band, from 69% to 82%. That is a far smaller spread than the one we reported last time between running the system and not running it, where the gap from a model working alone to the full system was more than fifty points. Put plainly: which frontier model you pick moves the needle much less than whether you wrap it in the system at all. The choice of lab is a tuning decision; the system is the product.
Second: the system lifts every model — but not by the same amount. Each of the three delivered models scores higher than its own single pass: the harness added 27 points to Gemini, 12 to Claude, and 9 to GPT. So the agent loops earn their place regardless of which model drafts the model. But the lift is far from uniform, and that's the interesting part. The harness got the most out of Gemini — whose single pass was middling but clean, giving the analyst and remediation agents good raw material to build on — and the least out of GPT-5.2, which generates dramatically more elaborate models (roughly twice the line items of the other two, around a hundred named metrics versus fifty to seventy). More model is more surface to audit, and GPT's delivered builds kept slipping on exactly the criteria that reward auditability: whether a spot-checked formula visibly reproduces the value it displays, whether a reader can trace every number, whether there are any unexplained hardcoded constants. A bigger, busier model isn't automatically a better one once a system is doing the checking.
Third: the production default is the strongest, and that's not an accident of who's judging. Gemini 3.1 Pro — the model we actually ship — posts the top delivered score by a clear margin, 82% against 70% and 69% for Claude and GPT. It's worth being skeptical of that result, so note two things. The judge is OpenAI's model, which has no reason to favor Google; it rates Gemini's output highest and its own lab's model lowest. And the gap traces to concrete, checkable criteria — formula-level fidelity, provenance, no-plugs — not to style. The reassuring read for us is that the system isn't quietly propping up a weak generator: our model choice stands on its own. The strategic read is better still — because the pipeline treats the generator as a component, we can adopt a stronger base model the day one clearly pulls ahead, without rebuilding anything.
Why the differences compress
It's worth being precise about the mechanism, because "the system matters more" can sound like hand-waving. It isn't. Every model, whatever its style, hands its draft to the same deterministic validators — the same compiler and linter — that reject a balance sheet that doesn't balance, a sign convention that's inverted, a reference that doesn't resolve. Every model's draft is then read cold by the same analyst agent, its findings applied by the same remediation agent, and the result re-read by an independent verifier that is never shown the fixes it's checking. Whatever a given model is weak at, the system is engineered to catch and correct — so the models converge toward the definition of done rather than toward each other's idiosyncrasies.
That convergence is the whole point of representing the model as code. On a grid of anonymous cells, each model's mistakes would be different flavors of the same invisible problem — numbers that look fine until someone audits them by hand. As code, those mistakes are the same kind of thing regardless of which model made them: a failing check, a flagged finding, a criterion the verifier won't sign off on. The system doesn't need to know in advance how a particular model tends to fail. It just needs the model to be code, and then it can find, fix, and verify the same way every time.
What this means for us — and for you
We take three things from this. Our current model choice is validated by the data, not by loyalty. We are insulated from the leaderboard churn that dominates AI right now — a new state-of-the-art model is an upgrade we can adopt, not a rewrite we're forced into. And the engineering leverage sits where we've been putting it: in the validators, the agent loops, and the evaluation harness, not in betting on a single lab.
The honest headline isn't that one model builds the best financial model. It's that the question is slightly wrong. A frontier model is a superb first-drafter and an unreliable finisher, and that's true across labs — every one of them scored higher after the harness than before it. What turns a good draft into a model you can stake a decision on is the system around it: the structure that makes defects visible, the checks that make them catchable, and the loops that make them rare. Pick a strong model, by all means. Then put it inside something that treats its output like code.