My model notes became useful after I stopped looking for one winner and started recording which mistakes each workflow could survive.
The first pass looked like a familiar evaluation exercise. Give several models the same inputs, inspect the outputs, and decide which one performed best. It produced opinions quickly, which should've made me suspicious. One model wrote with more ease. Another followed a schema reliably. A third appeared stronger on code. The phrase “best model” began showing up in my notes before the tests had earned it.
I reset the log around tasks and consequences.
Drafting went first because it's forgiving in a specific way. An awkward paragraph is visible to the reviewer. A weak outline can be discarded. The output doesn't become true merely because it exists. For that route, latency, useful coverage, and revision effort mattered more than perfect factual precision on the first attempt, provided the workflow preserved sources and kept the result in draft state.
The interesting failure was bland competence. Some outputs were clean enough to pass a quick glance but required substantial rewriting because they had flattened the point. Others made a sharper start and occasionally wandered. A single preference score hid that trade. The former created steady editing work; the latter created a smaller number of obvious rejections. Which failure was better depended on whether a reviewer would actually read closely.
Next I tested extraction from known material into a constrained structure. Here, fluent prose had no value. I recorded missing fields, unsupported fields, malformed structure, and explicit abstentions separately. A model that returned a complete object every time initially looked productive. Its unsupported values were expensive because they resembled successful extraction. Another left more fields unresolved but made the gaps easy to route.
That changed the candidate for extraction. I would rather send a visible missing field into review than let an invented value travel downstream under a valid key. The evaluation had to run through the parser and validation path, not stop at a chat response. Text that looked correct could still fail the consumer. Valid structure could still contain claims with no anchor in the input.
Code change evaluation forced another reset. A model can produce a plausible patch, explain it well, and pass a narrow test while misunderstanding the surrounding behavior. I separated mechanical edits from changes that required architectural judgment. Repetitive transformations benefited from consistency and strong instruction following. Changes crossing ownership, authentication, or persistence boundaries required broader inspection no matter which model drafted them.
The failure categories became concrete: patch doesn't apply, build fails, targeted test fails, change exceeds scope, existing behavior regresses, or explanation claims proof that was never run. I also noted false confidence. A hesitant explanation attached to a correct diff is inconvenient. A polished declaration that “all tests pass” when no tests ran is an operating hazard.
No aggregate code score captured that distinction. Routing did. A model that was efficient for bounded edits could prepare a candidate and evidence request. A more capable model could analyze a wider change. Neither route authorized acceptance; the delivered candidate still needed the relevant build, tests, and review.
High-consequence judgment was the shortest section of the experiment and the hardest to score. These were tasks where the output could influence access, an external communication, production state, or another consequential decision. I wasn't looking for the model that sounded most decisive. I looked for source anchoring, recognition of missing context, handling of conflict, and willingness to leave the decision unresolved.
Some models improved their apparent completion rate by resolving ambiguity silently. Others surfaced uncertainty so often that a person had to revisit routine cases. The safe-looking system could become unusable; the productive-looking system could convert unknowns into assertions. That's the tradeoff I still haven't reduced to a satisfying threshold.
False confidence mattered across every task, but its cost changed. In drafting, it creates editing. In extraction, it can contaminate a structured record. In code, it can misstate the evidence behind a candidate. In high-consequence judgment, it can steer a person toward an action that the sources don't support. Treating all four as one quality problem would guarantee the wrong routing policy.
I ended up with task cards rather than a leaderboard. Each card names the input shape, expected output, validation available, common failure classes, cost of an unsupported answer, and the route when confidence is insufficient. Model results attach to those cards. When a model version changes, I can rerun the relevant tasks without pretending its drafting behavior predicts its extraction behavior.
The log also records where a cheaper or faster model is adequate. Routine formatting and constrained extraction don't automatically deserve the most capable option. Likewise, high-consequence review doesn't become safe merely because an expensive model handled it. Capability changes the candidate set; evidence and workflow controls determine acceptance.
There is still an unresolved edge between useful abstention and excessive deferral. Raising the bar reduces unsupported automatic output but pushes more work to people, who can become hurried and inconsistent when the queue grows. Lowering it increases coverage and creates quieter errors. I can measure both sides, but the acceptable balance remains a workflow decision, not a property I expect the next model release to settle for me.
The current routing notes don't name a champion. They tell me where each model can help, which mistake it tends to make there, and what should catch that mistake. I still haven't settled the line between useful abstention and a review queue that people rush through. For now, that unresolved line stays in the log instead of disappearing into a score.