The Scaffolding Tax: Investigating the Optimal Harness for a Model

Agent building has entered a new era. New models arrive at a relentless pace, each bringing different capabilities, behaviors, and failure modes. For teams building production agents, adopting a new model is not simply a matter of changing an API endpoint. The harder question is: how should the harness evolve with the model?

At NeoSigma, a central question in our research is what is an optimal harness for a given model and use case. The answer is not straightforward. Models with similar levels of intelligence can respond differently to prompts, tools, context, and workflow constraints.

Each model is inherently different, even when models share the same intelligence or model family. Behavioral evaluations have suggested this for some time. OpenAI reports that GPT-5 is substantially less sycophantic than GPT-4o, making it more likely to challenge a user’s assumptions. Similarly, Anthropic found that even though Opus 4 was a more powerful model, it exhibited more misaligned behavior in red-teaming scenarios than Sonnet 4.

AI models, even within the same company or model family, can differ in behavioral tendencies, failure modes, and sensitivity to prompting. As a result, harnesses need to adapt to the respective model. The future of building AI agents requires understanding the relationship between model and harness for specific tasks.

A model, however, never acts alone. Every production agent sits inside a harness: the execution layer that extends a model with tools, MCP servers, prompts, subagents, retry logic, verification logic, context management, and more. Harness builds the execution layer that determines how effectively a frontier model can translate raw intelligence into sustained execution.

As agents take on longer-horizon tasks, this surrounding system can amplify a model’s capabilities.

But it can also suppress them. As models get stronger, they require less scaffolding. Anthropic engineers describe regularly removing old system prompts and tooling to test what the newest model is capable of. Boris Cherny, later confirmed that old rules and prompting can actually harm the quality of the agent.

When a new, capable model is ported to a stale harness, the harness can misguide the model with unnecessary context, irrelevant tooling, and overcomplicated procedures. In other words, there is a loss in performance with an older harness even with a smarter model. This is what we call the scaffolding tax.

Setup. We tested how harness performance changes across models and reasoning-effort levels using difficult Terminal-Bench 2.0 tasks. The tasks span document processing, code security, scientific computing, distributed systems, and technical formatting. We selected tasks that no model could solve reliably, ensuring that performance differences would remain visible. Each condition was run at least 10 times while either model capability or reasoning effort was varied and all other experimental conditions were held constant.

Harness Performance as Model Capability Increases:

We evaluated how Claude Code, Codex, and Pi perform as their underlying models become more capable, while keeping each harness fixed. Codex and Pi were tested with increasingly capable GPT models, while Claude Code was tested with a corresponding sequence of Claude models.

We observe divergent performance trends across agent harnesses. Codex and Claude Code improve reliably with increasing model capability: every stronger model translates into better task performance. Pi, however, shows an interesting behavior. Even when paired with a substantially stronger model with GPT-5.6 Sol, its performance regresses.

We argue that this difference is a direct consequence of model-harness optimization. OpenAI and Anthropic continuously adapt their corresponding harnesses Codex and Claude Code to the capabilities and behavior of their latest models. Though there is no specific information on how Claude Code does, Codex specifically uses per-model harness prompt files. This is one simple way these companies optimize agent performance.

Pi, on the other hand, being a largely fixed open-source harness, does not benefit from a stronger and more capable model. Its scaffolding may therefore be less well suited to increasingly capable models, creating a scaffolding tax that can offset - or even reverse - the gains expected from greater model intelligence.

This makes continuous, model-specific harness testing and optimization essential.

Harness Performance as Reasoning Effort Increases

We next tested how a harness performs when the model remains fixed but receives more reasoning time. Reasoning effort increases the amount of test-time compute available to a model and is generally expected to improve performance. Using the Pi harness, we evaluated GPT-5.1, GPT-5.6 Sol, Sonnet 4.6, and Haiku 4.5 at medium, high, and xhigh effort. For each model, we held the harness and benchmark tasks constant while varying only the effort level. Each condition was run at least 10 times.

Performance remained broadly stable as reasoning effort increased. Although Sonnet 4.6 declined slightly from high to xhigh effort, the results did not imply evidence of an effort-related scaffolding tax.

This suggests that changing the underlying model produces much larger performance differences than increasing the reasoning effort of a fixed model.

The natural next question is how to build effective harnesses. The goal is to preserve fluid intelligence: allowing a model to adapt, reason, and use its capabilities without being constrained by rigid or outdated scaffolding.

Today's models are far more capable than you think. AI has disproved a long-standing conjecture in combinatorial geometry, disproved a decades-old math conjecture, found hundreds of zero-day bugs, and as of recently found new proofs for high-dimensional sphere packing and the closest vector problem, along with eight others. It therefore makes no sense to treat each new model the way you did the last. If intelligence is exponential, the mental model of yesterday no longer works today. A harness optimized for one model may constrain another or fail to provide the support it needs. Switching models should therefore be treated as a new empirical question, not a simple infrastructure substitution.

Harness engineering should look like the scientific process. Pushing the limits of an agent, observing what it can do, updating the mental model, and forming a hypothesis for what will improve the agent.

Model–harness co-design begins by evaluating each model independently on tasks that reflect the real user workflows. As we argued in The Most Important Eval Isn’t on a Leaderboard, public benchmarks are only proxies; the most valuable evaluations come from production, where real users, edge cases, and domain-specific expectations define success.

These evaluations should measure more than overall accuracy. They should reveal where the model succeeds, how it fails, which tools and context it uses effectively, and where additional structure improves completion. These findings can then guide model-specific decisions about prompts, skills, tools, context management, retries, verification, and other harness infrastructure.

The result is not one universal harness. It is a continuously evolving set of model and use-case-specific configurations, informed by empirical data and optimized for the user tasks. As models, workflows, and user expectations change, the harness must be reevaluated with them. The goal is to give each model the specific environment it needs to deliver its strongest performance for the use case at hand.

As AI models become more capable, building reliable production agents increasingly requires harness-level optimization.

Designing a harness means shaping the execution environment around the capabilities of a specific model and the requirements of a specific use case. As model capabilities grow, user expectations rise with them: tasks become longer, workflows become more complex, and successful completion demands greater reliability. The harness must therefore evolve continuously alongside the model, and the changing definition of success for the specific tasks.

At NeoSigma, we extend this empirical research to every company and every agent. By learning from production data and systematically optimizing the relationship between models and harnesses, we are building the infrastructure for continually improving agentic systems.