At NeoSigma, we build systems for self-improving agents, and one question we have begun researching is what the optimal harness is for a specific AI model. This is a tricky question. For one, each model has strikingly different behaviors even if the intelligence is similar.
Behavioral evaluations have suggested this for some time. OpenAI reports that GPT-5 Thinking is substantially less sycophantic than GPT-4o, while Anthropic found Claude Sonnet 4 more robustly aligned than Opus 4 on the alignment evaluations it ran, even though Opus 4 was generally the more capable model and exhibited more concerning behavior in certain highly adversarial, artificial scenarios. The broader implication is that models, even within the same company or model family, can differ in behavioral tendencies, failure modes, and sensitivity to prompting. The implication is that different models, even from the same company or model family, act differently, and thus harnesses need to adapt to each model.
Moreover, harness engineering is difficult. Optimizing which hooks, tools, and skills to use is a non-deterministic question that is challenging to evaluate. When the action and state space of an agent is so large, how do you know what is optimal?
This motivates research to find the optimal harness for a specific model. Our hypothesis is that there is an optimal point of intelligence for a specific harness, and vice versa. We can test this by varying models, effort levels, and harnesses, fixing two of the dimensions while varying the other. We open-source our GitHub repo for reproducibility and experimentation.
We test agents on Terminal-Bench 2 using Claude Code, Codex, and Pi, and select tasks that are not fully saturated by models and relatively difficult.
The ablation. We fix two of {model, harness, effort} and vary the third, scoring on Terminal-Bench 2.0 in an E2B sandbox. Because different harnesses scaffold models differently, ladders from different harnesses sit side by side but are not cross-comparable. Compare the shape of each curve, not the levels between them.
Our first experiment alters the model and uses the same harness and effort, where the x-axis is the model and the y-axis is the score on Terminal-Bench.
Aggregate Terminal-Bench score by harness along a shared weak-to-strong model axis. codex and claude-code climb monotonically; pi flattens and ticks down at the top rung. Different families and harnesses share the axis, so compare the shape of each line, not the level between them.
Since we use different models between harnesses, we keep the x-axis simple, with the stronger side containing models such as GPT-5.6 Sol and Opus 4.8 and the weaker side containing models such as Haiku 4.5 and GPT-5.1 mini.
We run each experiment 10+ times over several Terminal-Bench tasks. We observe that the Pi graph is non-monotonic whereas Codex and Claude-Code are monotonic. While we can argue a performance drop most strongly between Pi and Codex since we use the same model families, other experiments we ran suggested similar drop-offs with Claude as well. In all, we argue that Pi — unsurprisingly, given its open-source nature and simple base harness — creates a performance gap because of its harness. While this gap is minimal, it is a strong argument for consistent testing since performance was saturated earlier.
Optimally, all harnesses exhibit monotonic behavior on their performance when changing the model. Given that Anthropic stripped more than 80% of their system prompt for Claude Code because of Opus 5, we suspect that both Anthropic and OpenAI conduct far more regular harness optimizations and tests than other agent companies today.
Next, we see the impact effort might have on performance. We fix our harness using Pi and use a range of models with different effort levels.
Task score against reasoning effort on the pi harness, medium through xhigh. All four models stay flat within the noise. Sonnet's low xhigh point is a single timed-out run, not a decline.
Effort is far more consistent in our experiments, with minor fluctuations. While we see one drop-off with Sonnet 4.6, we do not claim here that there is a large scaffolding tax between effort and the harness.
The Scaffolding Tax
Harness engineering is a quickly evolving practice, and our initial research elucidates the performance gains or loss companies can incur with a more expensive model and an under-tested harness. And because each new model has its own characteristics and emergent behavior, optimizing harnesses cannot be done incrementally as if building on top of the harness of the previous model. Optimization should happen every time a company adopts a new model for their agents.
Anthropic is an example of what this type of work looks like. The AI lab continuously probes and studies every new model's characteristics, removes entire system prompts and harness tooling, and challenges the model on new capabilities.
NeoSigma wants to extend this type of empirical research to every company and any agent. By working with production data and our customers' harnesses, we aim to democratize easy and continuous optimization at the harness level.
Improving Harnesses
The natural next question is how to build effective harnesses. Our answer starts from the nature of the problem. Agent systems solve non-deterministic problems, so the way we think about agents has to be non-deterministic too. What we want is fluid intelligence: intelligence that stays adaptive and avoids being pinned down by rigid systems and scaffolding. Three ideas help unlock it.
Be empirical
A doctor diagnoses a symptom that could come from several sources. Engineers should build the same way, developing a working mental model of what their agents and the newest models can actually do.
Today's models are far more capable than you think. AI has disproved a long-standing conjecture in combinatorial geometry, disproved a decades-old math conjecture, found hundreds of zero-day bugs, and as of recently found new proofs for high-dimensional sphere packing and the closest vector problem along with 8 others. Given that, it makes little sense to only test your agent within the envelope of what you already believe is possible. The engineering loop should be: ask the agent to do hard things, observe where it breaks, and form hypotheses about what to change. Even a naive model of your agent's capabilities surfaces intuitions for how to improve the harness.
We use this loop on our own harness. Our engineers point our agent at production traces, have it surface issues, and let it propose harness changes and regression tests against those failures.
One of the most direct ways to push a model is through instructions and skills. Fluid intelligence does not mean avoiding specificity. Modern models rarely need hand-holding on generic tasks, but production workflows still depend on it, because every company has its own context, operating procedures, and style of interacting with customers.
Here is how we think about it. The fact that current models are so capable is exactly why skills can be as in-depth as you want. A skill can tell the agent to talk in a specific register, use a specific tool a specific number of times, run a dynamic workflow, and then run adversarial testing over its own output. The crux is not that skills can be hyper-specific. It is that most engineers underestimate how much a capable model can absorb from a well-written skill. As intelligence compounds, scientific thinking is how we keep pace, by testing reality faster than our intuitions harden.
Hierarchical State Management
Strong agents need more than a large context window. As tasks get longer and more complex, the harder problem is preserving a reliable understanding of what has happened, what is still unresolved, and what should happen next.
This is the case for hierarchical state management. On memoryArena, a hierarchical state tree improved scores by 7.8 to 20.4 percentage points while reducing token usage by 55%.
Today, agent state usually lives in a markdown file that the model updates as it goes. That works with an attentive user in the loop, but it becomes fragile in autonomous workflows. Compaction is opaque, important context gets dropped, and a flat memory file cannot represent subgoals, parallel work, and intermediate decisions. A stronger approach builds state into the harness as two complementary layers:
events.jsonl is the ground truth: tool calls and results, subagent activity, errors, and state transitions, recorded deterministically by the harness rather than remembered by the model.
state.md is the map: the current objective, completed and unresolved work, important assumptions, skills invoked, and the next recommended action, synthesized by the model from the event log.
The event log preserves exactly what happened. The state file distills it into what matters next. A trace alone is too noisy to guide action, and a summary alone is too lossy for debugging and verification. Keeping both lets an agent resume interrupted work, recover from failures, coordinate across parallel workstreams, and avoid reconstructing the entire task history. The storage format can vary, but the principle holds: preserve both fine-grained evidence and high-level direction, and treat state as a first-class part of the harness.
We ran a simple experiment to probe this split on gpt-5.1 through the pi harness. We toggled each half of the state layer across four configurations — bare (off), the state map alone, the event log alone, and both — on two tasks of deliberately different shape: sanitize-git-repo, a long-running, multi-step job that rewrites a repository's history, and sqlite-db-truncate, a small and precise change. Each configuration was run four to five times per task.
Toggling each half of hierarchical state management for gpt-5.1. off = bare, state-map = state.md only, event-log = events.jsonl only, full = both. The fine-grained event log is the useful half: it beats the bare floor on both tasks (0.0 to 0.4 on the long-running sanitize task, 0.4 to 1.0 on the small sqlite change). The high-level state map is neutral-to-harmful on its own, and on the small task it even drags the full configuration (0.2) below event-log-only (1.0), most likely because a staleness nudge interrupts an agent that was already on track.
The event-log strongly improves both tasks, while the state-map improves the sanitize-git-repo task marginally. With only four to five runs per configuration, a single run shifts a score by about 0.2, so there is a lot of variance in these numbers and we read them as directional rather than precise.
Taken together, these graphs suggest that state management deserves more attention than most default agents and harnesses give it today, and a few simple changes produced large improvements. Some of the results are still confusing — running both halves at once did worse than the event log on its own — but this was minimally tested, and we expect the numbers to improve with more tuning and more runs.
This minimal work reflects what it means to be empirical: test things, see what works, and build a mental model.
The Adversarial Harness
Even strong workflows and good state management are not enough. As agents make decisions in production, the failure that matters is no longer only whether the task completed. It is whether the agent made good decisions along the way.
This is where adversarial review helps: asking another agent to review the work and find flaws. Multi-agent debate and adversarial review consistently improve results on benchmarks, and while it is already standard in code review, it extends to any hard work. Our suggestion is to build it into the harness as structured opposition. At key moments an independent reviewer inspects the main agent's assumptions, tool use, reasoning, and outputs. A natural trigger is the end of a workflow, either by instruction in the prompt or in the final skill the agent invokes.
This matters because agents are poor self-critics. They regularly rank their own actions as highly or more highly than others' when their actions are presented as their own. An independent reviewer sidesteps that bias, the same way LLM-as-a-judge works in evaluation by scoring reasoning, tone, and context in ways a simple assertion cannot.
We tested where that review should fire, and the simplest useful signal is at the end.
Where an adversarial review should fire, for gpt-5.1 on two tasks. "no review" is the bare agent; "end" adds a single review pass at the end of the run. That one pass helps on both tasks, lifting sanitize from 0 to 1.0 and sqlite from 0.4 to 0.8. The lift comes from the review itself, not from any extra structure around it.
Where this goes
Across these experiments, the effect of any single change depended on the model and the task it ran on. Effort had different impacts across models and their harnesses, a phenomenon we saw between Claude and GPT. State management helped on one task and hurt on another. Heavier scaffolding mostly added turns without adding score. A review pass at the end of a run was the one change that helped on both tasks we tried. Taken together, the data suggests there is no single best harness. What counts as a good setting is specific to the model and the workload it is measured on.
This is why we think harness optimization has to be empirical and continuous. Rather than fixing a harness in advance, the useful approach is to run the agent on the hard, real tasks it will face, analyze the traces where it fails, and use that evidence to decide the next change. We do this on our own system today. And because each new model can shift every one of these effects, this is not a one-time setup. Optimization should happen every time a team adopts a new model.
Limitations
This is a controlled but underpowered map, not a population-level claim.
Small, uneven samples. Cells hold 4 to 12 trials, so Wilson 95% intervals run about plus or minus 0.3 to 0.45. Read shapes within a study, not point values across studies.
Deliberately hard tasks. We selected non-saturated Terminal-Bench 2.0 tasks so that effort and scaffolding effects were measurable at all. That choice means the findings are specific to these tasks, models, and harnesses. We do not claim the capability dip generalizes to all models, or that harnesses generally make smarter models dumber. We observed one clear reversal, not six confirmed ones.
One benchmark, partial crossing. Everything here is Terminal-Bench 2.0. The harness-versus-model-family confound is resolved for Claude but not yet for the GPT-through-claude-code direction, and opus is not yet drivable under our pi build.
What would make it a population claim. Fully crossing harness against model family, running the scaffolding ablations across the whole model ladder rather than gpt-5.1 alone, powering up to 30 to 40 trials per cell, and pre-registering a held-out task suite.
The honest summary is simple. Capability is the dominant driver of agent success. Harness and scaffolding produce smaller effects that are conditional on the model and the task. The reliable way to know whether a harness helps your agent is to measure it, on your model, every time the model changes.