Environments: A Playground for Agents to Replay, Explore, and Learn from Experience

A trace gives you an agent lifecycle - what the user asks, and the actions the agent took. To test on the same user request again, the agent has to replay again, with the same code, data, tools and services it had the first time. NeoSigma Environments makes that possible.

Today's agents operate in complex enviroments that consists of filesystem, Docker, customer repositories, databases, development services, networking, tools. With our environments, each run rebuilds your agent and its environment in a fresh sandbox, so you can replay production tasks and let the agent explore with its real tools.

Can you put the same agent back in the same environment, on demand, for every failure you care about?

For most teams, the answer is no. Building a working copy of a multi-service app is slow, and wiring the agent into it is slower. So teams skip the replay check, and fixes ship without being tested against the failure they were meant to fix.

NeoSigma rebuilds your agent's environment for every run, with its code, data, tools, and services ready in minutes. We tested it on one customer's agent with 42 replays of real production failures, and 90% of the graded replays reproduced the original failure.

An environment is everything outside the model that shapes what the agent does.

For a typical production agent, it has five parts:

1. 1.

   Agent Harness

   The harness, with underlying models, prompts, skills, tools, integrations, and subagents.

2. 2.

   Source Code

   The repository and any sibling repositories at the exact commit the agent ran on.

3. 3.

   Services

   The databases, queues, workflow engines, servers, or any external provider that the agent relies on.

4. 4.

   State & Configuration

   Filesystem, seed data, migrations, and environment variables, so the app isn't an empty shell.

5. 5.

   User

   The user on the other side of the conversation, whose replies steer the agent turn by turn.

If any of these is missing, you are no longer replaying the failure you saw. Without them, a replay often comes down to rerunning a single tool call or regenerating the agent's last message.

Each replay has five steps. Nothing is shared between replays.

1. 1.

   Define the task

   The task includes the original request, the commit the agent ran, the simulated user's goal, and the checks the agent failed.

2. 2.

   Rebuild the environment

   A new sandbox checks out every repository at that commit, loads the workspace's environment variables, and starts the stack.

3. 3.

   Wake the agent

   NeoSigma checks that the agent is ready to respond, then sends the first message.

4. 4.

   Play the User

   A simulated user outside the sandbox writes each next message from the original user's goal, until it ends the conversation or reaches the turn limit.

5. 5.

   Grade the trajectory

   A judge outside the sandbox reads every turn, tool call, and tool result, checks them against the rubric, and returns pass or fail.

We replayed real production failures from a multi-service agent whose stack combines a Postgres database, a workflow engine, a web app, and background workers. Three experiments were run in production: two datasets of 16 failures and one of 10, with 42 replays, 8 at a time. The datasets overlap, so 42 counts replays, not distinct failures. In each replay, the agent talked to a simulated user for up to three turns, and a Claude Sonnet 5.5 judge graded the trajectory.

Finding 1: Environments come up with the agent answering

40 of 42 environments (95%) reached an answering agent. The median time from a bare sandbox to a ready agent was 8.3 minutes, with a p95 of 10.5 minutes. In one failed boot, the Docker daemon never started. In the other, the agent never became ready.

Finding 2: Fresh environments reproduce failures

Of the 40 environments that came up, 30 ran to the end and got a usable grade. In 4 of the other 10, the agent stopped responding during the replay. In the remaining 6, the judge returned no usable grade. Of the 30 graded replays, 27 (90%) failed the task or its checks again, reproducing the original production failure. Each one ran from a bare sandbox to a graded result without anyone stepping in.

These results come from one production app, 42 replays, and a single judge model. A failing grade reproduces a failure only as well as the rubric each task was built from.

To work autonomously, agents need a way to verify their own work. A rich set of tools and a clear reward signal are only useful when the agent has a real environment in which to act, observe the results, and try again.

NeoEval turns failed production sessions into replayable tasks grounded in your users’ real workflows. Each eval runs your agent in an environment like the ones above and uses your own verifiers to measure success. Together, these tasks, environments, and reward signals provide a test and training bed for evaluating and optimizing your agent.

Environments become a playground for your agent: a place to replay real work, explore changes, and learn from experience.

Request a demo to sign up for our platform to supercharge your agents.