AI Agent Testing: Catch Changed Requirements

Use AI agent testing to check changed requirements, inspect real output files, and expose stale graders with a runnable Python lab before a team handoff.

Three columns compare baseline and changed artifact requirements: Chrome.xml becomes chrome.xml, an offset timestamp becomes UTC Z, and CSV columns beta,alpha become alpha,beta.

AI agent testing should check the result against the current requirement, inspect the artifacts and actions, and preserve evidence for a human decision. For coding agents, one useful test is a paired task: run a small baseline task, change one requirement, and evaluate the new result against the changed contract. A passing baseline does not establish that the changed task passes.

This guide builds that check with a runnable Python lab. It compares filenames, timestamp serialization and CSV column order, then deliberately uses an outdated grader. The lab validates the evaluator with scripted controls. It makes no model calls and measures no coding agent's intelligence, accuracy or production reliability.

Start with the task, the environment and the acceptance evidence

Write down what the agent must deliver before choosing an evaluation score. For a repository task, that usually means a reviewed requirement, a starting revision, permitted actions and observable acceptance criteria. Keep the expected result outside the candidate's output so the candidate cannot declare itself correct.

For example, “generate a browser report” leaves several decisions unresolved. Must the filename preserve browser-name capitalization? Is a duplicate report allowed? Does success mean creating the file, printing its name, or explaining how someone could create it? Those are different acceptance conditions.

Use three layers of checks:

Layer What to inspect Example failure
Artifact Actual files or records against the reviewed requirement The agent reports success but leaves the old filename
Actions Tool calls, arguments and observed effects The correct report appears after an unapproved write
Handoff Requirement revision, evidence and unresolved decisions The next agent receives an obsolete acceptance rule

The lab below exercises the artifact layer. The other layers still need their own tests. IBM's overview of agent testing describes the broader component, trajectory and end-to-end coverage a complete evaluation program needs.

If the team has not agreed on observable behavior, start with a reviewed specification and acceptance contract. This procedure extends that work by comparing the original requirement with a deliberately changed one.

Why changed requirements deserve their own case

A baseline and a changed requirement answer different questions. The baseline checks whether an implementation solves the original task. The changed case checks whether the implementation follows the revised instruction while preserving the behavior that should stay unchanged. Keep both results visible instead of collapsing them into one average.

MaxxBench's research introduction provides the motivation. Its authors changed instructions or starting code in 89 existing coding tasks, including filename capitalization and timestamp representation. Their reported results show that similar scores on the original tasks can separate after these changes. These are the authors' findings, not a benchmark reproduced here.

This paired check measures one agent against a changed requirement. If the failure you need to expose appears only when several agents divide, combine, or contest work, use the multi-agent benchmark comparison to choose a collaboration or collision suite instead.

Our smaller exercise asks whether a grader can detect that kind of stale answer. It also asks a second question: what happens when the implementation follows the new instruction but the grader still expects the old result?

This is a paired requirement-adaptation check. Traditional mutation testing usually changes program code to assess a test suite. A requirement change instead changes what correct behavior means. Keep those purposes separate when interpreting results.

Define the baseline and the changed contract

Change one acceptance property in each task family. Freeze the input so an unrelated change cannot explain the difference. Each family below has an exact expected artifact, including its path and bytes.

Family Baseline requirement Changed requirement Behavior preserved
Filename Write Chrome.xml Write chrome.xml Report contents remain ok plus a newline
Timestamp Preserve 2026-11-12T10:00:00+02:00 Write 2026-11-12T08:00:00Z The instant remains the same
CSV columns Write beta,alpha with values 2,1 Write alpha,beta with values 1,2 Each value stays with its original column

The timestamp example deliberately checks representation as well as meaning. Both strings describe the same instant, but only one satisfies a contract requiring UTC with a trailing Z. A semantic-equivalence grader would miss that requirement.

Use exact bytes only when the task requires them. If whitespace, wording or column order is unconstrained, an exact-match test can reject a valid solution. The reviewer must settle that distinction before an agent runs, rather than tightening the rubric after seeing an answer.

Run the paired artifact-grading lab

Save the following script as paired_requirements.py in a disposable local directory. It uses the Python standard library, creates temporary output directories, and writes results.json and results.csv in the directory where you run it. Those two result files are overwritten on another run.

The reference control follows the selected revision. The stale control always follows the baseline. Both are ordinary scripted producers, clearly named in the records. The grader reads the actual output files and compares them with independently declared literal expectations; it does not trust a producer's success message.

"""Validate a paired-requirement grader with scripted controls, not live models."""
import csv
import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from tempfile import TemporaryDirectory

TASKS = {
    "filename": {
        "input": "Chrome.xml",
        "v1": {"Chrome.xml": "ok\n"},
        "v2": {"chrome.xml": "ok\n"},
    },
    "timestamp": {
        "input": "2026-11-12T10:00:00+02:00",
        "v1": {"result.txt": "2026-11-12T10:00:00+02:00\n"},
        "v2": {"result.txt": "2026-11-12T08:00:00Z\n"},
    },
    "columns": {
        "input": {"beta": "2", "alpha": "1"},
        "v1": {"result.csv": "beta,alpha\n2,1\n"},
        "v2": {"result.csv": "alpha,beta\n1,2\n"},
    },
}

def produce(folder, family, revision, control):
    # This is a transparent scripted reference, not an AI agent.
    current = revision if control == "reference" else "v1"
    value = TASKS[family]["input"]
    if family == "filename":
        files = {value.lower() if current == "v2" else value: "ok\n"}
    elif family == "timestamp":
        rendered = value
        if current == "v2":
            rendered = datetime.fromisoformat(value).astimezone(timezone.utc)
            rendered = rendered.isoformat().replace("+00:00", "Z")
        files = {"result.txt": rendered + "\n"}
    else:
        keys = sorted(value) if current == "v2" else list(value)
        files = {"result.csv": ",".join(keys) + "\n" +
                 ",".join(value[key] for key in keys) + "\n"}
    for name, content in files.items():
        (folder / name).write_text(content, encoding="utf-8", newline="")

def grade(folder, expected):
    actual = {str(p.relative_to(folder)): p.read_bytes()
              for p in folder.rglob("*") if p.is_file()}
    return actual == {name: value.encode() for name, value in expected.items()}

def main():
    rows = []
    for family, task in TASKS.items():
        for revision in ("v1", "v2"):
            for control in ("stale", "reference"):
                with TemporaryDirectory() as temporary:
                    folder = Path(temporary)
                    produce(folder, family, revision, control)
                    passed = grade(folder, task[revision])
                    stale_grader_passed = grade(folder, task["v1"])
                    expected_pass = revision == "v1" or control == "reference"
                    assert passed == expected_pass, (family, revision, control)
                    assert stale_grader_passed == (revision == "v1" or control == "stale")
                    observed = {str(p.relative_to(folder)): p.read_text(encoding="utf-8")
                                for p in folder.rglob("*") if p.is_file()}
                    rows.append({"family": family, "revision": revision,
                                 "control": control, "passed": passed,
                                 "stale_grader_passed": stale_grader_passed,
                                 "observed_files": observed,
                                 "artifact_sha256": {name: hashlib.sha256(value.encode()).hexdigest()
                                                     for name, value in observed.items()},
                                 "criterion_sha256": hashlib.sha256(json.dumps(
                                     task[revision], sort_keys=True).encode()).hexdigest()})
    # A claimed success cannot replace an observed artifact; residue must fail too.
    with TemporaryDirectory() as temporary:
        folder = Path(temporary)
        assert not grade(folder, TASKS["filename"]["v2"])
        produce(folder, "filename", "v2", "reference")
        (folder / "Chrome.xml").write_text("ok\n", encoding="utf-8")
        assert not grade(folder, TASKS["filename"]["v2"])
    assert sum(row["passed"] for row in rows) == 9
    assert sum(row["stale_grader_passed"] for row in rows
               if row["revision"] == "v2" and row["control"] == "stale") == 3
    result = {"scope": "Scripted controls and a deterministic grader; no model calls.",
              "observations": rows, "expected_passes": 9, "expected_failures": 3,
              "negative_controls": ["missing artifact rejected", "extra stale artifact rejected"],
              "contract_sha256": hashlib.sha256(json.dumps(TASKS, sort_keys=True).encode()).hexdigest()}
    Path("results.json").write_text(json.dumps(result, indent=2) + "\n")
    with Path("results.csv").open("w", newline="") as stream:
        writer = csv.DictWriter(stream, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    print(json.dumps(result, indent=2))

if __name__ == "__main__":
    main()

Run it with:

python3 paired_requirements.py

On Windows, python paired_requirements.py works when that command selects Python 3. The exercise was executed with Python 3.14.4. It requires no model account, API key or downloaded package.

The JSON records the observed file contents, their SHA-256 hashes, the acceptance-criterion hash and each verdict. The CSV makes the paired rows easy to compare. These records help a reviewer inspect what the grader evaluated after the temporary output directories have been removed.

Hashes identify bytes. They do not prove who approved a requirement or prevent someone from changing the grader and its recorded hash together. Keep approval and access control separate from content identification.

Read the results without turning them into a model score

The run contains six task cases: three families, each with two requirement revisions. Running two scripted controls on each case produces 12 observations. The current grader gives nine expected passes and three expected failures. Both additional negative controls also succeed: a missing artifact fails, and an extra stale artifact fails.

Scripted producer Baseline cases accepted Changed cases accepted Changed cases accepted by the stale grader
Stale control 3 of 3 0 of 3 3 of 3
Reference control 3 of 3 3 of 3 0 of 3

The last column is the useful trap. Keeping the old grader makes every stale changed-case output look acceptable. It also rejects every updated reference output. The all-green stale run would reward preserving the old behavior.

These outcomes establish properties of this fixture and grader. They do not estimate how often a live agent ignores changed instructions. There is no sampled model population, repeated model experiment or statistical confidence interval behind the table.

The missing-file and residual-file checks matter because a partial artifact check can be misleading. Checking only that chrome.xml exists would overlook a leftover Chrome.xml. Here, the expected artifact set is exact. A real repository evaluator should inspect the task's intended paths and relevant side effects without rejecting every unrelated tracked file.

Connect the procedure to a real coding agent

Replace the scripted producer with your existing coding-agent runner while keeping the evaluator under separate review. Give the runner an isolated checkout, the frozen input, the current task instruction and only the access required for the exercise. Capture its outputs before grading them.

Keep the reviewed grader and expectations outside the candidate's writable checkout. This demonstration is not a sandbox or a hardened parser for hostile filesystem entries. The runner must provide execution isolation and the relevant file-access boundaries.

For each task pair:

  1. Start the baseline from a known repository revision and environment.
  2. Run the baseline instruction and retain the actual patch, files, tool trace and configuration.
  3. Reset the environment and run the changed instruction from the same starting revision.
  4. Grade each output with its corresponding reviewed contract.
  5. Inspect whether unrelated behavior changed and whether the execution respected its permitted actions.
  6. Record failures and the reviewer's decision before promoting a configuration.

Fresh runs measure adaptation to the changed instruction without earlier conversation state. A second experiment can use an in-session reviewer correction, but label it separately: that also tests memory and context handling. Do not combine those conditions into one score and call it instruction-following ability.

For a live model, repeat trials and retain unsuccessful attempts. Report the number of distinct tasks and the number of attempts separately. Include the model identifier, runner version, prompt and tool configuration, repository revision, requirement revision and grader revision. A successful retry should not erase the first failure.

The runner may also fail before delivering any artifact. Record that as an execution failure or blocked task, according to the reviewed rubric. It supplies no evidence that the model correctly implemented the requirement.

Interruption also needs a recovery policy. The agent harness recovery example separates an effect from its saved receipt, which helps explain why rerunning a task and grading its final files are different operations.

Keep artifact checks beside trajectory and safety checks

Correct output is one part of agent behavior. A coding agent could create the expected file after touching an unrelated configuration, using an unapproved tool or discarding another worker's changes. This fixture does not inspect those actions.

Add trace-based checks for the specific workflow rather than treating a file comparison as a complete evaluation. Splunk's testing guide covers trajectory and lifecycle evaluation; our AI guardrails example examines separate input, action and output boundaries.

Subjective judgments need a documented rubric and human calibration. A model judge can assist, but its disagreement is another result to investigate. For facts with a precise oracle, such as an actual filename, read the artifact directly.

Production failures should also become reviewed cases. PostHog's guide to testing agents explains the feedback loop from user failures and traces into evaluation suites. The synthetic cases here validate the checking method; they cannot represent your users' full task distribution.

Make the test result useful at the next handoff

A useful handoff names the current requirement and shows what was observed. Include the task ID, repository revision, acceptance-criterion hash, actual artifact locations, verdicts, unresolved exceptions and the person responsible for the decision. Keep the result available to the next worker rather than leaving it only in one terminal session.

When a requirement changes, decide what happens to work already in progress. A worker may need to stop, rebase its task on the new requirement, or finish a deliberately preserved baseline. That decision belongs in the shared work record. A version number without the decision can leave two agents correctly implementing different contracts.

SYNCStaff can hold shared work records and human decisions while declared intents and file leases help coordinate repository work. The evaluator remains the team's responsibility. SYNCStaff does not turn this fixture into a model benchmark or make a passing test a production approval.

Use the human review checkpoint to accept the relevant evidence and name what still needs testing. Before the next coding-agent handoff, put the changed requirement, paired results and review decision on the shared board so the next worker starts from the contract that is actually current.

Frequently asked questions

How do you test an AI agent?

Define a realistic task, its permitted actions and observable acceptance criteria. Run the agent in a controlled environment, inspect the actual artifacts and tool trace, and compare them with the reviewed requirement. Repeat meaningful trials, retain failures and use human review for judgments that an exact assertion cannot settle.

Is AI agent testing the same as using AI to test software?

They have different subjects. AI agent testing evaluates the agent's behavior, outputs and actions. Using AI to test software puts an agent in the tester role. A coding agent can need both: tests for the software it produces and evaluations showing whether it followed the assigned task and access boundaries.

How can a test catch an agent following an outdated requirement?

Keep the original task and create a paired case with one changed requirement. Freeze the starting state, run both instructions and grade each output against its own reviewed contract. Include a deliberately stale implementation as a negative control. It should fail the changed case even when it passes the baseline.

Does the twelve-observation lab measure coding-agent accuracy?

No. It uses scripted producers to verify a deterministic grader, including false passes caused by an outdated expectation. The nine expected passes and three expected failures describe those controls. Measuring a real coding agent requires actual model runs, representative tasks, repeated attempts and a documented environment and scoring method.

Does a passing agent evaluation authorize production deployment?

A passing evaluation supplies evidence for a separate release decision. It does not establish coverage of every task, secure tool use or production reliability. A reviewer still needs to examine the relevant action boundaries, unresolved failures, monitoring and recovery plan, and the specific scope that the evaluation actually exercised.