Multi-Agent Benchmarks: Collaboration vs Collision

Compare eight multi-agent benchmarks for coding, conflict resolution, dependencies, resource contention, teamwork, and coordination cost.

Coding-agent paths converge on a shared repository while a benchmark panel contrasts a coordinated green result with a collision and failed red result.

A multi-agent benchmark should tell you whether several agents produce a correct combined result, not merely whether each agent looks productive alone. For coding teams, the most useful current benchmarks split into three groups: ParallACT, CooperBench, and AsynCodeBench test shared software work; TeamBench tests whether enforced roles add value; and CLASHarness, SILO-Bench, DPBench, and MultiAgentBench isolate conflict resolution, information sharing, resource contention, and communication topology.

No single score answers all of those questions. This guide compares eight benchmarks on the engineering decision each one can support, the failure modes it exposes, and the evidence it leaves out. It is a landscape comparison based on the benchmark authors' published materials as of November 18, 2026. SyncStaff did not run these suites for this article, and the reported findings are not our independent reproduction.

Choose the benchmark from the failure you need to detect

Start with the production risk, not the most impressive leaderboard. A team worried about two agents changing one repository has a different evaluation problem from a research group studying 100 agents with partial information.

Engineering question Best starting point Why
Can independent coding agents combine compatible and contradictory requests? ParallACT It pairs separately assigned changes in a shared codebase and evaluates the combined result.
Can two agents deliver related features with messaging or a team structure? CooperBench It compares solo, peer-cooperation, and lead/member settings on repository tasks.
When do asynchronous producer-consumer dependencies become valid? AsynCodeBench It probes explicit dependencies during execution, not only at the final test run.
Do planner, executor, and verifier roles cause the improvement? TeamBench OS-enforced role separation and ablations make each role's contribution testable.
Can an agent resolve conflicting claims and shared-state conflicts? CLASHarness It tests verifiable conflict-resolution episodes, including handoffs and shared locks.
Does communication overcome information silos at scale? SILO-Bench It measures success, tokens, and communication density across distributed tasks.
Can agents coordinate simultaneous access to scarce resources? DPBench It isolates deadlock, fairness, throughput, and decision timing.
Which communication topology helps in interactive scenarios? MultiAgentBench It compares star, chain, tree, and graph protocols with milestone-based indicators.

The practical conclusion is not that one benchmark wins. It is that each suite turns a different coordination assumption into something observable. Before adopting one, write down the failure that would hurt your team: a rejected merge, a clean merge with broken behavior, a stale dependency, duplicated implementation, unresolved resource contention, or a decision no agent was allowed to make.

A normalized comparison of eight multi-agent benchmarks

The table below normalizes what each suite is designed to test. “Not central” means the published benchmark is not primarily built around that dimension; it does not prove that the implementation can never represent it.

Benchmark Work setting Interaction under test Primary evidence What it does not settle
ParallACT Independent agents changing a shared codebase Complementary interfaces and contradictory requests Joint task success under different visibility, ordering, awareness, and communication conditions Centrally managed role systems, human review, or production access control
CooperBench Related features from real repositories Solo versus peers versus lead/member teamwork, with optional messaging and Git exchange Per-feature tests, joint success, trajectories, patches, and coordination indicators Broad organizational policies or many-agent scale
AsynCodeBench Repository tasks with directed dependencies Whether and when one agent satisfies another agent's dependency Test pass rate, ADPR, DRS, final success, and token use Same-file contradiction and resource locking as primary targets
TeamBench Software, data, and incident-response tasks Structurally separated planner, executor, and verifier roles Full-team and role-ablation results plus Teamwork Necessity Index Independent principals making unrelated requests to one repository
CLASHarness Multi-agent computer-use conflict episodes Conflicting claims, workspace diffs, multimodal evidence, closing handoffs, and shared locks Per-check scores, full-episode pass, trajectories, and cost A repository-specific coding benchmark or normal Git integration flow
SILO-Bench Algorithmic work split across information silos Free-form distributed coordination across protocols and team sizes Success, token consumption, and communication density Software artifacts, merge quality, or human approval
DPBench Dining Philosophers resource-contention environment Sequential versus simultaneous decisions, communication, deadlock, and fairness Deadlock, throughput, fairness, and message-action consistency Coding quality or multi-file integration
MultiAgentBench Interactive collaboration and competition across domains Communication topology and planning strategy Task score and milestone-based collaboration indicators Shared-repository coding collisions and file ownership

No final percentage is portable across that table. The task distribution, number of agents, tool access, model, budget, and definition of success all change. Use the matrix to choose a suite, then compare configurations inside its frozen protocol.

What the coding-focused benchmarks reveal

ParallACT tests independent work that must combine

ParallACT is the closest match for organizations in which different people launch agents against the same repository. It contains complementary pairs, where both changes are useful but must agree on an interface, and contradictory pairs, where the requests cannot both be satisfied naively.

The authors compare a single-agent oracle that sees both requests with multi-agent conditions that vary order, visibility, awareness, and communication. Their abstract reports a sharp result for one configuration: Claude Code with Claude Opus 4.8 fell from 62 percent joint success for the oracle to 16 percent on contradictory parallel tasks. The paper also reports that simply telling an agent a peer exists did not materially close the gap, while communication recovered only part of the loss at additional token cost.

That finding should not be generalized into a universal model ranking. It establishes a controlled gap under the paper's tasks and configurations. Its strongest contribution is the task construction: the combined specification, rather than one agent's local tests, decides whether the team succeeded.

CooperBench compares solo, peer, and team configurations

CooperBench packages 652 tasks built around paired features across 12 repositories in Python, TypeScript, Go, and Rust. Its current harness supports a solo setting, peer agents with Redis messaging, and a lead/member team setting with a shared task list and scratchpad.

The repository reports that coordinating agents performed worse than a single agent given the same total work in its evaluated configurations. It also reports communication consuming up to 20 percent of the budget while reducing conflicts without improving overall success. Treat those as the project authors' results, not as a universal forecast for every model or coordination system.

CooperBench is particularly useful because the agent trajectories and patches make different failure stages inspectable. When evaluating your own tooling, keep the feature pairs, model, harness, starting images, and budgets fixed. Change one coordination mechanism at a time.

AsynCodeBench measures dependency closure during the run

AsynCodeBench focuses on a failure that final pass rates can hide: one agent produces an API or artifact that another agent depends on, but the dependency is resolved late, transiently, or not at all. Its 19 repository tasks expose 52 directed dependencies through executable checkers.

The benchmark introduces Asynchronous Dependency Pass Rate, which measures how many cross-agent dependencies ultimately hold, and Dependency Resolution Step, which records when each dependency first becomes valid. That trajectory matters when agents work at different speeds. A final green build cannot show whether downstream work spent most of the run blocked on a stale contract.

Use it when handoffs and producer-consumer relationships are the main risk. Pair it with a contradiction-oriented suite if agents can receive mutually incompatible requirements.

Benchmarks for roles, conflict, information, and resources

TeamBench asks whether the team structure earns its cost

TeamBench contains 851 templates and 931 seeded instances spanning software engineering, data engineering, and incident response. Planner, executor, and verifier roles run in isolated containers. Five ablation conditions compare the full team with restricted or missing capabilities, and the Teamwork Necessity Index estimates when teamwork provides value over the alternatives.

This is useful when the architecture itself is the hypothesis. A team should not claim “multi-agent is better” after comparing a coordinated team with a weaker baseline that also changed tools, context, and budget. Role ablations make the narrower question testable: did planning, execution, or verification account for the result?

CLASHarness grades the complete resolution, not isolated checks

CLASHarness is a computer-use conflict benchmark rather than a coding benchmark. Its published corpus contains 600 episodes built from verifiable conflict scenarios, including competing peer claims, workspace file differences, multimodal evidence, asynchronous effects, closing handoffs, and shared locks.

The distinction between individual checks and episode success is valuable for software teams. An agent can make several locally correct moves and still fail the overall protocol by stopping before the handoff or leaving shared state unresolved. The repository publishes generation, evaluation, and analysis artifacts while documenting known dataset issues. That transparency is a reason to inspect the corpus, not a reason to treat every number as timeless.

SILO-Bench separates talking from useful distributed computation

SILO-Bench evaluates free-form collaboration when each agent observes only part of the problem. It includes 30 algorithmic tasks at three communication-complexity levels and reports 1,620 experiments across 54 combinations of protocols, team sizes, and models.

Its three main measures are success rate, token consumption, and communication density. The authors report a communication-reasoning gap: agents may communicate actively without turning the interaction into a correct distributed result. That is a useful warning for product telemetry. Message count is activity, not evidence of coordination quality.

DPBench isolates simultaneous resource contention

DPBench adapts Dining Philosophers into controlled conditions that vary decision timing, group size, communication, and prompt strategy. The environment reports deadlock, throughput, fairness, and message-action consistency. Its controlled structure removes most coding variance so the experiment can focus on resource coordination.

The paper reports that sequential decision settings were much easier than simultaneous ones and that communication did not reliably remove deadlock. This does not prove a particular repository needs a lock service. It shows why a system that shares scarce resources should test simultaneous decisions explicitly instead of assuming polite messages serialize access.

MultiAgentBench compares topology and strategy

MultiAgentBench covers collaboration and competition in interactive scenarios and measures milestone progress as well as task completion. It compares star, chain, tree, and graph communication protocols alongside group discussion and cognitive planning.

Use it when topology or planning strategy is the experimental variable. Its broad scenarios make it less direct for a shared-repository engineering decision, but its milestone-based indicators are a useful alternative to collapsing an entire team run into one terminal score.

Why a clean merge is not a collaboration score

Git detects overlapping text. It does not decide whether two non-overlapping changes agree on an API, preserve a business rule, use the same migration order, or obey a human policy. A pair of branches can merge cleanly while the combined build fails or, worse, while tests miss the incompatible assumptions.

The inverse also matters. A textual conflict can be easy to resolve when both agents made compatible changes near the same lines. Counting Git conflicts alone therefore rewards and punishes the wrong things.

A coding benchmark should keep at least three verdicts separate:

  1. Local correctness: did each patch satisfy its own task?
  2. Integration correctness: does the combined repository satisfy both compatible requirements?
  3. Coordination correctness: were contradictions, dependencies, reservations, and human-only decisions surfaced and resolved through the allowed process?

Our AI agent testing guide explains why a fixture and its grader need independent controls before an agent score means anything. A benchmark adds a task distribution and protocol; it does not remove that obligation.

The important coverage gap for production coding teams

The eight benchmarks cover much more than a single-agent coding leaderboard, but their published designs leave an important combined scenario thinly tested:

  • agents launched by different human principals;
  • mixed coding clients or harnesses;
  • one shared repository plus non-file resources;
  • textual, interface, semantic, duplicate-work, and policy conflicts;
  • durable work records and human decisions;
  • the coordination cost of detecting and resolving each conflict.

Some suites cover pieces of this list. ParallACT directly models independent requests and tests multiple harness/model configurations. CooperBench exposes paired repository work and communication. AsynCodeBench makes dependencies observable. CLASHarness tests conflict resolution and shared locks. None of their published protocols combines every dimension into one production-software evaluation.

That gap should shape the next experiment, not a marketing claim. A coordination layer can be tested as a treatment while the tasks, models, starting commits, and budgets stay fixed.

A fair coordination benchmark for multi-agent coding

Use four task families and at least one negative control:

Task family Required interaction Failure to detect
Independent control None; agents touch unrelated modules Coordination adds overhead without benefit
Complementary interface Producer and consumer must agree on one contract Both patches pass locally but fail together
Contradictory requirement Two requests cannot both be implemented as written One specification silently wins
Shared resource or decision Agents need the same file, port, migration, deployment slot, or human ruling Deadlock, duplicate action, stale ownership, or unauthorized choice

Run the same tasks under distinct coordination arms, such as no coordination, free-form messages, structured work intents, advisory reservations, enforced pre-write checks, worktrees with an integration owner, and human escalation. Do not change all of these at once and attribute the result to “orchestration.”

At minimum, record:

  • joint acceptance-test pass rate;
  • each agent's local task pass rate;
  • textual conflicts and clean-merge semantic failures;
  • duplicate or abandoned work;
  • dependency resolution time;
  • unresolved reservations and human escalations;
  • wall time, model tokens, tool calls, and coordination messages;
  • patches, starting revisions, prompts, configuration, and failed attempts.

The independent control matters. If two tasks never interact, a complex protocol may only make the run slower. The contradictory case matters for the opposite reason: a clean merge can still be wrong when both patches encode incompatible assumptions.

The existing agent orchestration guide provides a small deterministic shared-file fixture and the architecture choices around it. Treat that as a baseline check, not a model benchmark. The agent harness guide explains the runtime layer that also has to remain fixed when comparing coordination treatments.

Where SyncStaff belongs in the experiment

SyncStaff's relevant features are measurable treatments: agent identity, declared work intent, file leases, change notices, shared board state, and recorded human decisions. A fair experiment would enable them in separate arms and ask which failure each mechanism detects, what it costs, and whether the combined software improves.

That wording is deliberate. Work records and leases can surface overlapping plans and reserve declared paths. They are not proof that every process on an operating system is blocked from writing, and they cannot resolve a product decision that no human has made. The benchmark should preserve those boundaries.

For high-impact work, pair coordination evidence with AI guardrails and a reviewed specification. Our spec-driven development guide shows how to make acceptance conditions explicit before implementation. Keep the reviewed grader outside the candidate's writable output so an agent cannot declare its own work correct.

A practical evaluation sequence

  1. Name the failure. Choose interface drift, contradiction, dependency delay, resource contention, or another observable risk.
  2. Select the closest benchmark. Use the table above instead of defaulting to a generic coding leaderboard.
  3. Freeze the protocol. Pin repository revisions, task instances, models, tools, budgets, prompts, and graders.
  4. Validate the evaluator. Run reference and intentionally failing controls before spending model budget.
  5. Keep a single-agent baseline. Parallelism has to beat or complement a competent sequential alternative.
  6. Change one coordination treatment. Messaging, work records, leases, topology, and human review are separate variables.
  7. Grade the combined outcome. Run integration and semantic checks after patches merge.
  8. Publish artifacts and limitations. Include failures, null results, configuration, and costs, not only a headline score.

The best multi-agent benchmark is therefore not the largest suite. It is the one that makes your expected coordination failure necessary, observable, and reproducible. For shared-code engineering, start with ParallACT or CooperBench, add AsynCodeBench when dependencies matter, and borrow controlled conflict and resource tests from CLASHarness and DPBench. Then report joint correctness and coordination cost together.

Frequently asked questions

What is a multi-agent benchmark?

A multi-agent benchmark evaluates what happens when several agents must coordinate, compete, share information, integrate work, or resolve conflicts. A useful benchmark measures more than final task completion: it also records joint success, communication or token cost, dependency closure, conflicts, resource use, and the conditions under which each agent operated.

Which benchmark is most useful for multi-agent coding?

Use ParallACT when the question is whether independent agents can combine complementary or contradictory changes in one codebase. Use CooperBench for paired feature work with solo, peer, and lead-member settings. Use AsynCodeBench when asynchronous dependencies and their resolution time are the main concern.

Does a clean merge mean multi-agent collaboration worked?

No. A clean textual merge can still combine incompatible interfaces, assumptions, or policies. Grade the merged result against the combined specification, integration tests, and any required human decision rather than using the absence of Git conflicts as the success criterion.

Can scores from different multi-agent benchmarks be compared directly?

Usually not. The benchmarks use different tasks, models, tools, communication protocols, budgets, success definitions, and levels of isolation. Compare systems within the same benchmark and frozen protocol. Across benchmarks, compare the failure modes and coverage dimensions rather than treating unlike percentages as one leaderboard.

What should a company measure in its own multi-agent coding evaluation?

Start with joint acceptance-test pass rate, individual-task pass rate, semantic integration failures, duplicate work, unresolved dependencies, human escalations, elapsed time, token or compute use, and coordination overhead. Include an independent-work negative control so the process does not receive credit for coordination when the tasks never interact.

Does passing a multi-agent benchmark make a system safe for production?

No. A benchmark supplies evidence within a defined task set and protocol. Production approval still depends on access boundaries, review requirements, rollback and monitoring, repository-specific tests, data sensitivity, and the failure modes the benchmark did not exercise.