Eval examples
Swarm runs, every agent's transcript included
Each example is one real run, published as it happened: the tree of agents, every message, tool call and tool result (long results trimmed), and the final answer. Research runs are graded on a rubric by a separate model; code migrations by tests the agents never saw, with the raw results (the patch, the checks) to download.
236 agents upgrade django-oscar to Django 4.2
Upgrade a real e-commerce framework, django-oscar 1.6, from Django 1.11 to Django 4.2: 689 files, 217 of which use APIs Django has removed. 236 agents split it: a root, 15 group leads, one agent per file that needs changing, and three reviewers, each in its own Blaxel sandbox, up to 100 at once.
236 agents · succeeded
Can Dr. Patel supervise by video?
A physician wants to supervise another office's visits by video. Which of five visits can be billed incident-to under virtual supervision in 2026, and what should happen with the rest? The clinic's compliance documents disagree on whether it is even allowed. 21 agents work through the schedule, the payer policy and the CMS rules.
21 agents · succeeded
Can she dispute her surgery bills?
An uninsured patient's three surgery bills all came in above her good faith estimate. Which of them qualify for the federal patient-provider dispute process, by when, and at what cost? 14 agents check the bills, the estimate and the clinic's own billing note against the federal rules.
14 agents · succeeded
Can we still correct this claim?
Can a clinic still correct a denied claim, and if not, what is the route and the exact deadline? The answer is buried in a six-part payer manual that a 2024 quick-reference card contradicts. 15 agents split the manual, check it against state law, and work out the dates.
15 agents · succeeded
Does this MRI need prior auth?
The front desk wants a yes or no today: does this MRI need prior authorization, and from whom? The honest answer is that the documents can't settle it. 12 agents check the clinic's papers and the payer's sources, and say what can and can't be determined before Monday.
12 agents · succeeded
Was the Medicare Advantage prior auth late?
A Medicare Advantage plan took 12 days to decide a standard prior authorization and denied it with one line. Was it within the federal timeframe, and what can the clinic demand? The payer contract, a vendor advisory and the clinic's own SOP disagree. 14 agents check them against the CMS rules.
14 agents · succeeded
When should a 42-year-old start mammograms?
A 42-year-old at average risk asks whether to start screening mammograms. The clinic's protocol says 50, its patient portal says every year from 40, and a scheduling email says to code screening as diagnostic. 17 agents sort out what the national guidelines say, and where they disagree.
17 agents · succeeded
Which Medicare patients can get GLP-1s?
A clinic asks which of six Medicare patients can get Wegovy or Zepbound, and how. Its own documents disagree, and one hides an instruction aimed at AI. 21 agents split the work, check the documents against CMS and FDA sources, and answer patient by patient.
21 agents · succeeded