A protocol for evaluating frontier AI in deployments where errors have consequences beyond the screen — therapy, clinical scribing, tutoring, customer service, caregiving.
Existing AI benchmarks measure accuracy on discrete tasks. Assistants deployed in the real world execute jobs: long sequences of decisions in which a single error can unwind everything before it. We define RUN — Reliability Under Naturalistic conditions — a protocol for measuring how long a frontier AI agent can run before a critical mistake, across six task families spanning mental health, education, service, and embodied care.
Frontier AI has already revolutionized software. The physical world is next. Models are being deployed for therapy, tutoring, and caregiving, yet we have remarkably little insight into how effective or safe these systems are once they leave the lab.
Existing benchmarks focus on accuracy on discrete tasks — did the model solve this problem, yes or no. Assistants deployed in the real world don't work that way. They execute jobs: sequences of decisions unfolding over minutes, hours, or days, in which a single error can unwind everything that came before.
The quality we measure is the same one reliability engineers have asked of aircraft, power plants, and surgical tools for decades: how long since the last mistake?
Six pilot task families spanning ARIA's Work Packages. We aim for 5 tasks per family at v0.1.
Three tiers, increasing fidelity. Each tier runs the same set of tasks; results are reconciled across tiers using mixed-effects models.
for task in family.tasks:
for model in frontier_models:
for trial in range(N=5):
transcript = run(model, task, tier=T1)
outcome = grade(transcript, rubric=task.harm_rubric)
record(model, task, trial, outcome, duration)
fit pass_rate ~ log(task.duration) | model # logistic
horizon(model) = duration where pass_rate(model) == 0.50Each task carries a rubric authored by stakeholders, not researchers — clinicians for clinical tasks, teachers for tutoring, caregivers for caregiving.
We define a critical mistake as any output that, if acted upon by the user the assistant is serving, would lead to material harm. The rubric for a CBT (Cognitive Behavioral Therapy) protocol task, for example, is co-authored by a licensed therapist and includes things like:
We fit pass-rate vs. log task length per (model, family). The 50% crossing is the horizon. 80% is reported as a secondary metric.
Per model m, we fit pass rate as a logistic in log task length:
Horizon: t50(m) = exp(α_m / β_m).
Confidence bands from 1000-replicate bootstrap over trials. Measurements above 16 hr are reported with reduced reliability.
Every transcript, rubric, and fit is published with the scoreboard. The harness, persona scripts, and grading prompts are open-source. We commit to publishing a quarterly errata page documenting any retracted measurements.
We work with frontier labs to access compute and infrastructure where possible. No lab has editorial input on what gets measured or how.