ARIA RUN measures time-to-critical-mistake on real-world agentic tasks — therapy, tutoring, caregiving, customer service. Built on METR's time-horizon methodology, adapted for the physical world.
For each task family, success rate falls as the job gets longer. Where the curve crosses your chosen threshold (50%) is the horizon for that family. Pick another frontier model to see how horizons collapse as you move back in time.
Logistic fit over log task length. Confidence bands omitted in this view; see methodology §5 for the fit.
The quality of deployed assistants is the same one reliability engineers have asked of aircraft, power plants, and surgical tools for decades. Every deployed AI agent's goal is to run forever without errors. ARIA RUN measures how close they are.
Builds on METR's Time Horizon, adapted for ARIA's multi-domain task space and stakeholder-authored harm rubrics.
For each frontier model, success rate falls as tasks get longer. Where the curve crosses 50% is that model's time horizon. METR (Model Evaluation and Threat Research, the AI-safety org that pioneered this measurement) found that the 50% horizon doubles roughly every six months.
We adopt this methodology wholesale, then extend it to settings METR doesn't cover: clinical encounters, classrooms, homes.
Curve shapes reflect task-specific progress. Seven pilot families — visual cognition anchors the “0-minute” end (the summer-camp focus); more agentic families in development with partner labs.
AI systems deployed in mental and behavioral health (and elsewhere) are increasingly tested against benchmarks borrowed from coding, math, or general knowledge — domains where “trustworthy” reduces to “correct.” But the cognitive tasks that matter for human users — perspective taking, reasoning under uncertainty, metacognition, sensitivity to social and clinical context — are largely absent from public LLM evaluations, and the human-baseline data needed to interpret model performance on such tasks is scattered across labs in non-standard formats.
The project is flexible and will be shaped by the interests of the students on the team — the benchmark, the platform, and the analyses are intended as a community resource that grows beyond the camp.
We're in the planning phase, and we're looking for collaborators at every level — students, clinicians, postdocs, engineers, and principal investigators — to help decide what we build, which tasks we measure, and what real-world reliability should mean.
Leave your email and we'll reach out as the project takes shape. One note per milestone — no more.