Physical AI looks impressive in demos. But nobody can tell you whether a model is ready for production. PhAIL measures that gap – on real robots, with real commercial tasks, using the metrics operations teams actually care about.
UPH – units per hour. How fast the system works.
MTBA – mean time between assists. How long it runs before a human needs to step in.
The first task is bin-to-bin order picking – transferring individual items between containers. Evaluations run on Franka Research 3 arms with Robotiq grippers. More tasks and platforms are coming.
The DROID teleoperation dataset used to fine-tune all models on the leaderboard. 352 episodes, 12GB. Available for non-commercial use.
uv run --with positronic \
python -m positronic.cfg.ds.phailpip install positronic
python -m positronic.cfg.ds.phailEvery evaluation run on the leaderboard is a downloadable Positronic dataset – multi-view video and robot telemetry.
uv run --with positronic \
positronic-server \
[email protected]_runspip install positronic
positronic-server \
[email protected]_runsBrowse individual runs in the Run explorer.
Full evaluation protocol, scoring, and reproducibility details are in the white paper.
Questions, feedback, or interested in submitting a model? Reach out at [email protected].