Benchmark
Eight tasks built from verified records, graded by re-running the physics. No language model judges anything.
Sec_001 · The tasks
| Task | The AI must | Graded by |
|---|---|---|
| 1 · Design repair ★ | Fix a failing design so it passes | Re-run simulation vs. same limit |
| 2 · Failure diagnosis | Say what fails, where, by how much | Spec limit and change log |
| 3 · Change prediction | Predict the result after a change | Real after-fix results |
| 4 · CAD edit | Build the new geometry from a change | Geometry diff vs. the real fix |
| 5 · Simulation auditing ★ | Find setup mistakes in a record | Real bugs our reviews found |
| 6 · Trade-off reasoning | Pick the best option and say why | Rejected options in change log |
| 7 · Multi-step iteration | Simulate, revise, retry until it passes | Converged? Steps used? |
| 8 · Production optimisation | Make it cheaper to produce at volume and still pass | Simulation + cost and cycle time saved |
★ Headline: design repair, run as a multi-step agent task. Simulation auditing is the hardest, because the mistakes are the real ones our reviews caught.
Sec_002 · The grader
The harness packages each record into a task with a hidden answer. The AI returns a design. The physics grader meshes it, runs the same solver with the same loads and boundary conditions, and checks the same spec limit. Open solvers only: CalculiX for structural and thermal, OpenFOAM for flow.
A baseline agent that reads CAD, runs the simulation, edits geometry and retries sets the bar, so you can see how far a raw model is from a working loop.
Scores publish with the first public release. Nothing on this page is a result yet.
Sec_003 · The board