OR/AUDIT View source

Evaluation infrastructure for procedural AI

Your model.Under a real test.

Package any procedural model. Bind it to a versioned task. OR-Audit executes every trial and ships the result as a replayable evidence bundle.

Run OR-Audit Inspect a complete run
OR-Audit execution map Versioned task and agent packages flow into the evaluation core, then produce trial vectors and a replayable artifact.
INPUT 01VERSIONED TASKorg/task@version
INPUT 02MODEL PACKAGEorg/agent@version
OR/AUDITBIND + EXECUTE
OUTPUTVECTOR ARTIFACTsha256:…
RUN A PINNED PUBLIC EVALUATIONPYTHON 3.11+
uvx --from git+https://github.com/SeldingerMed/or-audit.git@5f384e5874ee6769d48ca2d12cb1162cfb00d768 or-audit run -d seldingermed/video-nextstep@0 -a example/video-predictor@0 --out ./runs/video-nextstep

ONE HARNESS

SIM POLICIESPROCEDURAL VIDEOFROZEN MODELSWORLD MODELS

The execution path

From model package to inspectable result.

OR-Audit gives task authors control of the test and model teams a stable interface to run it. The kernel handles the contract between them.

  1. 01

    Package the test

    Pin the environment, inputs, verifier, metrics, and gates under one versioned task identity.

    org/task@version
  2. 02

    Bind the model

    Match the task and agent through an explicit port before any compute starts.

    video-predict ↔ video-predict
  3. 03

    Execute every trial

    Run the pinned packages and record metrics, gates, abstentions, and evidence trial by trial.

    TrialVector[]
  4. 04

    Replay the result

    Hand the complete bundle to another team. They can reconstruct the run and match its artifact head.

    or-audit replay

One run, complete evidence

A result you can open, replay, and hand off.

Each job carries the task package, model package, configuration, per-trial vectors, trajectory, scorecard, and a content-derived head. The evidence moves with the result.

Read the artifact contract
TRIAL VECTOR03 / 03
next_step_correct
assessed
outcome_correct
assessed
abstained
true
artifact_head
b09ed7b5…

Public example 001

Three held-out clips. One frozen predictor. Every outcome visible.

Exactly what was tested

A frozen predictor received three held-out procedural clips. For each clip, it had to predict the next step and outcome, report an unsafe state when present, or abstain.

Task
seldingermed/video-nextstep@0
Agent
example/video-predictor@0
Protocol
video-clip → next-step + outcome
Episodes
3 held-out clips
PUBLISHED RESULTb09ed7b5…391682
2 / 2assessed predictions matched the next-step and outcome labels
Abstained1 / 3
Unsafe gate1 failed
ArtifactReplay matched

Two assessed clips matched their labels. The predictor abstained on the third. One assessed trial emitted unsafe=true, which activated the task’s declared safety gate.

01Open the productSeldingerMed/or-audit ↗ 02Open the taskinputs + labels + verifier ↗ 03Open the agentpackage + predictions ↗ 04Open the published runvector + pins + artifact head ↗

Ship the test with the model

Your next claim starts with a run.

Bring a versioned task or a packaged agent. OR-Audit turns the pairing into evidence another team can inspect and replay.

Run your first evaluation Browse public packages