Separate project-level workflow. This page covers project evaluation templates and decision policies, not pull-request-based Validate. For a current PR evaluation, use Evaluate a pull request and the validation spec.
/projects
An eval template and a decision policy are two of the three Pro gating artifacts on a Project (the third is target metrics). Together they fix how progress gets measured before any variant exists: the template says how to score; the policy says what counts as progress — Ship, Reject, or Iterate. Both are committed ahead of time so agent drafts are graded against the same bar as human work, and so a result can’t be graded against a goalpost that moved after the numbers landed.
This page documents project-level evaluation templates and decision policies. For current pull-request evaluations, use the validation spec.
Eval templates
An eval template is a saved, project-level spec for scoring a variant against a baseline. It declares the runtime, the baseline, and the metrics once, so every evaluation under the project uses the same harness. Locking a template doesn’t run anything; it captures the spec. Two runtimes are supported:- Custom
- lighteval
Point Remyx at an evaluation script that already lives in your repo. You provide:
- Eval script path: the path in your repo Remyx invokes (e.g.
docker/eval_stage/). - Baseline asset: the model, dataset, or process variants are compared against (e.g. a HuggingFace model slug like
Qwen/Qwen2.5-VL-7B-Instruct).
Locking a template captures the spec; it does not launch an evaluation. To actually run the eval on compute, see Experiment validation below.
Decision policy
The decision policy is a per-disposition rule builder. For each disposition you define one rule, and a rule is a combinator (all or any) over a set of conditions:
iterate is the automatic fallback: when neither the ship rule nor the reject rule matches, the run classifies as iterate.
Conditions are predicates over the experiment’s aggregated metrics and a few standard fields:
- Metric conditions: a metric from your locked template, an operator (
>=,>,improves_by, and so on), a threshold, and an optional confidence floor. - Field conditions: gates on fields like
delta_confidence,validation_status, andsample_size(e.g.delta_confidence >= 0.80).
Visualization only
This is deliberate. The policy tells you whether the evidence meets a pre-committed standard; it does not make the call about whether a change actually moved a business outcome. Automated disposition is explicitly deferred: the system automates the toil up to the judgment and stops there. See Continuous Experimentation for why this gate stays human.Experiment validation (Modal / W&B Launch)
This is the experiment-level path: an eval template run on your own Modal or W&B Launch account, with results written back onto the experiment. It is not Validate, the PR agent that designs a test from a mention and runs it on Remyx compute. The MCP tools
provision_validation, check_validation_status and get_validation_results belong to this path.
Connect a provider from Connectors. When a validation run finishes, results flow back onto the experiment via a signed webhook. Fields like
observed_delta and delta_confidence land on the experiment and feed the decision-policy predicates described above.
1
Provision the run
Remyx submits the locked template’s harness to your connected provider (Modal or W&B Launch).
2
Run on your compute
The eval executes on the provider against the template’s baseline asset and metrics.
3
Results post back
The provider posts results to Remyx over a signed webhook.
observed_delta, delta_confidence, and related fields are written onto the experiment.4
Policy renders
The decision policy re-renders against the new metrics, and the team logs the decision.
Where to set these
Both artifacts are settable two ways:- Web UI
- MCP tools
The project’s Pro sections: Target metrics, Evaluation (the eval template), and Decision policy. The rendered policy (Ship / Reject / Iterate) appears against the evidence at the gate — right on the PR and in the project’s experiment history.
Related
Projects
Where eval templates and decision policies are declared (Pro)
Inbox
Where Decide cards carry the evidence to the human gate
Continuous Experimentation
Why the decision stays a human gate
Run your first validation
Evaluate a pull request using defined test criteria