Validation spec

A validation spec records what to measure, how to run the test, and how the result will be judged. A PR can use a committed .remyx/validation.yaml file or ask Validate to propose one. Review the comparison, required metrics and criteria before execution.

Start with a working example

The minimal example below uses a test or command that exists in the linked repository. Copy its files together and check the expected metric output before adapting it.

.remyx/validation.yaml

question:
  kind: capability
  ask: "Does AbsTopK recover negative signals without hurting reconstruction? Train one TopKTrainingSAE and one AbsTopKTrainingSAE on tiny-stories-1M activations with the same k, width, seed and tokens under 30 minutes on CPU, and compare run_evals explained_variance and L0 side by side."
loop:
  max_iterations: 8
  fix_code: true
benchmarks:
  - name: abstopk-vs-topk-tinystories
    suite:
      notebook:
        path: "eval/eval_abstopk_vs_topk.ipynb"
        packages: [wandb]
    baseline: none
    scorer: explained_variance_gap__topk_minus_abstopk
    policy:
      guardrail_veto: true
    metrics:
      - name: explained_variance_gap__topk_minus_abstopk
        direction: min
        threshold: 0.02
        role: target
        bar: goal
        reads_as: "How much of TopK's held-out explained_variance AbsTopK fails to recover; 0 or negative means parity or better."
      - name: abstopk_negative_latent_share
        direction: max
        threshold: 0.01
        role: guardrail
        bar: goal
        reads_as: "Share of AbsTopK's k selected latents that are negative on held-out tokens; TopK's is 0 by construction."
compute:
  tier: cpu
  timeout_s: 2700
requires: [wandb]

This example includes the test files it runs. Copy the spec and its referenced files together, then adjust the repository paths, data and criteria for your change. Full spec on the branch · The notebook it runs · Executed notebook · Read the GitHub result

Full schema

The reference below defines supported questions, suites, metrics, policy, compute, repair, model selection, artifacts and defaults.

Location and precedence

Where	Used when
`.remyx/validation.yaml` at the PR head	Always first. A spec on the branch is what runs.
A locked eval template on the project	When the branch has no spec.
A proposal	When neither exists: the bot designs one, commits it to the branch, and waits for `@remyx-ai validate`.

Rules the parser applies before anything runs:

A file that only carries model: is configuration, not a spec. Its model: still applies to the design.
A flat spec (benchmark keys at the top level) is lifted into a one-entry benchmarks: list. Both shapes are accepted.
Keys the proposer sometimes writes under the benchmark (compute, metrics, policy, baseline, variants) are hoisted; top level wins on collision.
A hand-written spec is never revised by feedback on the PR. Edit the file and mention the bot with validate.
With a committed spec, instructions in your mention are ignored and the started comment says so.

Lint before you commit: POST /api/v1.0/validate/specs/lint {yaml} runs the same parser and returns problems; the Studio spec editor uses it on every keystroke.

Key reference

Top level

Key	Type	Default	Meaning
`name`	string	`validation`	Label for the run.
`question`	map	derived	`kind` and `ask`. See below.
`checks`	map or `true`	none	”Does it work?” `command` to run on the feature arm; `true` means `pytest -x -q`.
`benchmarks`	list	none	”Does it do what it claims?” Only the first entry runs today.
`variants`	map	none	Per-arm env overrides: `{feature: {env: {LR: "1e-4"}}}`. Reserved platform keys are dropped.
`seeds`	int	`1`	How many seeds per arm. Multiplies the arms.
`compute`	map	`{tier: cpu}`	Tier, budget and rigor. See below.
`loop`	map	all on	The fix loop’s bounds. See below.
`model`	map or string	none	`provider` and `name` of the model that designs and repairs. `model: zai` is shorthand.
`requires`	list or string	none	Connected accounts the run needs, e.g. `[wandb]`. A `logging.<provider>` block implies it.
`logging`	map	none	`wandb: {entity, project, base_url}`. See W&B.
`held_constant`	list	none	Factors the design holds fixed. Checked against provenance by the no-signal check.
`avoid`	list	none	What the design must not do.
`provenance`	map	none	Free-form provenance the report echoes.
`production`	map	none	Production signal wiring for calibration.
`report`	map	none	`headline`, `caveat`, `next`, `findings[]`, `establishes[]`, `does_not_establish[]`, `not_measured[]`. Templates may reference metrics with `{metric}`; a finding whose metric was never emitted is dropped.

`question`

Key	Values	Meaning
`kind`	`compare`, `parity`, `capability`, `regression`	How many arms run and what the baseline is. Derived when absent: `baseline: none` → capability, a published baseline → parity, benchmarks → compare, checks only → works.
`ask`	string	The request this test answers, in the requester’s words. Reported only when this PR is where the ask came from.

`question.kind`	Asks	Arms	Baseline
`compare`	Did the PR change a measured outcome against the base branch?	baseline + feature	the base branch
`parity`	Does it match a published reference or known result?	feature only	published numbers
`capability`	Does the new capability meet an absolute criterion?	feature only	none, absolute bars
`regression`	Does a refactor stay within the allowed tolerance?	baseline + feature	the base branch

Baselines. The baseline arm is skipped when the repo publishes its own numbers (baseline: with a reference dict, or a results corpus the harness reads) or when a prior validation on the same base commit and the same spec wiring already measured it. Reuse is exact-match only; anything less re-runs the arm. baseline: none runs one arm with absolute bars.

`benchmarks[0]`

Key	Type	Meaning
`name`	string	Names the committed test, notebook and Dockerfile stems.
`suite`	string or map	What runs. Forms below.
`baseline`	string or map	A ref (`main`), `none` / `self` / `feature` / `single-arm` for one arm with absolute bars, or `{source, values: {metric: number}}` for published numbers that replace the baseline arm.
`metrics`	list	The criteria. See below.
`policy`	map	`thresholds: {metric: value}` (per-metric `threshold` wins), `auto_promote: off	on_pass`,` guardrail_veto: true`.
`dataset_ref`	string	Recorded on the run.
`compute`	map	Accepted here as well as at the top level.

suite forms

Form	Runs
`lighteval:<task>`	A lighteval task. Defaults the tier to GPU.
`command:<cmd>` or a bare command	A shell command on each arm. It must print JSON metrics.
`path/to/script.py`	A script on each arm.
`notebook: {path, parameters, packages, commit, commit_path, max_bytes}` or `notebook:path.ipynb`	A notebook under papermill. `commit` defaults to `true`; `max_bytes` to 2,000,000.
`harness: {runner, notebook, experiments, workdir, results_glob, method, packages, base_image, preimport, smoke}` or `harness:path/run.py`	The repo’s own eval apparatus. `results_glob` is read as a published baseline and as a duration source. `smoke: false` skips the smoke stage.

metrics[]

Key	Values	Default	Meaning
`key` or `name`	string	required	The metric the test emits.
`direction`	`>=`, `<=`, or words like `increase` / `lower`	`>=`	Which way is better.
`threshold`	number	`0`, or `policy.thresholds[key]`	Your bar.
`role`	`target`, `guardrail`, `cost`, `latency`, `diagnostic`	`target`	`primary` / `main` read as target; `report` / `observe` as diagnostic.
`bar`	`goal`, `floor`, `sanity`	`goal`	How the threshold reads in the verdict.
`reads_as`	string	none	One line the table shows next to the metric.

Target rows gate with warn severity, guardrails with fail severity. Cost, latency and diagnostic rows are measured and shown but never gate; a threshold on them is a bound shown for context.

`compute`

Key	Default	Meaning
`tier`	`cpu`	`cpu` or `gpu`.
`gpu`	follows `tier`	Kept consistent with `tier`.
`timeout_s`	`auto`	Measurement budget in seconds. `auto` is the estimate times `timeout_padding`, else 1800. Clamped to 60 s and `timeout_max_s`.
`timeout_padding`	`1.5`	Multiplier applied to an estimate. Not applied to a pinned `timeout_s`.
`timeout_max_s`	`14400`	The ceiling `auto` may never exceed. Raise it yourself to consent to more.
`rigor`	derived	`A` (scaled, directional) or `B` (the repo’s own harness at its own regime).
`budget_cap_usd`	none	A spend ceiling the fix loop checks before each dispatch.

`loop`

Key	Default	Meaning
`max_iterations`	`8`	Per stage (smoke, then measurement). Hard ceiling 10.
`fix_env`	`true`	Install missing packages and re-run.
`fix_code`	`true`	Edit the repo and re-run. `false` returns Validate to reporting only.
`fix_spec`	`true`	Adjust the spec (a wrong results path, for instance).
`budget_cap_usd`	none	Spend ceiling for the loop.

The loop can never edit the confirmed test or the criteria, and a second identical failure after a fix stops it regardless of these settings.

Examples

1. Does it work?

Tests only. The verdict is Your tests pass or do not pass.

checks:
  command: pytest tests/test_scheduler.py -x -q

2. Compare with a guardrail

Both arms run on the base branch and the PR; the target must improve and the guardrail must not regress.

question:
  kind: compare
  ask: "does the fused attention kernel speed up decoding without changing outputs?"
benchmarks:
  - name: decode-bench
    suite: command:python bench/decode.py --json
    baseline: main
    metrics:
      - key: tokens_per_s
        direction: ">="
        threshold: 120
        role: target
      - key: output_match_rate
        direction: ">="
        threshold: 0.999
        role: guardrail
      - key: peak_memory_gb
        direction: "<="
        role: cost
    policy:
      guardrail_veto: true
seeds: 2
compute:
  tier: gpu
  timeout_s: 1200

3. Parity with published numbers

No baseline arm. The reference values come from the repo’s own results corpus.

question:
  kind: parity
  ask: "does the new config reach the MetaMathQA accuracy the maintainers publish?"
benchmarks:
  - name: metamath-parity
    suite:
      harness:
        runner: method_comparison/MetaMathQA/run.py
        experiments: experiments/supertuning
        results_glob: results/*.json
    baseline:
      source: method_comparison/MetaMathQA/results/full-finetuning.json
      values: {metamath_accuracy: 0.62, peak_memory_gb: 31.2}
    metrics:
      - key: metamath_accuracy
        direction: ">="
        threshold: 0.60
        role: target
      - key: peak_memory_gb
        direction: "<="
        threshold: 32
        role: guardrail
compute:
  tier: gpu
  rigor: B
  timeout_s: 7200
  timeout_max_s: 10800

4. Capability with no baseline

The base branch cannot run this code, so one arm runs against absolute bars. Illustrative configuration.

question:
  kind: capability
  ask: "does the new forecaster train and beat the naive baseline on the sample set?"
benchmarks:
  - name: forecaster-smoke
    suite: python-script
    baseline: none
    metrics:
      - key: mase
        direction: "<="
        threshold: 1.0
        role: target
        reads_as: "below 1.0 beats the seasonal naive forecast"
      - key: train_steps
        direction: ">="
        threshold: 50
        role: diagnostic
        bar: sanity

5. GPU notebook with a smoke path and W&B

A notebook that reads REMYX_SMOKE=1 gets a smoke stage before the full run. The W&B block pulls your connected key into the pod.

question:
  kind: compare
  ask: "does rank-16 LoRA match rank-32 on eval loss at half the trainable parameters?"
benchmarks:
  - name: lora-rank16
    suite:
      notebook:
        path: .remyx/validations/lora-rank16.ipynb
        parameters: {max_steps: 400, eval_every: 50}
        packages: [peft==0.14.0]
        commit: true
        max_bytes: 2000000
    baseline: main
    metrics:
      - key: eval_loss
        direction: "<="
        threshold: 1.25
        role: target
      - key: trainable_params
        direction: "<="
        threshold: 3500000
        role: guardrail
variants:
  feature:
    env: {LORA_RANK: "16"}
  baseline:
    env: {LORA_RANK: "32"}
seeds: 3
compute:
  tier: gpu
  timeout_s: auto
  timeout_padding: 1.5
logging:
  wandb:
    entity: my-team
    project: lora-rank-study
loop:
  max_iterations: 6
  budget_cap_usd: 15

The notebook reads REMYX_SMOKE and, when set, runs a handful of steps and still emits every declared metric. The smoke run posts no verdict and is never charged.

Lint, push, and the PR

Edit the spec from the repo’s spec card in Studio. The editor lints against the engine’s own parser as you type and Open a PR commits the yaml to a new branch and opens a pull request as remyx-ai[bot]. It never writes to the default branch; nothing changes until that PR merges. The same path is POST /api/v1.0/validate/specs/push {repo, yaml, project_id?}.

auto_promote never changes PR state unless you opt in. The default off reports only. on_pass marks the PR ready for review when the verdict is a pass.

Next steps

Run a validation tutorial · Read the validation spec · Use the API

Validate

How a validation runs and how to read the verdict

Billing

What a run costs and when it is charged

GET STARTED

WORKFLOWS

HOW IT WORKS

CONFIGURE

REFERENCE

Start with a working example

Full schema

Location and precedence

Key reference

Top level

`question`

`benchmarks[0]`

`compute`

`loop`

Examples

1. Does it work?

2. Compare with a guardrail

3. Parity with published numbers

4. Capability with no baseline

5. GPU notebook with a smoke path and W&B

Lint, push, and the PR

Next steps

Validate

Billing

​Start with a working example

​Full schema

​Location and precedence

​Key reference

​Top level

​question

​benchmarks[0]

​compute

​loop

​Examples

​1. Does it work?

​2. Compare with a guardrail

​3. Parity with published numbers

​4. Capability with no baseline

​5. GPU notebook with a smoke path and W&B

​Lint, push, and the PR

​Next steps

​Related

Validate

Billing

Start with a working example

Full schema

Location and precedence

Key reference

Top level

`question`

`benchmarks[0]`

`compute`

`loop`

Examples

1. Does it work?

2. Compare with a guardrail

3. Parity with published numbers

4. Capability with no baseline

5. GPU notebook with a smoke path and W&B

Lint, push, and the PR

Next steps

Related