> ## Documentation Index
> Fetch the complete documentation index at: https://docs.remyx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation spec

> Configure the question, test, metrics, criteria, compute and repair behavior for a PR validation.

A validation spec records what to measure, how to run the test, and how the result will be judged.

A PR can use a committed `.remyx/validation.yaml` file or ask Validate to propose one. Review the comparison, required metrics and criteria before execution.

## Start with a working example

The minimal example below uses a test or command that exists in the linked repository. Copy its files together and check the expected metric output before adapting it.

```yaml .remyx/validation.yaml theme={null}
question:
  kind: capability
  ask: "Does AbsTopK recover negative signals without hurting reconstruction? Train one TopKTrainingSAE and one AbsTopKTrainingSAE on tiny-stories-1M activations with the same k, width, seed and tokens under 30 minutes on CPU, and compare run_evals explained_variance and L0 side by side."
loop:
  max_iterations: 8
  fix_code: true
benchmarks:
  - name: abstopk-vs-topk-tinystories
    suite:
      notebook:
        path: "eval/eval_abstopk_vs_topk.ipynb"
        packages: [wandb]
    baseline: none
    scorer: explained_variance_gap__topk_minus_abstopk
    policy:
      guardrail_veto: true
    metrics:
      - name: explained_variance_gap__topk_minus_abstopk
        direction: min
        threshold: 0.02
        role: target
        bar: goal
        reads_as: "How much of TopK's held-out explained_variance AbsTopK fails to recover; 0 or negative means parity or better."
      - name: abstopk_negative_latent_share
        direction: max
        threshold: 0.01
        role: guardrail
        bar: goal
        reads_as: "Share of AbsTopK's k selected latents that are negative on held-out tokens; TopK's is 0 by construction."
compute:
  tier: cpu
  timeout_s: 2700
requires: [wandb]
```

This example includes the test files it runs. Copy the spec and its referenced files together, then adjust the repository paths, data and criteria for your change.

[Full spec on the branch](https://github.com/salma-remyx/SAELens/blob/6989234239f49130fd1e77412f04a0513268fc87/.remyx/validation.yaml) · [The notebook it runs](https://github.com/salma-remyx/SAELens/blob/6989234239f49130fd1e77412f04a0513268fc87/eval/eval_abstopk_vs_topk.ipynb) · [Executed notebook](https://github.com/salma-remyx/SAELens/blob/6989234239f49130fd1e77412f04a0513268fc87/.remyx/validations/abstopk-vs-topk-tinystories.ipynb) · [Read the GitHub result](https://github.com/salma-remyx/SAELens/pull/5)

## Full schema

The reference below defines supported questions, suites, metrics, policy, compute, repair, model selection, artifacts and defaults.

## Location and precedence

| Where | Used when |
| - | - |
| `.remyx/validation.yaml` at the PR head | Always first. A spec on the branch is what runs. |
| A locked eval template on the project | When the branch has no spec. |
| A proposal | When neither exists: the bot designs one, commits it to the branch, and waits for `@remyx-ai validate`. |

Rules the parser applies before anything runs:

* A file that only carries `model:` is configuration, not a spec. Its `model:` still applies to the design.
* A flat spec (benchmark keys at the top level) is lifted into a one-entry `benchmarks:` list. Both shapes are accepted.
* Keys the proposer sometimes writes under the benchmark (`compute`, `metrics`, `policy`, `baseline`, `variants`) are hoisted; top level wins on collision.
* A hand-written spec is never revised by feedback on the PR. Edit the file and mention the bot with `validate`.
* With a committed spec, instructions in your mention are ignored and the started comment says so.

Lint before you commit: `POST /api/v1.0/validate/specs/lint {yaml}` runs the same parser and returns problems; the Studio spec editor uses it on every keystroke.

***

## Key reference

### Top level

| Key | Type | Default | Meaning |
| - | - | - | - |
| `name` | string | `validation` | Label for the run. |
| `question` | map | derived | `kind` and `ask`. See below. |
| `checks` | map or `true` | none | "Does it work?" `command` to run on the feature arm; `true` means `pytest -x -q`. |
| `benchmarks` | list | none | "Does it do what it claims?" Only the first entry runs today. |
| `variants` | map | none | Per-arm env overrides: `{feature: {env: {LR: "1e-4"}}}`. Reserved platform keys are dropped. |
| `seeds` | int | `1` | How many seeds per arm. Multiplies the arms. |
| `compute` | map | `{tier: cpu}` | Tier, budget and rigor. See below. |
| `loop` | map | all on | The fix loop's bounds. See below. |
| `model` | map or string | none | `provider` and `name` of the model that designs and repairs. `model: zai` is shorthand. |
| `requires` | list or string | none | Connected accounts the run needs, e.g. `[wandb]`. A `logging.<provider>` block implies it. |
| `logging` | map | none | `wandb: {entity, project, base_url}`. See [W\&B](https://docs.remyx.ai/platform/agents/validate-execution.md#logging-to-wb). |
| `held_constant` | list | none | Factors the design holds fixed. Checked against provenance by the no-signal check. |
| `avoid` | list | none | What the design must not do. |
| `provenance` | map | none | Free-form provenance the report echoes. |
| `production` | map | none | Production signal wiring for calibration. |
| `report` | map | none | `headline`, `caveat`, `next`, `findings[]`, `establishes[]`, `does_not_establish[]`, `not_measured[]`. Templates may reference metrics with `{metric}`; a finding whose metric was never emitted is dropped. |

### `question`

| Key | Values | Meaning |
| - | - | - |
| `kind` | `compare`, `parity`, `capability`, `regression` | How many arms run and what the baseline is. Derived when absent: `baseline: none` → capability, a published baseline → parity, benchmarks → compare, checks only → works. |
| `ask` | string | The request this test answers, in the requester's words. Reported only when this PR is where the ask came from. |

| `question.kind` | Asks | Arms | Baseline |
| - | - | - | - |
| `compare` | Did the PR change a measured outcome against the base branch? | baseline + feature | the base branch |
| `parity` | Does it match a published reference or known result? | feature only | published numbers |
| `capability` | Does the new capability meet an absolute criterion? | feature only | none, absolute bars |
| `regression` | Does a refactor stay within the allowed tolerance? | baseline + feature | the base branch |

**Baselines.** The baseline arm is skipped when the repo publishes its own numbers (`baseline:` with a reference dict, or a results corpus the harness reads) or when a prior validation on the same base commit and the same spec wiring already measured it. Reuse is exact-match only; anything less re-runs the arm. `baseline: none` runs one arm with absolute bars.

### `benchmarks[0]`

| Key | Type | Meaning | |
| - | - | - | - |
| `name` | string | Names the committed test, notebook and Dockerfile stems. | |
| `suite` | string or map | What runs. Forms below. | |
| `baseline` | string or map | A ref (`main`), `none` / `self` / `feature` / `single-arm` for one arm with absolute bars, or `{source, values: {metric: number}}` for published numbers that replace the baseline arm. | |
| `metrics` | list | The criteria. See below. | |
| `policy` | map | `thresholds: {metric: value}` (per-metric `threshold` wins), \`auto\_promote: off | on\_pass`, `guardrail\_veto: true\`. |
| `dataset_ref` | string | Recorded on the run. | |
| `compute` | map | Accepted here as well as at the top level. | |

**`suite` forms**

| Form | Runs |
| - | - |
| `lighteval:<task>` | A lighteval task. Defaults the tier to GPU. |
| `command:<cmd>` or a bare command | A shell command on each arm. It must print JSON metrics. |
| `path/to/script.py` | A script on each arm. |
| `notebook: {path, parameters, packages, commit, commit_path, max_bytes}` or `notebook:path.ipynb` | A notebook under papermill. `commit` defaults to `true`; `max_bytes` to 2,000,000. |
| `harness: {runner, notebook, experiments, workdir, results_glob, method, packages, base_image, preimport, smoke}` or `harness:path/run.py` | The repo's own eval apparatus. `results_glob` is read as a published baseline and as a duration source. `smoke: false` skips the smoke stage. |

**`metrics[]`**

| Key | Values | Default | Meaning |
| - | - | - | - |
| `key` or `name` | string | required | The metric the test emits. |
| `direction` | `>=`, `<=`, or words like `increase` / `lower` | `>=` | Which way is better. |
| `threshold` | number | `0`, or `policy.thresholds[key]` | Your bar. |
| `role` | `target`, `guardrail`, `cost`, `latency`, `diagnostic` | `target` | `primary` / `main` read as target; `report` / `observe` as diagnostic. |
| `bar` | `goal`, `floor`, `sanity` | `goal` | How the threshold reads in the verdict. |
| `reads_as` | string | none | One line the table shows next to the metric. |

Target rows gate with warn severity, guardrails with fail severity. Cost, latency and diagnostic rows are measured and shown but never gate; a threshold on them is a bound shown for context.

### `compute`

| Key | Default | Meaning |
| - | - | - |
| `tier` | `cpu` | `cpu` or `gpu`. |
| `gpu` | follows `tier` | Kept consistent with `tier`. |
| `timeout_s` | `auto` | Measurement budget in seconds. `auto` is the estimate times `timeout_padding`, else 1800. Clamped to 60 s and `timeout_max_s`. |
| `timeout_padding` | `1.5` | Multiplier applied to an estimate. Not applied to a pinned `timeout_s`. |
| `timeout_max_s` | `14400` | The ceiling `auto` may never exceed. Raise it yourself to consent to more. |
| `rigor` | derived | `A` (scaled, directional) or `B` (the repo's own harness at its own regime). |
| `budget_cap_usd` | none | A spend ceiling the fix loop checks before each dispatch. |

### `loop`

| Key | Default | Meaning |
| - | - | - |
| `max_iterations` | `8` | Per stage (smoke, then measurement). Hard ceiling 10. |
| `fix_env` | `true` | Install missing packages and re-run. |
| `fix_code` | `true` | Edit the repo and re-run. `false` returns Validate to reporting only. |
| `fix_spec` | `true` | Adjust the spec (a wrong results path, for instance). |
| `budget_cap_usd` | none | Spend ceiling for the loop. |

The loop can never edit the confirmed test or the criteria, and a second identical failure after a fix stops it regardless of these settings.

***

## Examples

### 1. Does it work?

Tests only. The verdict is **Your tests pass** or **do not pass**.

```yaml theme={null}
checks:
  command: pytest tests/test_scheduler.py -x -q
```

### 2. Compare with a guardrail

Both arms run on the base branch and the PR; the target must improve and the guardrail must not regress.

```yaml theme={null}
question:
  kind: compare
  ask: "does the fused attention kernel speed up decoding without changing outputs?"
benchmarks:
  - name: decode-bench
    suite: command:python bench/decode.py --json
    baseline: main
    metrics:
      - key: tokens_per_s
        direction: ">="
        threshold: 120
        role: target
      - key: output_match_rate
        direction: ">="
        threshold: 0.999
        role: guardrail
      - key: peak_memory_gb
        direction: "<="
        role: cost
    policy:
      guardrail_veto: true
seeds: 2
compute:
  tier: gpu
  timeout_s: 1200
```

### 3. Parity with published numbers

No baseline arm. The reference values come from the repo's own results corpus.

```yaml theme={null}
question:
  kind: parity
  ask: "does the new config reach the MetaMathQA accuracy the maintainers publish?"
benchmarks:
  - name: metamath-parity
    suite:
      harness:
        runner: method_comparison/MetaMathQA/run.py
        experiments: experiments/supertuning
        results_glob: results/*.json
    baseline:
      source: method_comparison/MetaMathQA/results/full-finetuning.json
      values: {metamath_accuracy: 0.62, peak_memory_gb: 31.2}
    metrics:
      - key: metamath_accuracy
        direction: ">="
        threshold: 0.60
        role: target
      - key: peak_memory_gb
        direction: "<="
        threshold: 32
        role: guardrail
compute:
  tier: gpu
  rigor: B
  timeout_s: 7200
  timeout_max_s: 10800
```

### 4. Capability with no baseline

The base branch cannot run this code, so one arm runs against absolute bars.

Illustrative configuration.

```yaml theme={null}
question:
  kind: capability
  ask: "does the new forecaster train and beat the naive baseline on the sample set?"
benchmarks:
  - name: forecaster-smoke
    suite: python-script
    baseline: none
    metrics:
      - key: mase
        direction: "<="
        threshold: 1.0
        role: target
        reads_as: "below 1.0 beats the seasonal naive forecast"
      - key: train_steps
        direction: ">="
        threshold: 50
        role: diagnostic
        bar: sanity
```

### 5. GPU notebook with a smoke path and W\&B

A notebook that reads `REMYX_SMOKE=1` gets a smoke stage before the full run. The W\&B block pulls your connected key into the pod.

```yaml theme={null}
question:
  kind: compare
  ask: "does rank-16 LoRA match rank-32 on eval loss at half the trainable parameters?"
benchmarks:
  - name: lora-rank16
    suite:
      notebook:
        path: .remyx/validations/lora-rank16.ipynb
        parameters: {max_steps: 400, eval_every: 50}
        packages: [peft==0.14.0]
        commit: true
        max_bytes: 2000000
    baseline: main
    metrics:
      - key: eval_loss
        direction: "<="
        threshold: 1.25
        role: target
      - key: trainable_params
        direction: "<="
        threshold: 3500000
        role: guardrail
variants:
  feature:
    env: {LORA_RANK: "16"}
  baseline:
    env: {LORA_RANK: "32"}
seeds: 3
compute:
  tier: gpu
  timeout_s: auto
  timeout_padding: 1.5
logging:
  wandb:
    entity: my-team
    project: lora-rank-study
loop:
  max_iterations: 6
  budget_cap_usd: 15
```

The notebook reads `REMYX_SMOKE` and, when set, runs a handful of steps and still emits every declared metric. The smoke run posts no verdict and is never charged.

***

## Lint, push, and the PR

Edit the spec from the repo's spec card in Studio. The editor lints against the engine's own parser as you type and **Open a PR** commits the yaml to a new branch and opens a pull request as remyx-ai\[bot]. It never writes to the default branch; nothing changes until that PR merges. The same path is `POST /api/v1.0/validate/specs/push {repo, yaml, project_id?}`.

> [!NOTE]
>
> **`auto_promote` never changes PR state unless you opt in.** The default `off` reports only. `on_pass` marks the PR ready for review when the verdict is a pass.

***

## Next steps

[Run a validation tutorial](https://docs.remyx.ai/tutorials/validate-first-run.md) · [Read the validation spec](https://docs.remyx.ai/platform/agents/validate-spec.md) · [Use the API](https://docs.remyx.ai/api-reference/validate.md)

## Related

> [card] [Validate](https://docs.remyx.ai/platform/agents/validate.md)
>
> How a validation runs and how to read the verdict

> [card] [Billing](https://docs.remyx.ai/platform/configure/billing.md)
>
> What a run costs and when it is charged


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.