What Validate can change
Validate writes proposed test files and configuration to the pull request branch. Executed notebooks are committed by default. Automatic repair may edit repository code unless you disable that behavior in the validation spec. A committed spec or an explicit request to revise and run may execute without an additional confirmation. Automatic repair cannot change the confirmed test or acceptance criteria. Review the exact permissions, repair settings, write boundaries, run states and cancellation behavior below.Before you run
Validate uses the repository’s GitHub App installation to write tests, configuration, repairs, and results to the PR branch. It never merges the PR. An owner, member, or collaborator must request execution, but a committed spec or an explicit request to revise and run may start without an additional confirmation. To prevent automatic code edits, setloop.fix_code: false in .remyx/validation.yaml. To stop an active run and its repair loop, comment @remyx-ai cancel.
Command reference · GitHub App permissions
Detailed execution lifecycle
The reference below covers proposals, comparison selection, execution, repair bounds, artifacts, cancellation, and status transitions.From proposal to result
1
Resolve the spec
In order: a committed
.remyx/validation.yaml at the PR head, then a locked eval template on the project, otherwise Validate proposes one and waits. A yaml that only carries model: is configuration, not a spec.2
Propose
The reply is Designing the test now, nothing for you to do yet. Validate then commits a proposed test to the PR branch and posts it: what it measures, the metric and bar, its rationale, and what the design could not cover. Nothing runs until you say so. With “and run it” in your ask it runs as soon as the proposal lands.
3
Confirm
@remyx-ai validate. The reply is Validation started with a cost estimate on the first line, and the Remyx Validation check opens on the PR. Your instructions are ignored at this point if the PR already has committed criteria. The started comment says so.4
Smoke (conditional, never charged)
For a harness with a parameter file, the same test runs first on a few steps. For a notebook that reads
REMYX_SMOKE=1, the notebook runs its smallest end-to-end version of itself. The smoke run proves the environment, the imports and the results file before the budget is spent. It posts no result, and it is never charged. A notebook with no REMYX_SMOKE path goes straight to the full run.5
Measure
Validate chooses a comparison based on the question and the available reference. Some tests compare the base branch with the PR branch. Others test a new capability against an absolute threshold or a published baseline. The run records which comparison it used. The full measurement may use one or more arms, times the declared seeds. Progress is edited into the started comment and streamed to the check.
6
Result
The result posts on the PR, the check completes (success, failure, or neutral for an abstention), the executed notebook is committed, and the run is charged. An errored run posts one comment and completes the check as neutral. A cancelled run marks it cancelled.
What gets committed to your PR
With
commit: false on the notebook suite, the executed notebook is served from the artifact store instead (presigned links, 7 days) and linked from the result. Each run may keep up to 12 artifacts of 64 MB each, readable from GET /validate/runs/<id>.
Every commit is authored by remyx-ai[bot] on the PR branch, never on the default branch. Revert any you do not want.
Fix loop and trust guards
What the loop can never do:
- Edit the confirmed test or the criteria. A loop allowed to move the thresholds it is judged against would optimise the judge.
- Write the metrics. A harness or notebook that assigns the result literals is blocked as forged before it runs, and the rejection goes back to the fixer.
- Install a look-alike.
from utils import xthat resolved to a PyPI package instead of the repo’s own file is triaged as a missing file, not a missing dependency, and never pip-installed.
loop: {fix_code: false} to go back to reporting only.
Logging to W&B
Logging is triggered bylogging.wandb: {entity, project, base_url} or requires: [wandb] in the spec, and detected from the PR template, the harness’s own config, and the repo’s contribution docs. Runs are tagged remyx-validate. Your own key is injected into the measurement pods, never a Remyx key.
Before the run, the started comment shows Where results will land with the provenance of each value. A spec that logs to W&B on an account with no W&B connection, a bad entity or an invalid project is refused at start.