Skip to main content
Measure a change against defined criteria before merging it. Validate proposes and runs an evaluation on a pull request in a connected GitHub repository. It works on PRs written by your team, another coding agent, or Outrider.

​
Run a validation

  1. Connect the repository through the Remyx GitHub App and enable Validations.
  2. Comment @remyx-ai validate on an open pull request to request a test.
  3. Review the proposed test and criteria. Confirm the run when the request requires a separate approval.
  4. Inspect the measured results, executed notebook, and cost on the pull request.
A committed validation spec can start a run directly. An explicit request to revise and run may also skip separate confirmation. Run your first validation · Command reference
GitHub PR conversation showing an @remyx-ai validate request and the bot acknowledging that it is designing a proposed test.

Request an evaluation directly from a GitHub pull request.

​
What gets measured

Some validations compare the base branch with the PR. Others test a new capability against an absolute threshold or reference. The result identifies the comparison used and which criteria were met.

​
See a completed result

The SAELens validation compared TopK and AbsTopK sparse autoencoders under two offline criteria. In the recorded single-seed CPU run, both criteria were met.
Validate's GitHub result for SAELens, with a passing verdict, parity-gap metric, negative-latent metric, and evidence table.

The SAELens validation met both specified offline criteria in one CPU run.

This result supports the specified test criteria. It does not establish production improvement or downstream steering and monitoring performance. Read the GitHub result · Open the notebook

​
What changes on the PR

Validate commits proposed test files and configuration to the PR branch. Executed notebooks are committed by default. Automatic repair may also edit repository code by default, but cannot change the confirmed test or criteria. Execution and safeguards

​
Cost

The first completed eligible validation is free per GitHub App installation. Later completed runs use prepaid credits or an applicable grant. A valid inconclusive result may still count as a completed, chargeable run. Billing · Results and evidence · Compute and limits