Results and evidence

A validation result connects the question, executed test, observed metrics, criteria, artifacts, and cost. Read the PR comment for the summary. Use the GitHub Check for execution status and the notebook and run artifacts for the underlying work. Results also appear in Studio under Reports → Validations.

How to read a result

Read the question, comparison, measured metrics, acceptance criteria and supporting artifacts together. A passing result means the required criteria were met in that run. It does not prove production improvement. An inconclusive result, a no-signal result, an execution error and a timeout have different meanings. Use the detailed status definitions below before deciding what to change or rerun. Interpret evaluation results

Measure	Measured value	Required criterion	Scope of evidence
Explained variance	TopK 0.9666, AbsTopK 0.9677	AbsTopK no more than 0.02 worse	Single-seed CPU run
Negative selected latents	TopK 0%, AbsTopK 49.2%	AbsTopK at least 1%	Single-seed CPU run

A missed criterion means the test did not meet its specified target. A diagnostic metric supplies context but does not determine the outcome unless the spec makes it required. An inconclusive or no-signal result means the run did not establish a reliable answer under its chosen test and data. Review missing metrics, comparability, uncertainty, or execution limitations before treating the result as a decision input.

Comparisons

Some validations compare base and PR branches. Others test a capability against an absolute threshold or reference value. The result identifies which comparison ran.

Artifacts

Executed notebooks are committed to the PR branch by default. If the notebook suite sets commit: false, Validate provides time-limited artifacts instead. Review artifacts before relying on the summary.

GitHub checks and merging

A passing or failing result is not an automatic merge decision. The Remyx Validation check affects merge protection only when your repository requires it.

Result fields and status reference

The tables below define exact result fields, outcome labels, and status semantics.

Status	Meaning
✅ Met your criteria	The target passed and no guardrail crossed.
⚠️ Met your criteria, with a warning	The target passed; something else needs a look.
❌ Missed your criteria	The target fell short of your bar.
❌ Missed your criteria, guardrail crossed	A guardrail moved past its line. Flagged for review.
⚠️ Inconclusive, `<metric>` was not measured	The test did not produce a metric the result needs.
📊 Measured	The target’s bar is a floor or a sanity check, not a goal. Read the numbers.
🔍 No signal	The run cannot answer the question. See No signal below.
⏱️ Ran out of time	The budget ended before a result. Says nothing about your code.
✅ / ❌ Your tests pass / do not pass on this PR	A `checks:` only spec.

Every result comment follows one layout: the status line, your ask, a one-sentence answer, what was found, the metric table, then the details.

Part	What it tells you
Status	The outcome in one line, from the table above.
You asked	Your own words, when this thread is where the ask came from. An inherited spec reports what it ran instead.
Answer	The target metric, the comparison, and whether your bar was met. A crossed guardrail or a metric the test never produced is named here.
What we found	Findings from the run. A finding whose metric was not emitted is dropped and listed under Not measured.
Table	One row per metric: direction, baseline, this PR, delta, your bar, pass mark. Guardrail, cost and diagnostic rows carry a legend.
Links	The executed notebook on GitHub, Colab and nbviewer; W&B dashboards when the run logged there.
Footer	What the run cost and what it was charged to. See the cost footer.

When a fix loop ran, a What I tried table lists each attempt, what failed, and the commit that tried to fix it. When the loop stopped, the first line says why: the test itself failed, a file is missing from the branch, or the circuit breaker tripped.

Illustrative example of a result comment

Illustrative example. The numbers are invented to show the layout. For a real one, open the completed validation above.

## ✅ Met your criteria

**You asked:** does spectral targeting halve trainable params without losing fit vs standard LoRA?

**Answer:** `gsm8k_accuracy` came in at **0.412**, vs **0.407** on `main` (+0.005). Your bar was ≥ 0.40 — met.

**What we found**
- Trainable parameters fell from 6.8M to 3.4M with the same rank.
- Eval loss moved inside seed noise across 3 seeds.

| Metric | Goal | main | this PR | Δ | Your bar | |
|---|---|---|---|---|---|---|
| `gsm8k_accuracy` | ↑ | 0.407 | 0.412 | +0.005 | ≥ 0.40 | ✅ |
| `trainable_params` | ↓ | 6.8M | 3.4M | -3.4M | ≤ 3.5M | ✅ |
| `eval_loss` 🛡️ | ↓ | 1.21 | 1.22 | +0.01 | ≤ 1.30 | ✅ |

🛡️ guardrail — a metric that must not regress, even if the target improves.

Notebook: [GitHub](…) · [Colab](…) · [nbviewer](…)

---
💳 **$1.84 debited from your Remyx balance — $18.16 left.**

Evidence strength

Tier A is the default answer: a scaled workload at a matched budget, labelled directional. Tier B is the repo’s own harness at its own regime, what a maintainer asks for. Declare compute.rigor: A|B; otherwise a reused harness is Tier B and everything else is Tier A. A smoke run is neither.

No signal

Before any result, deterministic checks compare what each arm declared with what it reported. Two arms that declare different settings and report identical provenance, a factor that drifted, or a difference inside the noise produce No signal, this run can’t answer the question instead of a confident number.

Targets, guardrails, and diagnostics

Role	Gates the result	Shown as
`target`	Yes (warn)	The headline row
`guardrail`	Yes (fail, with `policy.guardrail_veto`)	🛡️ must not regress even if the target improves
`cost`, `latency`	No	ℹ️ measured and reported
`diagnostic`	No	📊 measured to inform the reading

A metric’s bar is a goal (the default), a floor (a bound the measurement cannot fall below; missing it points at the harness) or a sanity check on the harness.

Basis	Footer
Free validation	🎁 That was this installation’s free validation. What it would have cost, and the two ways on: add credits, or a team pilot.
Balance	💳 What was debited and what is left. Add credits appears when the balance is below the larger of that debit and $5.
Counted grant	How many validations are left on the grant.
Unlimited grant	No footer.

Next steps

Run a validation tutorial · Read the validation spec · Use the API

GET STARTED

WORKFLOWS

HOW IT WORKS

CONFIGURE

REFERENCE

Results and evidence

How to read a result

Comparisons

Artifacts

GitHub checks and merging

Result fields and status reference

Evidence strength

No signal

Targets, guardrails, and diagnostics

The cost footer

Next steps

​How to read a result

​Comparisons

​Artifacts

​GitHub checks and merging

​Result fields and status reference

​Evidence strength

​No signal

​Targets, guardrails, and diagnostics

​The cost footer

​Next steps

How to read a result

Comparisons

Artifacts

GitHub checks and merging

Result fields and status reference

Evidence strength

No signal

Targets, guardrails, and diagnostics

The cost footer

Next steps