Skip to main content

Checks

Define how submitted results are evaluated.

A check assesses one part of a requirement. For the quantized classifier, one check tests whether accuracy reaches 90%; another tests whether p95 latency stays within 20 ms.

Add rules for recorded metrics, parameters, or tags. Choose No rules when the evidence needs human assessment, such as whether a dataset covers the conditions in the requirement.

Add a check#

Open the requirement’s Checks tab and select Add check. This example assesses the latency recorded by a profiling run:

  1. Describe the expected result: p95 latency of at most 20 ms on the target device.
  2. Select Profile so the check evaluates profiling runs.
  3. Choose Rules and set latency_p95_ms to At most 20.
  4. Select Add check to save.

Choose which runs to evaluate#

Selected types limits evaluation to matching run types within a submission. All runs evaluates every submitted run.

For a submission containing a profile run and an evaluation run, use a Profile filter for the latency check and an Eval filter for the accuracy check. Otherwise, the latency check would also look for latency data in the evaluation run.

Configure a rule#

A metric rule compares the latest recorded value with a threshold:

  1. Choose Metric and enter the exact logged Key.
  2. Set At least or At most and the threshold Value.
  3. Use Preview to check the rule against recorded runs.
  4. Select Save rule.

For a Parameter or Tag rule, enter the key and expected value. Leave Value blank to require the key to exist with any value.

Warning

Use the same units as the recorded metric. An accuracy logged as 0.92 needs a threshold such as 0.90, not 90. The Hub does not convert units.

Interpret results#

Drafts evaluate the current evidence against the requirement’s checks. Submitted revisions show their saved results. Expand a check to compare the observed value with the threshold. For example, a profile result of 18.4 ms passes the 20 ms latency rule. A check passes only when every matching submitted run satisfies every rule in that check.

ResultWhat to inspect
PassConfirm that the runs, model, and test conditions are appropriate to the requirement.
FailCompare the observed value with the configured expectation.
Missing dataCheck the run filter and whether the matching runs contain the required key.
Could not evaluateCheck whether the check has rules and whether the recorded value can be evaluated.

A check without rules has no automated pass or fail result. The reviewer assesses it using the description and submitted evidence.

Change a check#

Edit a check from the requirement’s Checks tab. After changing its rules or run filter, inspect the draft evaluation again before opening it for review.

Draft evaluations use the current configuration. Existing submission revisions preserve the checks and evaluation results captured when they were opened for review.

Next steps#