Skip to main content

Getting started

Review evidence for a quantized model.

Use this example to assess whether a quantized image classifier meets its accuracy and latency limits. You will record two runs, check the measurements, and ask a teammate to review the evidence for one requirement.

The requirement is: The quantized model must achieve at least 90% accuracy on the validation dataset and a p95 inference latency of at most 20 ms on the target device. The p95 latency is the time within which 95% of measured inferences complete.

Before you begin#

To record the example runs, complete SDK setup and select your project.

Use existing runs with equivalent metrics, or record the sample data below.

Record the two runs#

This example logs sample measurements for two runs. Replace my-project with your project name:

from embedl_hub.tracking import Client, RunType

client = Client()
client.set_project("my-project")

with client.start_run(RunType.PROFILE, name="Quantized classifier latency"):
    client.log_param("model_id", "classifier_int8_v1")
    client.log_param("precision", "int8")
    client.log_param("device", "target_device_a")
    client.log_param("batch_size", "1")
    client.log_param("sample_data", "true")
    client.log_metric("latency_p95_ms", 18.4)

with client.start_run(RunType.EVAL, name="Quantized classifier accuracy"):
    client.log_param("model_id", "classifier_int8_v1")
    client.log_param("precision", "int8")
    client.log_param("dataset", "validation_v1")
    client.log_param("sample_data", "true")
    client.log_metric("accuracy", 0.92)

For your own evaluations, record the model identifier, device configuration, and dataset version so the reviewer can identify what was tested.

Open Runs in your project and confirm that both runs have finished. See Direct tracking for more ways to record results and artifacts.

Create the requirement#

  1. Open Requirements in your project and select Add requirement.
  2. Choose Create in Embedl Hub.
  3. Enter Quantized classifier performance as the name.
  4. Add the requirement statement from the introduction as its description. Include the target device and validation dataset used for your own measurements.
  5. Save the requirement.

Add the two checks#

Open the requirement’s Checks tab and select Add check. Use Selected types and Rules for both checks:

  • Meet the latency budget: Profile runs, metric latency_p95_ms, At most 20.
  • Meet the accuracy target: Eval runs, metric accuracy, At least 0.90.
  1. Choose Metric and enter latency_p95_ms as the key.
  2. Set At most to 20.
  3. Use Preview to inspect the recorded result.
  4. Select Save rule, then Add check. Repeat for accuracy with the settings above.

Accuracy is recorded as a fraction (0.90 means 90%); latency is in milliseconds.

Prepare a submission#

  1. Open the profile run and select Submit as evidence.
  2. Add the evaluation run and choose Quantized classifier performance.
  3. Inspect the checks, then select Create draft.

See Submitting evidence for the illustrated steps.

Add a summary explaining why the candidate was chosen and what tradeoffs the reviewer should consider. Draw on notes in the run descriptions and link to any comparisons that informed the decision. See Explain the evidence for an example.

Inspect both check results on the draft. With these sample values, both should pass. Confirm that each check evaluates the intended run and that the evidence refers to the same model.

Open for review#

Save the summary, choose a Reviewer, and select Submit for review. Confirm in the dialog.

See Open for review for the annotated controls. The Hub creates a numbered revision containing the evidence and evaluation results for review.

The reviewer uses Overview to inspect the saved evidence, then selects Review to accept it or request changes. Requesting changes requires feedback. Conversation keeps the review history as the author updates the submission.

If changes are needed, follow Revise a submission.

Next steps#