Skip to main content

Safe integration

Choose how tracking failures affect your job.

By default, a failed tracking request raises an exception. If you do not handle it, your script stops. This helps you catch connection and configuration problems while setting up.

For a long training job or a shared pipeline, decide whether tracking failures should stop the work or let it continue with an incomplete record.

Handle failures, or use safe mode#

If a complete record matters, handle failures where they happen. You might save results locally, alert your team, retry after checking what reached the Hub, or stop the job. Make missing results visible so they do not go unnoticed.

Use safe mode when the workload should continue despite tracking failures. The SDK reports a warning instead of raising an exception for the operations you mark as safe. The tradeoff is that a run may have fewer results than you expected, or may not exist at all. Monitor the warnings before treating the record as complete.

What safe mode covers#

Safe mode covers failures in SDK tracking operations, such as an unreachable Hub or a rejected request. Each produces a TrackingFailureWarning naming the operation and underlying error at the calling line in your script.

Exceptions from your own code still propagate. Invalid SDK usage, such as starting a run inside another run, still raises.

Safe mode does not queue or replay failed requests. A warning does not prove that the data was lost: the Hub may have saved it before the response failed. Check the recorded results before retrying to avoid duplicates.

Track safely from Python#

from embedl_hub.tracking import Client

client = Client()
client.set_project("quantized-classifier", safe=True)

with client.start_run("training", name="Nightly training", safe=True) as run:
    with client.safe_log():
        for step, loss in enumerate([0.8, 0.5, 0.3]):
            client.log_metric("loss", loss, step=step)

Use all three settings for this workflow. safe=True on set_project covers selecting or creating the project. On start_run, it covers creating and finishing the run. safe_log() covers the logging calls inside it.

If the run cannot be created, run is None; the body still executes and its logging calls warn. Replace the sample values with your training code and measurements.

For decorated functions, @hub_run("eval", safe=True) lets the function run when its tracking run cannot be started. Its Tracked result then has .run set to None. Logging inside the function still needs safe_log(), and selecting the project still needs its own safe=True.

Track safely from the shell#

The CLI has a safe-mode switch only on exec:

embedl-hub exec --safe --type training python train.py

exec --safe runs the command even if tracking cannot start and returns the command’s exit status. It sets EMBEDL_HUB_SAFE=1 for the child process. Python code that adopts the wrapper’s run with client.use_inherited_run() uses that setting to tolerate tracking failures. If there is no run to adopt, it warns and returns None.

Logging is a separate step: use client.safe_log() around Python logging calls inside the command. Other CLI commands, including run start and embedl-hub log, have no safe-mode switch.

To keep log entries locally, use a batch. Entries stay on the machine until sent successfully. Creating and finishing a run still requires a connection to the Hub.

Next steps#

  • Direct tracking for what a run can record.
  • CLI for wrapping commands and queueing entries.