syne_tune.remote.remote_metrics_callback module

class syne_tune.remote.remote_metrics_callback.RemoteTuningMetricsCallback(metric, mode, config_space=None, resource_attr=None)

Bases: TunerCallback

Reports metrics related to the experiment run by Tuner. With remote tuning, if these metrics are registered with the SageMaker estimator running the experiment, they are visualized in the SageMaker console. Metrics reported are:

• BEST_METRIC_VALUE: Best value of metric reported to tuner so far

• BEST_TRIAL_ID: ID of trial for which the best metric value was reported so far

• BEST_RESOURCE_VALUE: Resource value for which the best metric value was reported so far. Only if resource_attr is given

• If config_space is given, then for each hyperparameter name in there (entry with domain), we add a metric BEST_HP_PREFIX + name. However, at most MAX_METRICS_SUPPORTED_BY_SAGEMAKER are supported

static get_metric_names(config_space, resource_attr=None)

register_metrics_with_estimator(estimator)

Registers metrics reported here at SageMaker estimator estimator. This should be the one which runs the remote experiment.

Note: The total number of metric definitions must not exceed MAX_METRICS_SUPPORTED_BY_SAGEMAKER. Otherwise, only the initial part of metric_names is registered.

Parameters:

estimator (EstimatorBase) – SageMaker estimator to run the experiment

on_tuning_start(tuner)

Called at start of tuning loop

Parameters:

tuner – Tuner object

on_trial_result(trial, status, result, decision)

Called when a new result (reported by a trial) is observed

The arguments here are inputs or outputs of scheduler.on_trial_result (called just before).

Parameters:

• trial (Trial) – Trial whose report has been received

• status (str) – Status of trial before scheduler.on_trial_result has been called

• result (Dict[str, Any]) – Result dict received

• decision (str) – Decision returned by scheduler.on_trial_result