syne_tune.optimizer.schedulers.multiobjective.moasha module

class syne_tune.optimizer.schedulers.multiobjective.moasha.MOASHA(config_space, metrics, do_minimize=True, time_attr='training_iteration', multiobjective_priority=None, max_t=100, grace_period=1, reduction_factor=3, brackets=1, random_seed=None)

Bases: TrialScheduler

Implements MultiObjective Asynchronous Successive HAlving with different multiobjective sort options. References:

A multi-objective perspective on jointly tuning hardware and hyperparameters

David Salinas, Valerio Perrone, Cedric Archambeau and Olivier Cruchant

NAS workshop, ICLR2021.

and

Multi-objective multi-fidelity hyperparameter optimization with application to fairness

Robin Schmucker, Michele Donini, Valerio Perrone, Cédric Archambeau

Parameters:

• config_space (Dict[str, Any]) – Configuration space

• metrics (List[str]) – List of metric names MOASHA optimizes over

• do_minimize (Optional[bool]) – If True, we minimize the objective function specified by metric . Defaults to True.

• time_attr (str) – A training result attr to use for comparing time. Note that you can pass in something non-temporal such as training_iteration as a measure of progress, the only requirement is that the attribute should increase monotonically. Defaults to “training_iteration”

• multiobjective_priority (Optional[MOPriority]) – The multiobjective priority that is used to sort multiobjective candidates. We support several choices such as non-dominated sort or linear scalarization, default is non-dominated sort.

• max_t (int) – max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have passed. Defaults to 100

• grace_period (int) – Only stop trials at least this old in time. The units are the same as the attribute named by time_attr. Defaults to 1

• reduction_factor (float) – Used to set halving rate and amount. This is simply a unit-less scalar. Defaults to 3

• brackets (int) – Number of brackets. Each bracket has a different grace_period and number of rung levels. Defaults to 1

• random_seed (Optional[int]) – Seed for the random number generator

metric_names()

Return type:

List[str]

metric_mode()

Return type:

str

suggest()

Returns a suggestion for a new trial, or one to be resumed

This method returns suggestion of type TrialSuggestion (unless there is no config left to explore, and None is returned).

If suggestion.spawn_new_trial_id is True, a new trial is to be started with config suggestion.config. Typically, this new trial is started from scratch. But if suggestion.checkpoint_trial_id is given, the trial is to be (warm)started from the checkpoint written for the trial with this ID. The new trial has ID trial_id.

If suggestion.spawn_new_trial_id is False, an existing and currently paused trial is to be resumed, whose ID is suggestion.checkpoint_trial_id. If this trial has a checkpoint, we start from there. In this case, suggestion.config is optional. If not given (default), the config of the resumed trial does not change. Otherwise, its config is overwritten by suggestion.config (see HyperbandScheduler with type="promotion" for an example why this can be useful).

Apart from the HP config, additional fields can be appended to the dict, these are passed to the trial function as well.

Return type:

Optional[TrialSuggestion]

Returns:

Suggestion for a trial to be started or to be resumed, see above. If no suggestion can be made, None is returned

on_trial_add(trial)

Called when a new trial is added to the trial runner.

Additions are normally triggered by suggest.

Parameters:

trial (Trial) – Trial to be added

on_trial_result(trial, result)

Called on each intermediate result reported by a trial.

At this point, the trial scheduler can make a decision by returning one of SchedulerDecision.CONTINUE, SchedulerDecision.PAUSE, or SchedulerDecision.STOP. This will only be called when the trial is currently running.

Parameters:

• trial (Trial) – Trial for which results are reported

• result (Dict[str, Any]) – Result dictionary

Return type:

str

Returns:

Decision what to do with the trial

metric_dict(reported_results)

Return type:

Dict[str, Any]

check_metrics_are_present(result)

on_trial_complete(trial, result)

Notification for the completion of trial.

Note that on_trial_result() is called with the same result before. However, if the scheduler only uses one final report from each trial, it may ignore on_trial_result() and just use result here.

Parameters:

• trial (Trial) – Trial which is completing

• result (Dict[str, Any]) – Result dictionary

on_trial_remove(trial)

Called to remove trial.

This is called when the trial is in PAUSED or PENDING state. Otherwise, call on_trial_complete().

Parameters:

trial (Trial) – Trial to be removed

metadata()

Return type:

Dict[str, Any]

Returns:

Metadata for the scheduler