syne_tune.optimizer.schedulers.asha module

class syne_tune.optimizer.schedulers.asha.AsynchronousSuccessiveHalving(config_space, metric, do_minimize=True, searcher='random_search', time_attr='training_iteration', max_t=100, grace_period=1, reduction_factor=3, brackets=1, random_seed=None, searcher_kwargs=None)

Bases: TrialScheduler

Implements Asynchronous Successive Halving. This code is adapted from the RayTune implementation.

References:

Massively Parallel Hyperparameter Tuning L. Li and K. Jamieson and A. Rostamizadeh and K. Gonina and M. Hardt and B. Recht and A. Talwalkar arXiv:1810.05934 [cs.LG]

Parameters:

• config_space (Dict[str, Any]) – Configuration space for the evaluation function.

• metric (str) – Name of metric to optimize, key in results obtained via on_trial_result.

• do_minimize (Optional[bool]) – If True, we minimize the objective function specified by metric . Defaults to True.

• searcher (Union[str, IndependentMultiFidelitySearcher, LastValueMultiFidelitySearcher, None]) – Searcher object to sample configurations.

• time_attr (str) – A training result attr to use for comparing time. Note that you can pass in something non-temporal such as training_iteration as a measure of progress, the only requirement is that the attribute should increase monotonically. Defaults to “training_iteration”

• max_t (int) – max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have passed. Defaults to 100

• grace_period (int) – Only stop trials at least this old in time. The units are the same as the attribute named by time_attr. Defaults to 1

• reduction_factor (float) – Used to set halving rate and amount. This is simply a unit-less scalar. Defaults to 3

• brackets (int) – Number of brackets. Each bracket has a different grace_period and number of rung levels. Defaults to 1

• random_seed (Optional[int]) – Seed for initializing random number generators.

• searcher_kwargs (Optional[dict]) – Additional keyword arguments for the searcher.

suggest()

Returns a suggestion for a new trial, or one to be resumed

This method returns suggestion of type TrialSuggestion (unless there is no config left to explore, and None is returned).

If suggestion.spawn_new_trial_id is True, a new trial is to be started with config suggestion.config. Typically, this new trial is started from scratch. But if suggestion.checkpoint_trial_id is given, the trial is to be (warm)started from the checkpoint written for the trial with this ID. The new trial has ID trial_id.

If suggestion.spawn_new_trial_id is False, an existing and currently paused trial is to be resumed, whose ID is suggestion.checkpoint_trial_id. If this trial has a checkpoint, we start from there. In this case, suggestion.config is optional. If not given (default), the config of the resumed trial does not change. Otherwise, its config is overwritten by suggestion.config (see HyperbandScheduler with type="promotion" for an example why this can be useful).

Apart from the HP config, additional fields can be appended to the dict, these are passed to the trial function as well.

Return type:

Optional[TrialSuggestion]

Returns:

Suggestion for a trial to be started or to be resumed, see above. If no suggestion can be made, None is returned

on_trial_add(trial)

Called when a new trial is added to the trial runner.

Additions are normally triggered by suggest.

Parameters:

trial (Trial) – Trial to be added

on_trial_error(trial)

on_trial_result(trial, result)

Called on each intermediate result reported by a trial.

At this point, the trial scheduler can make a decision by returning one of SchedulerDecision.CONTINUE, SchedulerDecision.PAUSE, or SchedulerDecision.STOP. This will only be called when the trial is currently running.

Parameters:

• trial (Trial) – Trial for which results are reported

• result (Dict[str, Any]) – Result dictionary

Return type:

str

Returns:

Decision what to do with the trial

on_trial_complete(trial, result)

Notification for the completion of trial.

Note that on_trial_result() is called with the same result before. However, if the scheduler only uses one final report from each trial, it may ignore on_trial_result() and just use result here.

Parameters:

• trial (Trial) – Trial which is completing

• result (Dict[str, Any]) – Result dictionary

on_trial_remove(trial)

Called to remove trial.

This is called when the trial is in PAUSED or PENDING state. Otherwise, call on_trial_complete().

Parameters:

trial (Trial) – Trial to be removed

metric_names()

Return type:

List[str]

metric_mode()

Return type:

str

metadata()

Return type:

Dict[str, Any]

Returns:

Metadata for the scheduler

class syne_tune.optimizer.schedulers.asha.Bracket(min_t, max_t, reduction_factor, s, priority=None)

Bases: object

Bookkeeping system to track recorded values.

Rungs are created in reversed order so that we can more easily find the correct rung corresponding to the current iteration of the result.

on_result(trial_id, cur_iter, metrics)

Return type:

str