syne_tune.optimizer.schedulers.hyperband module

syne_tune.optimizer.schedulers.hyperband.is_continue_decision(trial_decision)[source]
Return type:

bool

class syne_tune.optimizer.schedulers.hyperband.TrialInformation(config, time_stamp, bracket, keep_case, trial_decision, reported_result=None, largest_update_resource=None)[source]

Bases: object

The scheduler maintains information about all trials it has been dealing with so far. trial_decision is the current status of the trial. keep_case is relevant only if searcher_data == "rungs_and_last". largest_update_resource is the largest resource level for which the searcher was updated, or None. reported_result contains the last recent reported result, or None (task was started, but did not report anything yet). Only contains attributes self.metric and self._resource_attr.

config: Dict[str, Any]
time_stamp: float
bracket: int
keep_case: bool
trial_decision: str
reported_result: Optional[dict] = None
largest_update_resource: Optional[int] = None
restart(time_stamp)[source]
class syne_tune.optimizer.schedulers.hyperband.HyperbandScheduler(config_space, **kwargs)[source]

Bases: FIFOScheduler, MultiFidelitySchedulerMixin, RemoveCheckpointsSchedulerMixin

Implements different variants of asynchronous Hyperband

See type for the different variants. One implementation detail is when using multiple brackets, task allocation to bracket is done randomly, based on a distribution which can be configured.

For definitions of concepts (bracket, rung, milestone), see

Li, Jamieson, Rostamizadeh, Gonina, Hardt, Recht, Talwalkar (2018)
A System for Massively Parallel Hyperparameter Tuning
https://arxiv.org/abs/1810.05934

or

Tiao, Klein, Lienart, Archambeau, Seeger (2020)
Model-based Asynchronous Hyperparameter and Neural Architecture Search
https://arxiv.org/abs/2003.10865

Note

This scheduler requires both metric and resource_attr to be returned by the reporter. Here, resource values must be positive int. If resource_attr == "epoch", this should be the number of epochs done, starting from 1 (not the epoch number, starting from 0).

Rung levels and promotion quantiles

Rung levels are values of the resource attribute at which stop/go decisions are made for jobs, comparing their metric against others at the same level. These rung levels (positive, strictly increasing) can be specified via rung_levels, the largest must be <= max_t. If rung_levels is not given, they are specified by grace_period and reduction_factor or rung_increment:

  • If \(r_{min}\) is grace_period, \(\eta\) is reduction_factor, then rung levels are \(\mathrm{round}(r_{min} \eta^j), j=0, 1, \dots\). This is the default choice for successive halving (Hyperband).

  • If rung_increment is given, but not reduction_factor, then rung levels are \(r_{min} + j \nu, j=0, 1, \dots\), where \(\nu\) is rung_increment.

If rung_levels is given, then grace_period, reduction_factor, rung_increment are ignored. If they are given, a warning is logged.

The rung levels determine the quantiles to be used in the stop/go decisions. If rung levels are \(r_j\), define \(q_j = r_j / r_{j+1}\). \(q_j\) is the promotion quantile at rung level \(r_j\). On average, a fraction of \(q_j\) jobs can continue, the remaining ones are stopped (or paused). In the default successive halving case, we have \(q_j = 1/\eta\) for all \(j\).

Cost-aware schedulers or searchers

Some schedulers (e.g., type == "cost_promotion") or searchers may depend on cost values (with key cost_attr) reported alongside the target metric. For promotion-based scheduling, a trial may pause and resume several times. The cost received in on_trial_result only counts the cost since the last resume. We maintain the sum of such costs in _cost_offset(), and append a new entry to result in on_trial_result with the total cost. If the evaluation function does not implement checkpointing, once a trial is resumed, it has to start from scratch. We detect this in on_trial_result and reset the cost offset to 0 (if the trial runs from scratch, the cost reported needs no offset added).

Note

This process requires cost_attr to be set

Pending evaluations

The searcher is notified, by searcher.register_pending calls, of (trial, resource) pairs for which evaluations are running, and a result is expected in the future. These pending evaluations can be used by the searcher in order to direct sampling elsewhere.

The choice of pending evaluations depends on searcher_data. If equal to “rungs”, pending evaluations sit only at rung levels, because observations are only used there. In the other cases, pending evaluations sit at all resource levels for which observations are obtained. For example, if a trial is at rung level \(r\) and continues towards the next rung level \(r_{next}\), if searcher_data == "rungs", searcher.register_pending is called for \(r_{next}\) only, while for other searcher_data values, pending evaluations are registered for \(r + 1, r + 2, \dots, r_{next}\). However, if in this case, register_pending_myopic is True, we instead call searcher.register_pending for \(r + 1\) when each observation is obtained (not just at a rung level). This leads to less pending evaluations at any one time. On the other hand, when a trial is continued at a rung level, we already know it will emit observations up to the next rung level, so it seems more “correct” to register all these pending evaluations in one go.

Additional arguments on top of parent class FIFOScheduler:

Parameters:

  • searcher (str or BaseSearcher) – Searcher for get_config decisions. String values are passed to searcher_factory() along with search_options and extra information. Supported values: SUPPORTED_SEARCHERS_HYPERBAND. Defaults to “random” (i.e., random search)

  • resource_attr (str, optional) – Name of resource attribute in results obtained via on_trial_result, defaults to “epoch”

  • grace_period (int, optional) – Minimum resource to be used for a job. Ignored if rung_levels is given. Defaults to 1

  • reduction_factor (float, optional) – Parameter to determine rung levels. Ignored if rung_levels is given. Must be \(\ge 2\), defaults to 3

  • rung_increment (int, optional) – Parameter to determine rung levels. Ignored if rung_levels or reduction_factor are given. Must be postive

  • rung_levels (List[int], optional) – If given, prescribes the set of rung levels to be used. Must contain positive integers, strictly increasing. This information overrides grace_period, reduction_factor, rung_increment. Note that the stop/promote rule in the successive halving scheduler is set based on the ratio of successive rung levels.

  • brackets (int, optional) – Number of brackets to be used in Hyperband. Each bracket has a different grace period, all share max_t and reduction_factor. If brackets == 1 (default), we run asynchronous successive halving.

  • type (str, optional) –

    Type of Hyperband scheduler. Defaults to “stopping”. Supported values (see also subclasses of RungSystem):

    • stopping: A config eval is executed by a single task. The task is stopped at a milestone if its metric is worse than a fraction of those who reached the milestone earlier, otherwise it continues. See StoppingRungSystem.

    • promotion: A config eval may be associated with multiple tasks over its lifetime. It is never terminated, but may be paused. Whenever a task becomes available, it may promote a config to the next milestone, if better than a fraction of others who reached the milestone. If no config can be promoted, a new one is chosen. See PromotionRungSystem.

    • cost_promotion: This is a cost-aware variant of ‘promotion’, see CostPromotionRungSystem for details. In this case, costs must be reported under the name rung_system_kwargs["cost_attr"] in results.

    • pasha: Similar to promotion type Hyperband, but it progressively expands the available resources until the ranking of configurations stabilizes.

    • rush_stopping: A variation of the stopping scheduler which requires passing rung_system_kwargs and points_to_evaluate. The first rung_system_kwargs["num_threshold_candidates"] of points_to_evaluate will enforce stricter rules on which task is continued. See RUSHStoppingRungSystem and RUSHScheduler.

    • rush_promotion: Same as rush_stopping but for promotion, see RUSHPromotionRungSystem

    • dyhpo: A model-based scheduler, which can be seen as extension of “promotion” with rung_increment rather than reduction_factor, see DynamicHPOSearcher

  • cost_attr (str, optional) – Required if the scheduler itself uses a cost metric (i.e., type="cost_promotion"), or if the searcher uses a cost metric. See also header comment.

  • searcher_data (str, optional) –

    Relevant only if a model-based searcher is used. Example: For NN tuning and ``resource_attr == “epoch”’, we receive a result for each epoch, but not all epoch values are also rung levels. searcher_data determines which of these results are passed to the searcher. As a rule, the more data the searcher receives, the better its fit, but also the more expensive get_config may become. Choices:

    • ”rungs” (default): Only results at rung levels. Cheapest

    • ”all”: All results. Most expensive

    • ”rungs_and_last”: Results at rung levels, plus the most recent result. This means that in between rung levels, only the most recent result is used by the searcher. This is in between

    Note: For a Gaussian additive learning curve surrogate model, this has to be set to ‘all’.

  • register_pending_myopic (bool, optional) – See above. Used only if searcher_data != "rungs". Defaults to False

  • rung_system_per_bracket (bool, optional) – This concerns Hyperband with brackets > 1. Defaults to False. When starting a job for a new config, it is assigned a randomly sampled bracket. The larger the bracket, the larger the grace period for the config. If rung_system_per_bracket == True, we maintain separate rung level systems for each bracket, so that configs only compete with others started in the same bracket. If rung_system_per_bracket == False, we use a single rung level system, so that all configs compete with each other. In this case, the bracket of a config only determines the initial grace period, i.e. the first milestone at which it starts competing with others. This is the default. The concept of brackets in Hyperband is meant to hedge against overly aggressive filtering in successive halving, based on low fidelity criteria. In practice, successive halving (i.e., brackets = 1) often works best in the asynchronous case (as implemented here). If brackets > 1, the hedging is stronger if rung_system_per_bracket is True.

  • do_snapshots (bool, optional) – Support snapshots? If True, a snapshot of all running tasks and rung levels is returned by _promote_trial(). This snapshot is passed to searcher.get_config. Defaults to False. Note: Currently, only the stopping variant supports snapshots.

  • rung_system_kwargs (Dict[str, Any], optional) –

    Arguments passed to the rung system: * num_threshold_candidates: Used if ``type in [“rush_promotion”,

    ”rush_stopping”]``. The first num_threshold_candidates in points_to_evaluate enforce stricter requirements to the continuation of training tasks. See RUSHScheduler.

    • probability_sh: Used if type == "dyhpo". In DyHPO, we typically all paused trials against a number of new configurations, and the winner is either resumed or started (new trial). However, with the probability given here, we instead try to promote a trial as if type == "promotion". If no trial can be promoted, we fall back to the DyHPO logic. Use this to make DyHPO robust against starting too many new trials, because all paused ones score poorly (this happens especially at the beginning).

  • early_checkpoint_removal_kwargs (Dict[str, Any], optional) – If given, speculative early removal of checkpoints is done, see HyperbandRemoveCheckpointsCallback. The constructor arguments for the HyperbandRemoveCheckpointsCallback must be given here, if they cannot be inferred (key max_num_checkpoints is mandatory). This feature is used only for scheduler types which pause and resume trials.

does_pause_resume()[source]
Return type:

bool

Returns:

Is this variant doing pause and resume scheduling, in the sense that trials can be paused and resumed later?

property rung_levels: List[int]

Note that all entries of rung_levels are smaller than max_t (or config_space[max_resource_attr]): rung levels are resource levels where stop/go decisions are made. In particular, if rung_levels is passed at construction with rung_levels[-1] == max_t, this last entry is stripped off.

Returns:

Rung levels (strictly increasing, positive ints)

property num_brackets: int
Returns:

Number of brackets (i.e., rung level systems). If the scheduler does not use brackets, it has to return 1

property resource_attr: str
Returns:

Name of resource attribute in reported results

property max_resource_level: int
Returns:

Maximum resource level

property searcher_data: str
Returns:

Relevant only if a model-based searcher is used. Example: For NN tuning and resource_attr == "epoch", we receive a result for each epoch, but not all epoch values are also rung levels. searcher_data determines which of these results are passed to the searcher. As a rule, the more data the searcher receives, the better its fit, but also the more expensive get_config() may become. Choices:

  • ”rungs”: Only results at rung levels. Cheapest

  • ”all”: All results. Most expensive

  • ”rungs_and_last”: Results at rung levels plus last recent one. Not available for all multi-fidelity schedulers

on_trial_error(trial)[source]

Called when a trial has failed.

Parameters:

trial (Trial) – Trial for which error is reported.

on_trial_result(trial, result)[source]

We simply relay result to the searcher. Other decisions are done in on_trial_complete.

Return type:

str

on_trial_remove(trial)[source]

Called to remove trial.

This is called when the trial is in PAUSED or PENDING state. Otherwise, call on_trial_complete().

Parameters:

trial (Trial) – Trial to be removed

on_trial_complete(trial, result)[source]

Notification for the completion of trial.

Note that on_trial_result() is called with the same result before. However, if the scheduler only uses one final report from each trial, it may ignore on_trial_result() and just use result here.

Parameters:

  • trial (Trial) – Trial which is completing

  • result (Dict[str, Any]) – Result dictionary

callback_for_checkpoint_removal(stop_criterion)[source]
Parameters:

stop_criterion (Callable[[TuningStatus], bool]) – Stopping criterion, as passed to Tuner

Return type:

Optional[TunerCallback]

Returns:

CP removal callback, or None if CP removal is not activated

class syne_tune.optimizer.schedulers.hyperband.HyperbandBracketManager(scheduler_type, resource_attr, metric, mode, max_t, rung_levels, brackets, rung_system_per_bracket, cost_attr, random_seed, rung_system_kwargs, scheduler)[source]

Bases: object

Maintains rung level systems for range of brackets. Differences depending on scheduler_type manifest themselves mostly at the level of the rung level system itself.

Parameters:
static does_pause_resume(scheduler_type)[source]
Return type:

bool

Returns:

Is this variant doing pause and resume scheduling, in the sense that trials can be paused and resumed later?

on_task_add(trial_id, **kwargs)[source]

Called when new task is started (can be new trial or trial being resumed).

Since the bracket has already been sampled, not much is done here. We return the list of milestones for this bracket in reverse (decreasing) order. The first entry is max_t, even if it is not a rung level in the bracket. This list contains the resource levels the task would reach if it ran to max_t without being stopped.

Parameters:

  • trial_id (str) – ID of trial

  • kwargs – Further arguments passed to rung_sys.on_task_add

Return type:

List[int]

Returns:

List of milestones in decreasing order, where`` max_t`` is first

on_task_report(trial_id, result)[source]

This method is called whenever a new report is received. It returns a dictionary with all the information needed for making decisions (e.g., stop / continue task, update model, etc). Keys are:

  • task_continues: Should task continue or stop/pause?

  • milestone_reached: True if rung level (or max_t) is hit

  • next_milestone: If hit rung level < max_t, this is the subsequent rung level (otherwise: None)

  • bracket_id: Bracket in which the task is running

Parameters:

  • trial_id (str) – ID of trial

  • result (Dict[str, Any]) – Results reported

Return type:

Dict[str, Any]

Returns:

See above

on_task_remove(trial_id)[source]

Called when trial is stopped or completes

Parameters:

trial_id – ID of trial

on_task_schedule(new_trial_id)[source]

Samples bracket for task to be scheduled. Check whether any paused trial in that bracket can be promoted. If so, its trial_id is returned. We also return extra_kwargs to be used in _promote_trial. This contains the bracket which was sampled (key “bracket”).

Note: extra_kwargs can return information also if trial_id = None is returned. This information is passed to get_config of the searcher.

Note: extra_kwargs can return information also if trial_id = None is returned. This information is passed to get_config of the searcher.

Parameters:

new_trial_id (str) – ID for new trial as passed to _suggest()

Return type:

(Optional[str], dict)

Returns:

(trial_id, extra_kwargs)

snapshot_rungs(bracket_id)[source]
paused_trials(resource=None)[source]

Only for pause and resume schedulers (does_pause_resume() returns True), where trials can be paused at certain rung levels only. If resource is not given, returns list of all paused trials (trial_id, rank, metric_val, level), where level is the rung level, and rank is the rank of the trial in the rung (0 for the best metric value). If resource is given, only the paused trials in the rung of this level are returned.

Parameters:

resource (Optional[int]) – If given, paused trials of only this rung level are returned. Otherwise, all paused trials are returned

Return type:

List[Tuple[str, int, float, int]]

Returns:

See above

information_for_rungs()[source]
Return type:

List[Tuple[int, int, float]]

Returns:

List of (resource, num_entries, prom_quant), where resource is a rung level, num_entries the number of entries in the rung, and prom_quant the promotion quantile

support_early_checkpoint_removal()[source]
Return type:

bool