syne_tune.optimizer.schedulers.hyperband_stopping module

class syne_tune.optimizer.schedulers.hyperband_stopping.RungEntry(trial_id, metric_val)

Bases: object

Represents entry in a rung. This class is extended by rung level systems which need to maintain more information per entry.

Parameters:

• trial_id (str) – ID of trial

• metric_val (float) – Metric value

class syne_tune.optimizer.schedulers.hyperband_stopping.Rung(level, prom_quant, mode, data=None)

Bases: object

Parameters:

• level (int) – Rung level \(r_j\)

• prom_quant (float) – promotion quantile \(q_j\)

• data (Optional[List[RungEntry]]) – Data of all previous jobs reaching the level. This list is kept sorted w.r.t. metric_val, so that best values come first

add(entry)

pop(pos)

Return type:

RungEntry

quantile()

Returns same value as numpy.quantile(metric_vals, q), where metric_vals are the metric values in data, and q = prom_quant if mode == "min", q = ``1 - prom_quant otherwise. If len(data) < 2, we return None.

See here. The default for numpy.quantile is method="linear".

Return type:

Optional[float]

Returns:

See above

class syne_tune.optimizer.schedulers.hyperband_stopping.RungSystem(rung_levels, promote_quantiles, metric, mode, resource_attr, max_t)

Bases: object

Terminology: Trials emit results at certain resource levels (e.g., epoch numbers). Some resource levels are rung levels, this is where scheduling decisions (stop, continue or pause, resume) are taken. For a running trial, the next rung level (or max_t) it will reach is called its next milestone.

Note that rung_levels, promote_quantiles can be empty. All entries of rung_levels are smaller than max_t.

Parameters:

• rung_levels (List[int]) – List of rung levels (positive int, increasing)

• promote_quantiles (List[float]) – List of promotion quantiles at each rung level

• metric (str) – Name of metric to optimize

• mode (str) – “min” or “max”

• resource_attr (str) – Name of resource attribute

• max_t (int) – Largest resource level

on_task_schedule(new_trial_id)

Called when new task is to be scheduled.

For a promotion-based rung system, check whether any trial can be promoted. If so, return dict with keys “trial_id”, “resume_from” (rung level where trial is paused), “milestone” (next rung level the trial will reach, or None).

If no trial can be promoted, or if the rung system is not promotion-based, the returned dictionary must not contain the “trial_id” key. It is nevertheless passed back via extra_kwargs in on_task_schedule(). The default is to return an empty dictionary, but some special subclasses can use this to return information in case a trial is not promoted.

If no trial can be promoted, or if the rung system is not promotion-based, the returned dictionary must not contain the “trial_id” key. It is nevertheless passed back via extra_kwargs in on_task_schedule(). The default is to return an empty dictionary, but some special subclasses can use this to return information in case a trial is not promoted.

Parameters:

new_trial_id (str) – ID for new trial as passed to _suggest(). Only needed by specific subclasses

Return type:

Dict[str, Any]

Returns:

See above

on_task_add(trial_id, skip_rungs, **kwargs)

Called when new task is started.

Parameters:

• trial_id (str) – ID of trial to be started

• skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

• kwargs – Additional arguments

on_task_report(trial_id, result, skip_rungs)

Called when a trial reports metric results.

Returns dict with keys “milestone_reached” (trial reaches its milestone), “task_continues” (trial should continue; otherwise it is stopped or paused), “next_milestone” (next milestone it will reach, or None). For certain subclasses, there may be additional entries.

Parameters:

• trial_id (str) – ID of trial which reported results

• result (Dict[str, Any]) – Reported metrics

• skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

Return type:

Dict[str, Any]

Returns:

See above

on_task_remove(trial_id)

Called when task is removed.

Parameters:

trial_id (str) – ID of trial which is to be removed

get_first_milestone(skip_rungs)

Parameters:

skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

Return type:

int

Returns:

First milestone to be considered

get_milestones(skip_rungs)

Parameters:

skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

Return type:

List[int]

Returns:

All milestones to be considered, in decreasing order; does not include max_t

snapshot_rungs(skip_rungs)

A snapshot is a list of rung levels with entries (level, data), ordered from top to bottom (largest rung first).

Parameters:

skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

Return type:

List[Tuple[int, List[RungEntry]]]

Returns:

Snapshot (see above)

static does_pause_resume()

Return type:

bool

Returns:

Is this variant doing pause and resume scheduling, in the sense that trials can be paused and resumed later?

support_early_checkpoint_removal()

Return type:

bool

Returns:

Do we support early checkpoint removal via paused_trials()?

paused_trials(resource=None)

Only for pause and resume schedulers (does_pause_resume() returns True), where trials can be paused at certain rung levels only. If resource is not given, returns list of all paused trials (trial_id, rank, metric_val, level), where level is the rung level, and rank is the rank of the trial in the rung (0 for the best metric value). If resource is given, only the paused trials in the rung of this level are returned. If resource is not a rung level, the returned list is empty.

Parameters:

resource (Optional[int]) – If given, paused trials of only this rung level are returned. Otherwise, all paused trials are returned

Return type:

List[Tuple[str, int, float, int]]

Returns:

See above

information_for_rungs()

Return type:

List[Tuple[int, int, float]]

Returns:

List of (resource, num_entries, prom_quant), where resource is a rung level, num_entries the number of entries in the rung, and prom_quant the promotion quantile

class syne_tune.optimizer.schedulers.hyperband_stopping.StoppingRungSystem(rung_levels, promote_quantiles, metric, mode, resource_attr, max_t)

Bases: RungSystem

The decision on whether a trial \(\mathbf{x}\) continues or is stopped at a rung level \(r\), is taken in on_task_report(). To this end, the metric value \(f(\mathbf{x}, r)\) is inserted into \(r.data\). Then:

\[\mathrm{continues}(\mathbf{x}, r)\; \Leftrightarrow\; f(\mathbf{x}, r) \le \mathrm{np.quantile}(r.data, r.prom\_quant)\]

in case mode == "min". See also _task_continues().

on_task_schedule(new_trial_id)

Called when new task is to be scheduled.

For a promotion-based rung system, check whether any trial can be promoted. If so, return dict with keys “trial_id”, “resume_from” (rung level where trial is paused), “milestone” (next rung level the trial will reach, or None).

If no trial can be promoted, or if the rung system is not promotion-based, the returned dictionary must not contain the “trial_id” key. It is nevertheless passed back via extra_kwargs in on_task_schedule(). The default is to return an empty dictionary, but some special subclasses can use this to return information in case a trial is not promoted.

If no trial can be promoted, or if the rung system is not promotion-based, the returned dictionary must not contain the “trial_id” key. It is nevertheless passed back via extra_kwargs in on_task_schedule(). The default is to return an empty dictionary, but some special subclasses can use this to return information in case a trial is not promoted.

Parameters:

new_trial_id (str) – ID for new trial as passed to _suggest(). Only needed by specific subclasses

Return type:

Dict[str, Any]

Returns:

See above

on_task_report(trial_id, result, skip_rungs)

Called when a trial reports metric results.

Returns dict with keys “milestone_reached” (trial reaches its milestone), “task_continues” (trial should continue; otherwise it is stopped or paused), “next_milestone” (next milestone it will reach, or None). For certain subclasses, there may be additional entries.

Parameters:

• trial_id (str) – ID of trial which reported results

• result (Dict[str, Any]) – Reported metrics

• skip_rungs (int) – This number of the smallest rung levels are not considered milestones for this task

Return type:

Dict[str, Any]

Returns:

See above

static does_pause_resume()

Return type:

bool

Returns:

Is this variant doing pause and resume scheduling, in the sense that trials can be paused and resumed later?