syne_tune.optimizer.schedulers.median_stopping_rule module

class syne_tune.optimizer.schedulers.median_stopping_rule.MedianStoppingRule(scheduler, resource_attr, running_average=True, metric=None, grace_time=1, grace_population=5, rank_cutoff=0.5)[source]

Bases: TrialScheduler

Applies median stopping rule in top of an existing scheduler.

  • If result at time-step ranks less than the cutoff of other results observed at this time-step, the trial is interrupted and otherwise, the wrapped scheduler is called to make the stopping decision.

  • Suggest decisions are left to the wrapped scheduler.

  • The mode of the wrapped scheduler is used.

Reference:

Google Vizier: A Service for Black-Box Optimization.
Golovin et al. 2017.
Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, August 2017
Pages 1487–1495
https://dl.acm.org/doi/10.1145/3097983.3098043
Parameters:

  • scheduler (TrialScheduler) – Scheduler to be called for trial suggestion or when median-stopping-rule decision is to continue.

  • resource_attr (str) – Key in the reported dictionary that accounts for the resource (e.g. epoch).

  • running_average (bool) – If True, then uses the running average of observation instead of raw observations. Defaults to True

  • metric (Optional[str]) – Metric to be considered, defaults to scheduler.metric

  • grace_time (Optional[int]) – Median stopping rule is only applied for results whose resource_attr exceeds this amount. Defaults to 1

  • grace_population (int) – Median stopping rule when at least grace_population have been observed at a resource level. Defaults to 5

  • rank_cutoff (float) – Results whose quantiles are below this level are discarded. Defaults to 0.5 (median)

on_trial_result(trial, result)[source]

Called on each intermediate result reported by a trial.

At this point, the trial scheduler can make a decision by returning one of SchedulerDecision.CONTINUE, SchedulerDecision.PAUSE, or SchedulerDecision.STOP. This will only be called when the trial is currently running.

Parameters:

  • trial (Trial) – Trial for which results are reported

  • result (Dict) – Result dictionary

Return type:

str

Returns:

Decision what to do with the trial

grace_condition(time_step)[source]
Parameters:

time_step (float) – Value result[self.resource_attr]

Return type:

bool

Returns:

Decide for continue?

metric_names()[source]
Return type:

List[str]

Returns:

List of metric names. The first one is the target metric optimized over, unless the scheduler is a genuine multi-objective metric (for example, for sampling the Pareto front)

metric_mode()[source]
Return type:

str

Returns:

“min” if target metric is minimized, otherwise “max”. Here, “min” should be the default. For a genuine multi-objective scheduler, a list of modes is returned