syne_tune.optimizer.schedulers.hyperband_cost_promotion module

class syne_tune.optimizer.schedulers.hyperband_cost_promotion.CostPromotionRungEntry(trial_id, metric_val, cost_val, was_promoted=False)

Bases: PromotionRungEntry

Appends cost_val to the superclass. This is the cost value \(c(x, r)\) recorded for the trial at the resource level.

class syne_tune.optimizer.schedulers.hyperband_cost_promotion.CostPromotionRungSystem(rung_levels, promote_quantiles, metric, mode, resource_attr, cost_attr, max_t)

Bases: PromotionRungSystem

Cost-aware extension of promotion-based asynchronous Hyperband (ASHA).

This code is equivalent with base PromotionRungSystem, except the “promotable” condition in _find_promotable_trial() is replaced.

When a config \(\mathbf{x}\) reaches rung level \(r\), the result includes a metric \(f(\mathbf{x}, r)\), but also a cost \(c(\mathbf{x}, r)\). The latter is the cost (e.g., training time) spent to reach level \(r\).

Consider all trials who reached rung level \(r\) (whether promoted from there or still paused there), ordered w.r.t. \(f(\mathbf{x}, r)\), best first, and let their number be \(N\). Define

\[C(r, k) = \sum_{i\le k} c(\mathbf{x}_i, r)\]

For a promotion quantile \(q\), define

\[K = \max_k \mathrm{I}[ C(r, k) \le q C(r, N) ]\]

Any trial not yet promoted and ranked \(\le K\) can be promoted. As usual, we scan rungs from the top. If several trials are promotable, the one with the best metric value is promoted.

Note that costs \(c(\mathbf{x}, r)\) reported via cost_attr need to be total costs of a trial. If the trial is paused and resumed, partial costs have to be added up. See HyperbandScheduler for how this works.