syne_tune.optimizer.schedulers.searchers.bayesopt.models.cost.cost_model module

class syne_tune.optimizer.schedulers.searchers.bayesopt.models.cost.cost_model.CostValue(c0, c1)

Bases: object

Represents cost value \((c_0(x), c_1(x))\):

• \(c_0(x)\): Startup cost for evaluation at config \(x\)

• \(c_1(x)\): Cost per unit of resource \(r\) at config \(x\)

Our assumption is that, under the model, an evaluation at \(x\) until resource level \(r = 1, 2, 3, \dots\) costs \(c(x, r) = c_0(x) + r c_1(x)\)

c0: float

c1: float

class syne_tune.optimizer.schedulers.searchers.bayesopt.models.cost.cost_model.CostModel

Bases: object

Interface for (temporal) cost model in the context of multi-fidelity HPO. We assume there are configurations \(x\) and resource levels \(r\) (for example, number of epochs). Here, \(r\) is a positive int. Can be seen as simplified version of surrogate model, which is mainly used in order to draw (jointly dependent) values from the posterior over cost values \((c_0(x), c_1(x))\).

Note: The model may be random (in which case joint samples are drawn from the posterior) or deterministic (in which case the model is fitted to data, and then cost values returned are deterministic.

A cost model has an inner state, which is set by calling update() passing a dataset. This inner state is then used when sample_joint() is called.

property cost_metric_name: str

Returns:

Name of metric in TrialEvaluations of cases in TuningJobState

update(state)

Update inner representation in order to be ready to return cost value samples.

Note: The metric :attr``cost_metric_name`` must be dict-valued in state, with keys being resource values \(r\). In order to support a proper estimation of \(c_0\) and \(c_1\), there should (ideally) be entries with the same \(x\) and different resource levels \(r\). The likelihood function takes into account that \(c(x, r) = c_0(x) + r c_1(x)\).

Parameters:

state (TuningJobState) – Current dataset (only trials_evaluations is used)

resample()

For a random cost model, the state is resampled, such that calls of joint_sample before and after are conditionally independent. Normally, successive calls of sample_joint are jointly dependent. For example, for a linear model, the state resampled here would be the weight vector, which is then used in sample_joint().

For a deterministic cost model, this method does nothing.

sample_joint(candidates)

Draws cost values \((c_0(x), c_1(x))\) for candidates (non-extended).

If the model is random, the sampling is done jointly. Also, if sample_joint() is called multiple times, the posterior is to be updated after each call, such that the sample over the union of candidates over all calls is drawn jointly (but see resample()). Also, if measurement noise is allowed in update, this noise is not added here. A sample from \(c(x, r)\) is obtained as \(c_0(x) + r c_1(x)\). If the model is deterministic, the model determined in update() is just evaluated.

Parameters:

candidates (List[Dict[str, Union[int, float, str]]]) – Non-extended configs

Return type:

List[CostValue]

Returns:

List of \((c_0(x), c_1(x))\)

static event_time(start_time, level, next_milestone, cost)

If a task reported its last recent value at start_time at level level, return time of reaching level next_milestone, given cost cost.

Parameters:

• start_time (float) – See above

• level (int) – See above

• next_milestone (int) – See above

• cost (CostValue) – See above

Return type:

float

Returns:

Time of reaching next_milestone under cost model

predict_times(candidates, resources, cost_values, start_time=0)

Given configs \(x\), resource values \(r\) and cost values returned by sample_joint(), compute time predictions for when each config \(x\) reaches its resource level \(r\) if started at start_time.

Parameters:

• candidates (List[Dict[str, Union[int, float, str]]]) – Configs

• resources (List[int]) – Resource levels

• cost_values (List[CostValue]) – Cost values from sample_joint()

• start_time (float) – See above

Return type:

List[float]

Returns:

Predicted times