syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes module

syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.assign_active_metric(predictor, active_metric)

Checks that active_metric is provided when predictor consists of multiple output predictors. Otherwise, just sets active_metric to the only predictor output name available.

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.NextCandidatesAlgorithm

Bases: object

next_candidates()

Return type:

List[Dict[str, Union[int, float, str]]]

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.Predictor(state, active_metric=None)

Bases: object

Base class for probabilistic predictors used in Bayesian optimization. They support marginal predictions feeding into an acquisition function, as well as computing gradients of an acquisition function w.r.t. inputs.

In general, a predictor is created by an estimator. It wraps a posterior state, which allows for probabilistic predictions on arbitrary inputs.

Parameters:

• state (TuningJobState) – Tuning job state

• active_metric (Optional[str]) – Name of internal objective

keys_predict()

Keys of signals returned by predict().

Note: In order to work with AcquisitionFunction implementations, the following signals are required:

• “mean”: Predictive mean

• “std”: Predictive standard deviation

Return type:

Set[str]

Returns:

Set of keys for dict returned by predict()

predict(inputs)

Returns signals which are statistics of the predictive distribution at input points inputs. By default:

• “mean”: Predictive means. If the model supports fantasizing with a number nf of fantasies, this has shape (n, nf), otherwise (n,)

• “std”: Predictive stddevs, shape (n,)

If the hyperparameters of the surrogate model are being optimized (e.g., by empirical Bayes), the returned list has length 1. If its hyperparameters are averaged over by MCMC, the returned list has one entry per MCMC sample.

Parameters:

inputs (ndarray) – Input points, shape (n, d)

Return type:

List[Dict[str, ndarray]]

Returns:

List of dict with keys keys_predict(), of length the number of MCMC samples, or length 1 for empirical Bayes

hp_ranges_for_prediction()

Return type:

HyperparameterRanges

Returns:

Feature generator to be used for inputs in predict()

predict_candidates(candidates)

Convenience variant of predict()

Parameters:

candidates (Iterable[Dict[str, Union[int, float, str]]]) – List of configurations

Return type:

List[Dict[str, ndarray]]

Returns:

Same as predict()

current_best()

Returns the so-called incumbent, to be used in acquisition functions such as expected improvement. This is the minimum of predictive means (signal with key “mean”) at all current candidate locations (both state.trials_evaluations and state.pending_evaluations). Normally, a scalar is returned, but if the model supports fantasizing and the state contains pending evaluations, there is one incumbent per fantasy sample, so a vector is returned.

If the hyperparameters of the surrogate model are being optimized (e.g., by empirical Bayes), the returned list has length 1. If its hyperparameters are averaged over by MCMC, the returned list has one entry per MCMC sample.

Return type:

List[ndarray]

Returns:

Incumbent, see above

backward_gradient(input, head_gradients)

Computes the gradient \(\nabla_x f(x)\) for an acquisition function \(f(x)\), where \(x\) is a single input point. This is using reverse mode differentiation, the head gradients are passed by the acquisition function. The head gradients are \(\partial_k f\), where \(k\) runs over the statistics returned by predict() for the single input point \(x\). The shape of head gradients is the same as the shape of the statistics.

Lists have > 1 entry if MCMC is used, otherwise they are all size 1.

Parameters:

• input (ndarray) – Single input point \(x\), shape (d,)

• head_gradients (List[Dict[str, ndarray]]) – See above

Return type:

List[ndarray]

Returns:

Gradient \(\nabla_x f(x)\) (several if MCMC is used)

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.ScoringFunction(predictor=None, active_metric=None)

Bases: object

Class to score candidates. As opposed to acquisition functions, scores do not support gradient computation. Note that scores are always minimized.

score(candidates, predictor=None)

Parameters:

• candidates (Iterable[Dict[str, Union[int, float, str]]]) – Configurations for which scores are to be computed

• predictor (Union[Predictor, Dict[str, Predictor], None]) – Overrides default predictor

Return type:

List[float]

Returns:

List of score values, length of candidates

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.AcquisitionFunction(predictor=None, active_metric=None)

Bases: ScoringFunction

Base class for acquisition functions \(f(x)\).

Parameters:

• predictor (Union[Predictor, Dict[str, Predictor], None]) – Predictor(s) from surrogate model

• active_metric (Optional[str]) – Name of internal metric

compute_acq(inputs, predictor=None)

Note: If inputs has shape (d,), it is taken to be (1, d)

Parameters:

• inputs (ndarray) – Encoded input points, shape (n, d)

• predictor (Union[Predictor, Dict[str, Predictor], None]) – If given, overrides self.predictor

Return type:

ndarray

Returns:

Acquisition function values, shape (n,)

compute_acq_with_gradient(input, predictor=None)

For a single input point \(x\), compute acquisition function value \(f(x)\) and gradient \(\nabla_x f(x)\).

Parameters:

• input (ndarray) – Single input point \(x\), shape (d,)

• predictor (Union[Predictor, Dict[str, Predictor], None]) – If given, overrides self.predictor

Return type:

Tuple[float, ndarray]

Returns:

\((f(x), \nabla_x f(x))\)

score(candidates, predictor=None)

Parameters:

• candidates (Iterable[Dict[str, Union[int, float, str]]]) – Configurations for which scores are to be computed

• predictor (Union[Predictor, Dict[str, Predictor], None]) – Overrides default predictor

Return type:

List[float]

Returns:

List of score values, length of candidates

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.LocalOptimizer(hp_ranges, predictor, acquisition_class, active_metric=None)

Bases: object

Class that tries to find a local candidate with a better score, typically using a local optimization method such as L-BFGS. It would normally encapsulate an acquisition function and predictor.

acquisition_class contains the type of the acquisition function (subclass of AcquisitionFunction). It can also be a tuple of the form (type, kwargs), where kwargs are extra arguments to the class constructor.

Parameters:

• hp_ranges (HyperparameterRanges) – Feature generator for configurations

• predictor (Union[Predictor, Dict[str, Predictor]]) – Predictor(s) for acquisition function

• acquisition_class (Callable[[Any], AcquisitionFunction]) – See above

• active_metric (Optional[str]) – Name of internal metric

optimize(candidate, predictor=None)

Run local optimization, starting from candidate

Parameters:

• candidate (Dict[str, Union[int, float, str]]) – Starting point

• predictor (Union[Predictor, Dict[str, Predictor], None]) – Overrides self.predictor

Return type:

Dict[str, Union[int, float, str]]

Returns:

Configuration found by local optimization

class syne_tune.optimizer.schedulers.searchers.bayesopt.tuning_algorithms.base_classes.CandidateGenerator

Bases: object

Class to generate candidates from which to start the local minimization, typically random candidate or some form of more uniformly spaced variation, such as latin hypercube or Sobol sequence.

generate_candidates()

Return type:

Iterator[Dict[str, Union[int, float, str]]]

generate_candidates_en_bulk(num_cands, exclusion_list=None)

Parameters:

• num_cands (int) – Number of candidates to generate

• exclusion_list (Optional[ExclusionList]) – If given, these candidates must not be returned

Return type:

List[Dict[str, Union[int, float, str]]]

Returns:

List of num_cands candidates. If exclusion_list is given, the number of candidates returned can be < num_cands