syne_tune.optimizer.baselines module

class syne_tune.optimizer.baselines.RandomSearch(config_space, metric, **kwargs)[source]

Bases: LegacyFIFOScheduler

Random search.

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.GridSearch(config_space, metric, **kwargs)[source]

Bases: LegacyFIFOScheduler

Grid search.

See GridSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.BayesianOptimization(config_space, metric, **kwargs)[source]

Bases: LegacyFIFOScheduler

Gaussian process based Bayesian optimization.

See GPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHA(config_space, metric, resource_attr, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Asynchronous Sucessive Halving (ASHA).

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful.

See also HyperbandScheduler for kwargs parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.MOBSTER(config_space, metric, resource_attr, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Model-based Asynchronous Multi-fidelity Optimizer (MOBSTER).

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful, see also HyperbandScheduler.

MOBSTER can be run with different surrogate models. The model is selected by search_options["model"] in kwargs. The default is "gp_multitask" (jointly dependent multi-task GP model), another useful choice is "gp_independent" (independent GP models at each rung level, with shared ARD kernel).

See also:

HyperbandScheduler for kwargs parameters
HyperTuneSearcher for kwargs["search_options"] parameters
HyperTuneIndependentGPModel for implementation

Parameters:

config_space (Dict) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.DyHPO(config_space, metric, resource_attr, probability_sh=None, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Dynamic Gray-Box Hyperparameter Optimization (DyHPO)

One of max_t, max_resource_attr needs to be in kwargs. The latter is more useful (DyHPO is a pause-resume scheduler), see also HyperbandScheduler.

DyHPO can be run with the same surrogate models as MOBSTER, but search_options["model"] != "gp_independent". This is because DyHPO requires extrapolation to resource levels without any data, which cannot sensibly be done with independent GPs per resource level. Compared to MOBSTER or HyperTune, DyHPO is typically run with linearly spaced rung levels (the default being 1, 2, 3, …). Decisions whether to promote a paused trial are folded together with suggesting a new configuration, both are model-based. Our implementation is based on

Wistuba, M. and Kadra, A. and Grabocka, J.

Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning

https://arxiv.org/abs/2202.09774

However, there are important differences:

We do not implement their surrogate model based on a neural network kernel, but instead just use the surrogate models we provide for MOBSTER as well

We implement a hybrid of DyHPO with the asynchronous successive halving rule for promoting trials, controlled by probability_sh. With this probability, we promote a trial via the SH rule. This mitigates the issue that DyHPO tends to start many trials initially, because due to lack of any data at higher rungs, the score values for promoting a trial are much worse than those for starting a new one.

See HyperbandScheduler for kwargs parameters, and GPMultiFidelitySearcher for kwargs["search_options"] parameters. The following parameters are most important for DyHPO:

rung_increment (and grace_period): These parameters determine the rung level spacing. DyHPO is run with linearly spaced rung levels :math:`r_{min} + k

u`, where \(r_{min}\) is grace_period and

:math:`

u` is rung_increment. The default is 2.

probability_sh: See comment. The smaller this probability, the closer the method is to the published original, which tends to start many more trials than promote paused ones. On the other hand, if this probability is close to 1, you may as well run MOBSTER. The default is DEFAULT_SH_PROBABILITY.
search_options["opt_skip_period"]: DyHPO can be quite a bit slower than MOBSTER, because the GP surrogate model is used more frequently. It can be sped up a bit by changing opt_skip_period (general default is 1). The default here is 3.

class syne_tune.optimizer.baselines.PASHA(config_space, metric, resource_attr, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Progressive ASHA.

One of max_t, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.BOHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Asynchronous BOHB

Combines ASHA with TPE-like Bayesian optimization, using kernel density estimators.

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful, see also HyperbandScheduler.

See MultiFidelityKernelDensityEstimator for kwargs["search_options"] parameters, and HyperbandScheduler for kwargs parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.SyncHyperband(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous Hyperband.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.SyncBOHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous BOHB.

Combines SyncHyperband with TPE-like Bayesian optimization, using kernel density estimators.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.DEHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: GeometricDifferentialEvolutionHyperbandScheduler

Differential Evolution Hyperband (DEHB).

Combines SyncHyperband with ideas from evolutionary algorithms.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.SyncMOBSTER(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous MOBSTER.

Combines SyncHyperband with Gaussian process based Bayesian optimization, just like MOBSTER builds on top of ASHA in the asynchronous case.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

The default surrogate model (search_options["model"] in kwargs) is "gp_independent", different to MOBSTER.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.BORE(config_space, metric, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Bayesian Optimization by Density-Ratio Estimation (BORE).

See Bore for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHABORE(config_space, metric, resource_attr, random_seed=None, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Model-based ASHA with BORE searcher

See MultiFidelityBore for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.BoTorch(config_space, metric, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Bayesian Optimization using BoTorch

See BoTorchSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.REA(config_space, metric, population_size=100, sample_size=10, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Regularized Evolution (REA).

See RegularizedEvolution for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
population_size (int) – See RegularizedEvolution. Defaults to 100
sample_size (int) – See RegularizedEvolution. Defaults to 10
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

syne_tune.optimizer.baselines.create_gaussian_process_estimator(config_space, metric, random_seed=None, search_options=None)[source]

Return type:: Estimator

class syne_tune.optimizer.baselines.MORandomScalarizationBayesOpt(config_space, metric, mode='min', random_seed=None, estimators=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Uses MultiObjectiveMultiSurrogateSearcher with one standard GP surrogate model per metric (same as in BayesianOptimization, together with the MultiObjectiveLCBRandomLinearScalarization acquisition function.

If estimators is given, surrogate models are taken from there, and the default is used otherwise. This is useful if you have a good low-variance model for one of the objectives.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (List[str]) – Name of metrics to optimize
mode (Union[List[str], str]) – Modes of optimization. Defaults to “min” for all
random_seed (Optional[int]) – Random seed, optional
estimators (Optional[Dict[str, Estimator]]) – Use these surrogate models instead of the default GP one. Optional
kwargs – Additional arguments to FIFOScheduler. Here, kwargs["search_options"] is used to create the searcher and its GP surrogate models.

class syne_tune.optimizer.baselines.NSGA2(config_space, metric, mode='min', population_size=20, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (List[str]) – Name of metric to optimize
population_size (int) – The size of the population for NSGA-2
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.MOREA(config_space, metric, mode='min', population_size=100, sample_size=10, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (List[str]) – Name of metric to optimize
population_size (int) – See RegularizedEvolution. Defaults to 100
sample_size (int) – See RegularizedEvolution. Defaults to 10
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.MOLinearScalarizationBayesOpt(config_space, metric, scalarization_weights=None, **kwargs)[source]

Bases: LegacyLinearScalarizedScheduler

Uses LinearScalarizedScheduler together with a default GP surrogate model.

See GPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (List[str]) – Name of metric to optimize
scalarization_weights (Optional[List[float]]) – Positive weight used for the scalarization. Defaults to all 1
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ConstrainedBayesianOptimization(config_space, metric, constraint_attr, **kwargs)[source]

Bases: LegacyFIFOScheduler

Constrained Bayesian Optimization.

See ConstrainedGPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
constraint_attr (str) – Name of constraint metric
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ZeroShotTransfer(config_space, transfer_learning_evaluations, metric, mode='min', sort_transfer_learning_evaluations=True, use_surrogates=False, random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

A zero-shot transfer hyperparameter optimization method which jointly selects configurations that minimize the average rank obtained on historic metadata (transfer_learning_evaluations). Reference:

Sequential Model-Free Hyperparameter Tuning.

Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme.

IEEE International Conference on Data Mining (ICDM) 2015.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
transfer_learning_evaluations (Dict[str, LegacyTransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.
metric (str) – Name of metric to optimize
mode (str) – Whether to minimize (min) or maximize (max)
sort_transfer_learning_evaluations (bool) – Use False if the hyperparameters for each task in transfer_learning_evaluations are already in the same order. If set to True, hyperparameters are sorted.
use_surrogates (bool) – If the same configuration is not evaluated on all tasks, set this to True. This will generate a set of configurations and will impute their performance using surrogate models.
random_seed (Optional[int]) – Used for randomly sampling candidates. Only used if use_surrogates=True.
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHACTS(config_space, metric, resource_attr, transfer_learning_evaluations, mode='min', random_seed=None, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Runs ASHA where the searcher is done with the transfer-learning method:

A Quantile-based Approach for Hyperparameter Transfer Learning.

David Salinas, Huibin Shen, Valerio Perrone.

ICML 2020.

This is the Copula Thompson Sampling approach described in the paper where a surrogate is fitted on the transfer learning data to predict mean and variance of configuration performance given a hyperparameter. The surrogate is then sampled from, and the best configurations are returned as next candidate to evaluate.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
resource_attr (str) – Name of resource attribute
transfer_learning_evaluations (Dict[str, LegacyTransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.
mode (str) – Whether to minimize (min) or maximize (max)
random_seed (Optional[int]) – Used for randomly sampling candidates
kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.KDE(config_space, metric, **kwargs)[source]

Bases: LegacyFIFOScheduler

Single-fidelity variant of BOHB

Combines FIFOScheduler with TPE-like Bayesian optimization, using kernel density estimators.

See KernelDensityEstimator for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (str) – Name of metric to optimize
kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.CQR(config_space, metric, mode='min', random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Single-fidelity Conformal Quantile Regression approach proposed in:: Optimizing Hyperparameters with Conformal Quantile Regression.

David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, Cedric Archambeau.

ICML 2023.

The method predict quantile performance with gradient boosted trees and calibrate prediction with conformal predictions.

class syne_tune.optimizer.baselines.ASHACQR(config_space, metric, resource_attr, mode='min', random_seed=None, **kwargs)[source]

Bases: LegacyHyperbandScheduler

Multi-fidelity Conformal Quantile Regression approach proposed in:: Optimizing Hyperparameters with Conformal Quantile Regression.

David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, Cedric Archambeau.

ICML 2023.

The method predict quantile performance with gradient boosted trees and calibrate prediction with conformal predictions.

class syne_tune.optimizer.baselines.EHVI(config_space, metric, mode='min', random_seed=None, **kwargs)[source]

Bases: LegacyFIFOScheduler

Implements the Expected Hypervolume Improvement method.

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

config_space (Dict[str, Any]) – Configuration space for evaluation function
metric (List[str]) – Name of metric to optimize
population_size – See RegularizedEvolution. Defaults to 100
sample_size – See RegularizedEvolution. Defaults to 10
random_seed (Optional[int]) – Random seed, optional
kwargs – Additional arguments to FIFOScheduler