syne_tune.optimizer.baselines module

class syne_tune.optimizer.baselines.RandomSearch(config_space, metric, **kwargs)[source]

Bases: FIFOScheduler

Random search.

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.GridSearch(config_space, metric, **kwargs)[source]

Bases: FIFOScheduler

Grid search.

See GridSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.BayesianOptimization(config_space, metric, **kwargs)[source]

Bases: FIFOScheduler

Gaussian process based Bayesian optimization.

See GPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHA(config_space, metric, resource_attr, **kwargs)[source]

Bases: HyperbandScheduler

Asynchronous Sucessive Halving (ASHA).

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful.

See also HyperbandScheduler for kwargs parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.MOBSTER(config_space, metric, resource_attr, **kwargs)[source]

Bases: HyperbandScheduler

Model-based Asynchronous Multi-fidelity Optimizer (MOBSTER).

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful, see also HyperbandScheduler.

MOBSTER can be run with different surrogate models. The model is selected by search_options["model"] in kwargs. The default is "gp_multitask" (jointly dependent multi-task GP model), another useful choice is "gp_independent" (independent GP models at each rung level, with shared ARD kernel).

See also:

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.HyperTune(config_space, metric, resource_attr, **kwargs)[source]

Bases: HyperbandScheduler

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful, see also HyperbandScheduler.

Hyper-Tune is a model-based variant of ASHA with more than one bracket. It can be seen as extension of MOBSTER and can be used with search_options["model"] in kwargs being "gp_independent" or "gp_multitask". It has a model-based way to sample the bracket for every new trial, as well as an ensemble predictive distribution feeding into the acquisition function. Our implementation is based on:

Yang Li et al
Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale
VLDB 2022
https://arxiv.org/abs/2201.06834

See also:

Parameters:

  • config_space (Dict) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.DyHPO(config_space, metric, resource_attr, probability_sh=None, **kwargs)[source]

Bases: HyperbandScheduler

Dynamic Gray-Box Hyperparameter Optimization (DyHPO)

One of max_t, max_resource_attr needs to be in kwargs. The latter is more useful (DyHPO is a pause-resume scheduler), see also HyperbandScheduler.

DyHPO can be run with the same surrogate models as MOBSTER, but search_options["model"] != "gp_independent". This is because DyHPO requires extrapolation to resource levels without any data, which cannot sensibly be done with independent GPs per resource level. Compared to MOBSTER or HyperTune, DyHPO is typically run with linearly spaced rung levels (the default being 1, 2, 3, …). Decisions whether to promote a paused trial are folded together with suggesting a new configuration, both are model-based. Our implementation is based on

Wistuba, M. and Kadra, A. and Grabocka, J.
Dynamic and Efficient Gray-Box Hyperparameter Optimization for Deep Learning
https://arxiv.org/abs/2202.09774

However, there are important differences:

  • We do not implement their surrogate model based on a neural network kernel, but instead just use the surrogate models we provide for MOBSTER as well

  • We implement a hybrid of DyHPO with the asynchronous successive halving rule for promoting trials, controlled by probability_sh. With this probability, we promote a trial via the SH rule. This mitigates the issue that DyHPO tends to start many trials initially, because due to lack of any data at higher rungs, the score values for promoting a trial are much worse than those for starting a new one.

See HyperbandScheduler for kwargs parameters, and GPMultiFidelitySearcher for kwargs["search_options"] parameters. The following parameters are most important for DyHPO:

  • rung_increment (and grace_period): These parameters determine the rung level spacing. DyHPO is run with linearly spaced rung levels :math:`r_{min} + k

u`, where \(r_{min}\) is grace_period and

:math:`

u` is rung_increment. The default is 2.

  • probability_sh: See comment. The smaller this probability, the closer the method is to the published original, which tends to start many more trials than promote paused ones. On the other hand, if this probability is close to 1, you may as well run MOBSTER. The default is DEFAULT_SH_PROBABILITY.

  • search_options["opt_skip_period"]: DyHPO can be quite a bit slower than MOBSTER, because the GP surrogate model is used more frequently. It can be sped up a bit by changing opt_skip_period (general default is 1). The default here is 3.

class syne_tune.optimizer.baselines.PASHA(config_space, metric, resource_attr, **kwargs)[source]

Bases: HyperbandScheduler

Progressive ASHA.

One of max_t, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.BOHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: HyperbandScheduler

Asynchronous BOHB

Combines ASHA with TPE-like Bayesian optimization, using kernel density estimators.

One of max_t, max_resource_attr needs to be in kwargs. For type="promotion", the latter is more useful, see also HyperbandScheduler.

See MultiFidelityKernelDensityEstimator for kwargs["search_options"] parameters, and HyperbandScheduler for kwargs parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.SyncHyperband(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous Hyperband.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.SyncBOHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous BOHB.

Combines SyncHyperband with TPE-like Bayesian optimization, using kernel density estimators.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.DEHB(config_space, metric, resource_attr, **kwargs)[source]

Bases: GeometricDifferentialEvolutionHyperbandScheduler

Differential Evolution Hyperband (DEHB).

Combines SyncHyperband with ideas from evolutionary algorithms.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.SyncMOBSTER(config_space, metric, resource_attr, **kwargs)[source]

Bases: SynchronousGeometricHyperbandScheduler

Synchronous MOBSTER.

Combines SyncHyperband with Gaussian process based Bayesian optimization, just like MOBSTER builds on top of ASHA in the asynchronous case.

One of max_resource_level, max_resource_attr needs to be in kwargs. The latter is more useful, see also HyperbandScheduler.

If kwargs["brackets"] is not given, the maximum number of brackets is used. Choose kwargs["brackets"] = 1 for synchronous successive halving.

The default surrogate model (search_options["model"] in kwargs) is "gp_independent", different to MOBSTER.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • kwargs – Additional arguments to SynchronousGeometricHyperbandScheduler

class syne_tune.optimizer.baselines.BORE(config_space, metric, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

Bayesian Optimization by Density-Ratio Estimation (BORE).

See Bore for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHABORE(config_space, metric, resource_attr, random_seed=None, **kwargs)[source]

Bases: HyperbandScheduler

Model-based ASHA with BORE searcher

See MultiFidelityBore for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.BoTorch(config_space, metric, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

Bayesian Optimization using BoTorch

See BoTorchSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.REA(config_space, metric, population_size=100, sample_size=10, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

Regularized Evolution (REA).

See RegularizedEvolution for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • population_size (int) – See RegularizedEvolution. Defaults to 100

  • sample_size (int) – See RegularizedEvolution. Defaults to 10

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

syne_tune.optimizer.baselines.create_gaussian_process_estimator(config_space, metric, random_seed=None, search_options=None)[source]
Return type:

Estimator

class syne_tune.optimizer.baselines.MORandomScalarizationBayesOpt(config_space, metric, mode='min', random_seed=None, estimators=None, **kwargs)[source]

Bases: FIFOScheduler

Uses MultiObjectiveMultiSurrogateSearcher with one standard GP surrogate model per metric (same as in BayesianOptimization, together with the MultiObjectiveLCBRandomLinearScalarization acquisition function.

If estimators is given, surrogate models are taken from there, and the default is used otherwise. This is useful if you have a good low-variance model for one of the objectives.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (List[str]) – Name of metrics to optimize

  • mode (Union[List[str], str]) – Modes of optimization. Defaults to “min” for all

  • random_seed (Optional[int]) – Random seed, optional

  • estimators (Optional[Dict[str, Estimator]]) – Use these surrogate models instead of the default GP one. Optional

  • kwargs – Additional arguments to FIFOScheduler. Here, kwargs["search_options"] is used to create the searcher and its GP surrogate models.

class syne_tune.optimizer.baselines.NSGA2(config_space, metric, mode='min', population_size=20, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (List[str]) – Name of metric to optimize

  • population_size (int) – The size of the population for NSGA-2

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.MOREA(config_space, metric, mode='min', population_size=100, sample_size=10, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

See RandomSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (List[str]) – Name of metric to optimize

  • population_size (int) – See RegularizedEvolution. Defaults to 100

  • sample_size (int) – See RegularizedEvolution. Defaults to 10

  • random_seed (Optional[int]) – Random seed, optional

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.MOLinearScalarizationBayesOpt(config_space, metric, scalarization_weights=None, **kwargs)[source]

Bases: LinearScalarizedScheduler

Uses LinearScalarizedScheduler together with a default GP surrogate model.

See GPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (List[str]) – Name of metric to optimize

  • scalarization_weights (Optional[List[float]]) – Positive weight used for the scalarization. Defaults to all 1

  • kwargs – Additional arguments to FIFOScheduler

scalarization_weights: ndarray
single_objective_metric: str
base_scheduler: TrialScheduler
class syne_tune.optimizer.baselines.ConstrainedBayesianOptimization(config_space, metric, constraint_attr, **kwargs)[source]

Bases: FIFOScheduler

Constrained Bayesian Optimization.

See ConstrainedGPFIFOSearcher for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • constraint_attr (str) – Name of constraint metric

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ZeroShotTransfer(config_space, transfer_learning_evaluations, metric, mode='min', sort_transfer_learning_evaluations=True, use_surrogates=False, random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

A zero-shot transfer hyperparameter optimization method which jointly selects configurations that minimize the average rank obtained on historic metadata (transfer_learning_evaluations). Reference:

Sequential Model-Free Hyperparameter Tuning.
Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme.
IEEE International Conference on Data Mining (ICDM) 2015.
Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • transfer_learning_evaluations (Dict[str, TransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.

  • metric (str) – Name of metric to optimize

  • mode (str) – Whether to minimize (min) or maximize (max)

  • sort_transfer_learning_evaluations (bool) – Use False if the hyperparameters for each task in transfer_learning_evaluations are already in the same order. If set to True, hyperparameters are sorted.

  • use_surrogates (bool) – If the same configuration is not evaluated on all tasks, set this to True. This will generate a set of configurations and will impute their performance using surrogate models.

  • random_seed (Optional[int]) – Used for randomly sampling candidates. Only used if use_surrogates=True.

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.ASHACTS(config_space, metric, resource_attr, transfer_learning_evaluations, mode='min', random_seed=None, **kwargs)[source]

Bases: HyperbandScheduler

Runs ASHA where the searcher is done with the transfer-learning method:

A Quantile-based Approach for Hyperparameter Transfer Learning.
David Salinas, Huibin Shen, Valerio Perrone.
ICML 2020.

This is the Copula Thompson Sampling approach described in the paper where a surrogate is fitted on the transfer learning data to predict mean and variance of configuration performance given a hyperparameter. The surrogate is then sampled from, and the best configurations are returned as next candidate to evaluate.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • resource_attr (str) – Name of resource attribute

  • transfer_learning_evaluations (Dict[str, TransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.

  • mode (str) – Whether to minimize (min) or maximize (max)

  • random_seed (Optional[int]) – Used for randomly sampling candidates

  • kwargs – Additional arguments to HyperbandScheduler

class syne_tune.optimizer.baselines.KDE(config_space, metric, **kwargs)[source]

Bases: FIFOScheduler

Single-fidelity variant of BOHB

Combines FIFOScheduler with TPE-like Bayesian optimization, using kernel density estimators.

See KernelDensityEstimator for kwargs["search_options"] parameters.

Parameters:

  • config_space (Dict[str, Any]) – Configuration space for evaluation function

  • metric (str) – Name of metric to optimize

  • kwargs – Additional arguments to FIFOScheduler

class syne_tune.optimizer.baselines.CQR(config_space, metric, mode='min', random_seed=None, **kwargs)[source]

Bases: FIFOScheduler

Single-fidelity Conformal Quantile Regression approach proposed in:
Optimizing Hyperparameters with Conformal Quantile Regression.
David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, Cedric Archambeau.
ICML 2023.

The method predict quantile performance with gradient boosted trees and calibrate prediction with conformal predictions.

class syne_tune.optimizer.baselines.ASHACQR(config_space, metric, resource_attr, mode='min', random_seed=None, **kwargs)[source]

Bases: HyperbandScheduler

Multi-fidelity Conformal Quantile Regression approach proposed in:
Optimizing Hyperparameters with Conformal Quantile Regression.
David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, Cedric Archambeau.
ICML 2023.

The method predict quantile performance with gradient boosted trees and calibrate prediction with conformal predictions.