syne_tune.optimizer.baselines module

class syne_tune.optimizer.baselines.RandomSearch(config_space, metrics, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: SingleFidelityScheduler

Random search that samples hyperparameter configurations uniformly at random in each iteration. Supports both single- and multi-objective optimization.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metrics (List[str]) – Name(s) of the metric(s) to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.BORE(config_space, metric, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: SingleObjectiveScheduler

Bayesian Optimization by Density-Ratio Estimation (BORE) as proposed by:

BORE: Bayesian Optimization by Density-Ratio Estimation,

Tiao, Louis C and Klein, Aaron and Seeger, Matthias W and Bonilla, Edwin V. and Archambeau, Cedric and Ramos, Fabio

Proceedings of the 38th International Conference on Machine Learning

https://arxiv.org/abs/2102.09009

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.TPE(config_space, metric, do_minimize=True, random_seed=None, points_to_evaluate=None, num_min_data_points=None, top_n_percent=15, min_bandwidth=0.001, num_candidates=64, bandwidth_factor=3, random_fraction=0.33)[source]

Bases: SingleObjectiveScheduler

Tree-Parzen Estimator as proposed by:

Algorithms for Hyper-Parameter Optimization

J. Bergstra and R. Bardenet and Y. Bengio and B. K{‘e}gl

Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems

https://papers.nips.cc/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.
num_min_data_points (Optional[int]) – Minimum number of data points that we use to fit the KDEs. As long as less observations have been received in update(), randomly drawn configurations are returned in get_config(). If set to None, we set this to the number of hyperparameters. Defaults to None.
top_n_percent (int) – Determines how many datapoints we use to fit the first KDE model for modeling the well performing configurations. Defaults to 15
min_bandwidth (float) – The minimum bandwidth for the KDE models. Defaults to 1e-3
num_candidates (int) – Number of candidates that are sampled to optimize the acquisition function. Defaults to 64
bandwidth_factor (int) – We sample continuous hyperparameter from a truncated Normal. This factor is multiplied to the bandwidth to define the standard deviation of this truncated Normal. Defaults to 3
random_fraction (float) – Defines the fraction of configurations that are drawn uniformly at random instead of sampling from the model. Defaults to 0.33

class syne_tune.optimizer.baselines.REA(config_space, metric, do_minimize=True, random_seed=None, population_size=100, sample_size=10, points_to_evaluate=None)[source]

Bases: SingleObjectiveScheduler

Regularized Evolution as proposed by

Real, E., Aggarwal, A., Huang, Y., and Le, Q. V.

Regularized Evolution for Image Classifier Architecture Search.

In Proceedings of the Conference on Artificial Intelligence (AAAI’19)

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
population_size (int) – Size of the population, defaults to 100
sample_size (int) – Size of the candidate set to obtain a parent for the mutation, defaults to 10
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.BOTorch(config_space, metric, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: SingleObjectiveScheduler

Implements Gaussian-process based Bayesian optimization based on BOTorch.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.ASHA(config_space, metric, time_attr, max_t, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: AsynchronousSuccessiveHalving

Asynchronous Successive Halving (ASHA) as proposed by:

Li, Jamieson, Rostamizadeh, Gonina, Hardt, Recht, Talwalkar (2018)

A System for Massively Parallel Hyperparameter Tuning

https://arxiv.org/abs/1810.05934

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
time_attr (str) – A training result attr to use for comparing time. Note that you can pass in something non-temporal such as training_iteration as a measure of progress, the only requirement is that the attribute should increase monotonically. Defaults to “training_iteration”
max_t (int) – max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have passed. Defaults to 100
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.ASHABORE(config_space, metric, time_attr, max_t, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: AsynchronousSuccessiveHalving

Asynchronous Successive Halving (ASHA) which fits a BORE model on each rung level.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
time_attr (str) – A training result attr to use for comparing time. Note that you can pass in something non-temporal such as training_iteration as a measure of progress, the only requirement is that the attribute should increase monotonically. Defaults to “training_iteration”
max_t (int) – max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have passed. Defaults to 100
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.ASHACQR(config_space, metric, time_attr, max_t, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: AsynchronousSuccessiveHalving

Asynchronous Successive Halving (ASHA) which fits a CQR model on each rung level.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
time_attr (str) – A training result attr to use for comparing time. Note that you can pass in something non-temporal such as training_iteration as a measure of progress, the only requirement is that the attribute should increase monotonically. Defaults to “training_iteration”
max_t (int) – max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have passed. Defaults to 100
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.BOHB(config_space, metric, time_attr, max_t, do_minimize=True, random_seed=None, num_min_data_points=None, top_n_percent=15, min_bandwidth=0.001, num_candidates=64, bandwidth_factor=3, random_fraction=0.33, points_to_evaluate=None)[source]

Bases: AsynchronousSuccessiveHalving

Bayesian Optimization Hyperband combines ASHA with TPE-like Bayesian optimization, using kernel density estimators as proposed by:

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

S. Falkner and A. Klein and F. Hutter

Proceedings of the 35th International Conference on Machine Learning

https://arxiv.org/abs/1807.01774

Compared to the method proposed by Falkner et al. we use asynchronous successive halving for scheduling.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.

class syne_tune.optimizer.baselines.CQR(config_space, metric, do_minimize=True, random_seed=None, points_to_evaluate=None)[source]

Bases: SingleObjectiveScheduler

Single-fidelity Conformal Quantile Regression approach proposed in:: Optimizing Hyperparameters with Conformal Quantile Regression.

David Salinas, Jacek Golebiowski, Aaron Klein, Matthias Seeger, Cedric Archambeau.

ICML 2023.

The method predict quantile performance with gradient boosted trees and calibrate prediction with conformal predictions.

Parameters:

config_space (Dict[str, Any]) – Configuration space for the evaluation function.
metric (str) – Name of the metric to optimize.
do_minimize (Optional[bool]) – Set to True if the objective function should be minimized.
random_seed (Optional[int]) – Seed for initializing random number generators.
points_to_evaluate (Optional[List[dict]]) – A set of initial configurations to be evaluated before starting the optimization.