syne_tune.optimizer.schedulers.searchers.kde package

class syne_tune.optimizer.schedulers.searchers.kde.KernelDensityEstimator(config_space, points_to_evaluate=None, num_min_data_points=None, top_n_percent=15, min_bandwidth=0.001, num_candidates=64, bandwidth_factor=3, random_fraction=0.33, random_seed=None)[source]

Bases: SingleObjectiveBaseSearcher

Fits two kernel density estimators (KDE) to model the density of the top N configurations as well as the density of the configurations that are not among the top N, respectively. New configurations are sampled by optimizing the ratio of these two densities. KDE as model for Bayesian optimization has been originally proposed by Bergstra et al. Compared to their original implementation TPE, we use multi-variate instead of univariate KDE, as proposed by Falkner et al. Code is based on the implementation by Falkner et al: https://github.com/automl/HpBandSter/tree/master/hpbandster

Algorithms for Hyper-Parameter Optimization
J. Bergstra and R. Bardenet and Y. Bengio and B. K{‘e}gl
Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems
https://papers.nips.cc/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html

and

BOHB: Robust and Efficient Hyperparameter Optimization at Scale
S. Falkner and A. Klein and F. Hutter
Proceedings of the 35th International Conference on Machine Learning
https://arxiv.org/abs/1807.01774

Note: restrict_configurations is not supported here, this would require reimplementing the selection of configs in _get_config().

Additional arguments on top of parent class StochasticAndFilterDuplicatesSearcher:

Parameters:

  • mode – Mode to use for the metric given, can be “min” or “max”. Is obtained from scheduler in configure_scheduler(). Defaults to “min”

  • num_min_data_points (Optional[int]) – Minimum number of data points that we use to fit the KDEs. As long as less observations have been received in update(), randomly drawn configurations are returned in get_config(). If set to None, we set this to the number of hyperparameters. Defaults to None.

  • top_n_percent (int) – Determines how many datapoints we use to fit the first KDE model for modeling the well performing configurations. Defaults to 15

  • min_bandwidth (float) – The minimum bandwidth for the KDE models. Defaults to 1e-3

  • num_candidates (int) – Number of candidates that are sampled to optimize the acquisition function. Defaults to 64

  • bandwidth_factor (int) – We sample continuous hyperparameter from a truncated Normal. This factor is multiplied to the bandwidth to define the standard deviation of this truncated Normal. Defaults to 3

  • random_fraction (float) – Defines the fraction of configurations that are drawn uniformly at random instead of sampling from the model. Defaults to 0.33

on_trial_complete(trial_id, config, metric)[source]

Inform searcher about result

The scheduler passes every result. If update == True, the searcher should update its surrogate model (if any), otherwise result is an intermediate result not modelled.

The default implementation calls _update() if update == True. It can be overwritten by searchers which also react to intermediate results.

Parameters:

  • trial_id (int) – See on_trial_result()

  • config (Dict[str, Any]) – See on_trial_result()

  • metric (float) – See on_trial_result()

suggest(**kwargs)[source]

Suggest a new configuration.

Note: Query _next_points_to_evaluate() for initial configs to return first.

Parameters:

kwargs – Extra information may be passed from scheduler to searcher

Return type:

Optional[Dict[str, Any]]

Returns:

New configuration. The searcher may return None if a new configuration cannot be suggested. In this case, the tuning will stop. This happens if searchers never suggest the same config more than once, and all configs in the (finite) search space are exhausted.

class syne_tune.optimizer.schedulers.searchers.kde.MultiFidelityKernelDensityEstimator(config_space, points_to_evaluate=None, num_min_data_points=None, top_n_percent=15, min_bandwidth=0.001, num_candidates=64, bandwidth_factor=3, random_fraction=0.33, random_seed=None)[source]

Bases: KernelDensityEstimator

Adapts KernelDensityEstimator to the multi-fidelity setting as proposed by Falkner et al such that we can use it with Hyperband. Following Falkner et al, we fit the KDE only on the highest resource level where we have at least num_min_data_points. Code is based on the implementation by Falkner et al: https://github.com/automl/HpBandSter/tree/master/hpbandster

BOHB: Robust and Efficient Hyperparameter Optimization at Scale
S. Falkner and A. Klein and F. Hutter
Proceedings of the 35th International Conference on Machine Learning

Additional arguments on top of parent class KernelDensityEstimator:

on_trial_result(trial_id, config, metric, resource_level)[source]

Inform searcher about result

The scheduler passes every result. If update == True, the searcher should update its surrogate model (if any), otherwise result is an intermediate result not modelled.

The default implementation calls _update() if update == True. It can be overwritten by searchers which also react to intermediate results.

Parameters:

  • trial_id (int) – See on_trial_result()

  • config (Dict[str, Any]) – See on_trial_result()

  • metric (float) – See on_trial_result()

Submodules