syne_tune.optimizer.schedulers.searchers.kde package

class syne_tune.optimizer.schedulers.searchers.kde.KernelDensityEstimator(config_space, metric, points_to_evaluate=None, allow_duplicates=None, mode=None, num_min_data_points=None, top_n_percent=None, min_bandwidth=None, num_candidates=None, bandwidth_factor=None, random_fraction=None, **kwargs)[source]

Bases: StochasticAndFilterDuplicatesSearcher

Fits two kernel density estimators (KDE) to model the density of the top N configurations as well as the density of the configurations that are not among the top N, respectively. New configurations are sampled by optimizing the ratio of these two densities. KDE as model for Bayesian optimization has been originally proposed by Bergstra et al. Compared to their original implementation TPE, we use multi-variate instead of univariate KDE, as proposed by Falkner et al. Code is based on the implementation by Falkner et al: https://github.com/automl/HpBandSter/tree/master/hpbandster

Algorithms for Hyper-Parameter Optimization
J. Bergstra and R. Bardenet and Y. Bengio and B. K{‘e}gl
Proceedings of the 24th International Conference on Advances in Neural Information Processing Systems
https://papers.nips.cc/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html

and

BOHB: Robust and Efficient Hyperparameter Optimization at Scale
S. Falkner and A. Klein and F. Hutter
Proceedings of the 35th International Conference on Machine Learning
https://arxiv.org/abs/1807.01774

Note: restrict_configurations is not supported here, this would require reimplementing the selection of configs in _get_config().

Additional arguments on top of parent class StochasticAndFilterDuplicatesSearcher:

Parameters:

  • mode (Optional[str]) – Mode to use for the metric given, can be “min” or “max”. Is obtained from scheduler in configure_scheduler(). Defaults to “min”

  • num_min_data_points (Optional[int]) – Minimum number of data points that we use to fit the KDEs. As long as less observations have been received in update(), randomly drawn configurations are returned in get_config(). If set to None, we set this to the number of hyperparameters. Defaults to None.

  • top_n_percent (Optional[int]) – Determines how many datapoints we use to fit the first KDE model for modeling the well performing configurations. Defaults to 15

  • min_bandwidth (Optional[float]) – The minimum bandwidth for the KDE models. Defaults to 1e-3

  • num_candidates (Optional[int]) – Number of candidates that are sampled to optimize the acquisition function. Defaults to 64

  • bandwidth_factor (Optional[int]) – We sample continuous hyperparameter from a truncated Normal. This factor is multiplied to the bandwidth to define the standard deviation of this truncated Normal. Defaults to 3

  • random_fraction (Optional[float]) – Defines the fraction of configurations that are drawn uniformly at random instead of sampling from the model. Defaults to 0.33

configure_scheduler(scheduler)[source]

Some searchers need to obtain information from the scheduler they are used with, in order to configure themselves. This method has to be called before the searcher can be used.

Parameters:

scheduler (TrialScheduler) – Scheduler the searcher is used with.

clone_from_state(state)[source]

Together with get_state(), this is needed in order to store and re-create the mutable state of the searcher.

Given state as returned by get_state(), this method combines the non-pickle-able part of the immutable state from self with state and returns the corresponding searcher clone. Afterwards, self is not used anymore.

Parameters:

state (Dict[str, Any]) – See above

Returns:

New searcher object

class syne_tune.optimizer.schedulers.searchers.kde.MultiFidelityKernelDensityEstimator(config_space, metric, points_to_evaluate=None, allow_duplicates=None, mode=None, num_min_data_points=None, top_n_percent=None, min_bandwidth=None, num_candidates=None, bandwidth_factor=None, random_fraction=None, resource_attr=None, **kwargs)[source]

Bases: KernelDensityEstimator

Adapts KernelDensityEstimator to the multi-fidelity setting as proposed by Falkner et al such that we can use it with Hyperband. Following Falkner et al, we fit the KDE only on the highest resource level where we have at least num_min_data_points. Code is based on the implementation by Falkner et al: https://github.com/automl/HpBandSter/tree/master/hpbandster

BOHB: Robust and Efficient Hyperparameter Optimization at Scale
S. Falkner and A. Klein and F. Hutter
Proceedings of the 35th International Conference on Machine Learning

Additional arguments on top of parent class KernelDensityEstimator:

Parameters:

resource_attr (Optional[str]) – Name of resource attribute. Defaults to scheduler.resource_attr in configure_scheduler()

configure_scheduler(scheduler)[source]

Some searchers need to obtain information from the scheduler they are used with, in order to configure themselves. This method has to be called before the searcher can be used.

Parameters:

scheduler (TrialScheduler) – Scheduler the searcher is used with.

Submodules