syne_tune.optimizer.schedulers.searchers.bayesopt.gpautograd.hypertune.gp_model module

class syne_tune.optimizer.schedulers.searchers.bayesopt.gpautograd.hypertune.gp_model.HyperTuneDistributionArguments(num_samples, num_brackets=None)[source]

Bases: object

num_samples: int

num_brackets: Optional[int] = None

class syne_tune.optimizer.schedulers.searchers.bayesopt.gpautograd.hypertune.gp_model.HyperTuneModelMixin(hypertune_distribution_args)[source]

Bases: object

hypertune_bracket_distribution()[source]

Distribution [w_k] of support size num_supp_brackets, where num_supp_brackets <= args.num_brackets (the latter is maximum if not given) is maximum such that the first num_supp_brackets brackets have >= 6 labeled datapoints each.

If num_supp_brackets < args.num_brackets, the distribution must be extended to full size before being used to sample the next bracket.

Return type:: Optional[ndarray]

hypertune_ensemble_distribution()[source]

Distribution [theta_r] which is used to create an ensemble predictive distribution fed into the acquisition function. The ensemble distribution runs over all sufficiently supported rung levels, independent of the number of brackets.

Return type:: Optional[Dict[int, float]]

fit_distributions(poster_state, data, resource_attr_range, random_state)[source]

Return type:: Optional[Dict[int, float]]

class syne_tune.optimizer.schedulers.searchers.bayesopt.gpautograd.hypertune.gp_model.HyperTuneIndependentGPModel(kernel, mean_factory, resource_attr_range, hypertune_distribution_args, target_transform=None, separate_noise_variances=False, initial_noise_variance=None, initial_covariance_scale=None, optimization_config=None, random_seed=None, fit_reset_params=True)[source]

Bases: IndependentGPPerResourceModel, HyperTuneModelMixin

Variant of IndependentGPPerResourceModel which implements additional features of the Hyper-Tune algorithm, see

Yang Li et al

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

VLDB 2022

Our implementation differs from the Hyper-Tune paper in a number of ways. Most importantly, their method requires a sufficient number of observed points at the starting rung of the highest bracket. In contrast, we estimate ranking loss values already when the starting rung of the 2nd bracket is sufficiently occupied. This allows us to estimate the head of the distribution only (over all brackets with sufficiently occupied starting rungs), and we use the default distribution over the remaining tail. Eventually, we do the same as Hyper-Tune, but we move away from the default distribution earlier on.

Parameters:: hypertune_distribution_args (HyperTuneDistributionArguments) – Parameters for Hyper-Tune

create_likelihood(rung_levels)[source]

Delayed creation of likelihood, needs to know rung levels of Hyperband scheduler.

Note: last entry of rung_levels must be max_t, even if this is not a rung level in Hyperband.

Parameters:: rung_levels (List[int]) – Rung levels

hypertune_ensemble_distribution()[source]

Distribution [theta_r] which is used to create an ensemble predictive distribution fed into the acquisition function. The ensemble distribution runs over all sufficiently supported rung levels, independent of the number of brackets.

Return type:: Optional[Dict[int, float]]

fit(data)[source]

Fit the model parameters by optimizing the marginal likelihood, and set posterior states.

We catch exceptions during the optimization restarts. If any restarts fail, log messages are written. If all restarts fail, the current parameters are not changed.

Parameters:: data (Dict[str, Any]) – Input data

class syne_tune.optimizer.schedulers.searchers.bayesopt.gpautograd.hypertune.gp_model.HyperTuneJointGPModel(kernel, resource_attr_range, hypertune_distribution_args, mean=None, target_transform=None, initial_noise_variance=None, optimization_config=None, random_seed=None, fit_reset_params=True)[source]

Bases: GaussianProcessRegression, HyperTuneModelMixin

Variant of GaussianProcessRegression which implements additional features of the Hyper-Tune algorithm, see

Yang Li et al Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale VLDB 2022