syne_tune.optimizer.schedulers.transfer_learning package

class syne_tune.optimizer.schedulers.transfer_learning.TransferLearningTaskEvaluations(configuration_space, hyperparameters, objectives_names, objectives_evaluations)[source]

Bases: object

Class that contains offline evaluations for a task that can be used for transfer learning. Args:

configuration_space: Dict the configuration space that was used when sampling evaluations. hyperparameters: pd.DataFrame the hyperparameters values that were acquired, all keys of configuration-space

should appear as columns.

objectives_names: List[str] the name of the objectives that were acquired objectives_evaluations: np.array values of recorded objectives, must have shape

(num_evals, num_seeds, num_fidelities, num_objectives)

configuration_space: Dict
hyperparameters: DataFrame
objectives_names: List[str]
objectives_evaluations: array
objective_values(objective_name)[source]
Return type:

array

objective_index(objective_name)[source]
Return type:

int

top_k_hyperparameter_configurations(k, mode, objective)[source]

Returns the best k hyperparameter configurations. :type k: int :param k: The number of top hyperparameters to return. :type mode: str :param mode: ‘min’ or ‘max’, indicating the type of optimization problem. :type objective: str :param objective: The objective to consider for ranking hyperparameters. :rtype: List[Dict[str, Any]] :returns: List of hyperparameters in order.

class syne_tune.optimizer.schedulers.transfer_learning.TransferLearningMixin(config_space, transfer_learning_evaluations, metric_names, **kwargs)[source]

Bases: object

metric_names()[source]
Return type:

List[str]

top_k_hyperparameter_configurations_per_task(transfer_learning_evaluations, num_hyperparameters_per_task, mode, metric)[source]

Returns the best hyperparameter configurations for each task. :type transfer_learning_evaluations: Dict[str, TransferLearningTaskEvaluations] :param transfer_learning_evaluations: Set of candidates to choose from. :type num_hyperparameters_per_task: int :param num_hyperparameters_per_task: The number of top hyperparameters per task to return. :type mode: str :param mode: ‘min’ or ‘max’, indicating the type of optimization problem. :type metric: str :param metric: The metric to consider for ranking hyperparameters. :rtype: Dict[str, List[Dict[str, Any]]] :returns: Dict which maps from task name to list of hyperparameters in order.

class syne_tune.optimizer.schedulers.transfer_learning.BoundingBox(scheduler_fun, config_space, metric, transfer_learning_evaluations, mode=None, num_hyperparameters_per_task=1)[source]

Bases: TransferLearningMixin, TrialScheduler

Simple baseline that computes a bounding-box of the best candidate found in previous tasks to restrict the search space to only good candidates. The bounding-box is obtained by restricting to the min-max of the best numerical hyperparameters and restricting to the set of the best candidates on categorical parameters. Reference:

Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning.
Valerio Perrone, Huibin Shen, Matthias Seeger, Cédric Archambeau, Rodolphe Jenatton.
NeurIPS 2019.

scheduler_fun is used to create the scheduler to be used here, feeding it with the modified config space. Any additional scheduler arguments (such as points_to_evaluate) should be encoded inside this function. Example:

from syne_tune.optimizer.baselines import RandomSearch

def scheduler_fun(new_config_space: Dict[str, Any], mode: str, metric: str):
    return RandomSearch(new_config_space, metric, mode)

bb_scheduler = BoundingBox(scheduler_fun, ...)

Here, bb_scheduler represents random search, where the hyperparameter ranges are restricted to contain the best evalutions of previous tasks, as provided by transfer_learning_evaluations.

Parameters:

  • scheduler_fun (Callable[[dict, str, str], TrialScheduler]) – Maps tuple of configuration space (dict), mode (str), metric (str) to a scheduler. This is required since the final configuration space is known only after computing a bounding-box.

  • config_space (Dict[str, Any]) – Initial configuration space to consider, will be updated to the bounding of the best evaluations of previous tasks

  • metric (str) – Objective name to optimize, must be present in transfer learning evaluations.

  • mode (Optional[str]) – Mode to be considered, default to “min”.

  • transfer_learning_evaluations (Dict[str, TransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.

  • num_hyperparameters_per_task (int) – Number of the best configurations to use per task when computing the bounding box, defaults to 1.

suggest(trial_id)[source]

Returns a suggestion for a new trial, or one to be resumed

This method returns suggestion of type TrialSuggestion (unless there is no config left to explore, and None is returned).

If suggestion.spawn_new_trial_id is True, a new trial is to be started with config suggestion.config. Typically, this new trial is started from scratch. But if suggestion.checkpoint_trial_id is given, the trial is to be (warm)started from the checkpoint written for the trial with this ID. The new trial has ID trial_id.

If suggestion.spawn_new_trial_id is False, an existing and currently paused trial is to be resumed, whose ID is suggestion.checkpoint_trial_id. If this trial has a checkpoint, we start from there. In this case, suggestion.config is optional. If not given (default), the config of the resumed trial does not change. Otherwise, its config is overwritten by suggestion.config (see HyperbandScheduler with type="promotion" for an example why this can be useful).

Apart from the HP config, additional fields can be appended to the dict, these are passed to the trial function as well.

Parameters:

trial_id (int) – ID for new trial to be started (ignored if existing trial to be resumed)

Return type:

Optional[TrialSuggestion]

Returns:

Suggestion for a trial to be started or to be resumed, see above. If no suggestion can be made, None is returned

on_trial_add(trial)[source]

Called when a new trial is added to the trial runner.

Additions are normally triggered by suggest.

Parameters:

trial (Trial) – Trial to be added

on_trial_complete(trial, result)[source]

Notification for the completion of trial.

Note that on_trial_result() is called with the same result before. However, if the scheduler only uses one final report from each trial, it may ignore on_trial_result() and just use result here.

Parameters:

  • trial (Trial) – Trial which is completing

  • result (Dict[str, Any]) – Result dictionary

on_trial_remove(trial)[source]

Called to remove trial.

This is called when the trial is in PAUSED or PENDING state. Otherwise, call on_trial_complete().

Parameters:

trial (Trial) – Trial to be removed

on_trial_error(trial)[source]

Called when a trial has failed.

Parameters:

trial (Trial) – Trial for which error is reported.

on_trial_result(trial, result)[source]

Called on each intermediate result reported by a trial.

At this point, the trial scheduler can make a decision by returning one of SchedulerDecision.CONTINUE, SchedulerDecision.PAUSE, or SchedulerDecision.STOP. This will only be called when the trial is currently running.

Parameters:

  • trial (Trial) – Trial for which results are reported

  • result (Dict[str, Any]) – Result dictionary

Return type:

str

Returns:

Decision what to do with the trial

metric_mode()[source]
Return type:

str

Returns:

“min” if target metric is minimized, otherwise “max”. Here, “min” should be the default. For a genuine multi-objective scheduler, a list of modes is returned

class syne_tune.optimizer.schedulers.transfer_learning.RUSHScheduler(config_space, transfer_learning_evaluations, metric, type='stopping', points_to_evaluate=None, custom_rush_points=None, num_hyperparameters_per_task=1, **kwargs)[source]

Bases: TransferLearningMixin, HyperbandScheduler

A transfer learning variation of Hyperband which uses previously well-performing hyperparameter configurations as an initialization. The best hyperparameter configuration of each individual task provided is evaluated. The one among them which performs best on the current task will serve as a hurdle and is used to prune other candidates. This changes the standard successive halving promotion as follows. As usual, only the top-performing fraction is promoted to the next rung level. However, these candidates need to be at least as good as the hurdle configuration to be promoted. In practice this means that much fewer candidates can be promoted. Reference:

A resource-efficient method for repeated HPO and NAS.
Giovanni Zappella, David Salinas, Cédric Archambeau.
AutoML workshop @ ICML 2021.

Additional arguments on top of parent class HyperbandScheduler.

Parameters:

  • transfer_learning_evaluations (Dict[str, TransferLearningTaskEvaluations]) – Dictionary from task name to offline evaluations.

  • points_to_evaluate (Optional[List[dict]]) – If given, these configurations are evaluated after custom_rush_points and configurations inferred from transfer_learning_evaluations. These points are not used to prune any configurations.

  • custom_rush_points (Optional[List[dict]]) – If given, these configurations are evaluated first, in addition to top performing configurations from other tasks and also serve to preemptively prune underperforming configurations

  • num_hyperparameters_per_task (int) – The number of top hyperparameter configurations to consider per task. Defaults to 1

Subpackages

Submodules