syne_tune.optimizer.schedulers.transfer_learning package
- class syne_tune.optimizer.schedulers.transfer_learning.TransferLearningTaskEvaluations(configuration_space, hyperparameters, objectives_names, objectives_evaluations)[source]
Bases:
object
Class that contains offline evaluations for a task that can be used for transfer learning. Args:
configuration_space: Dict the configuration space that was used when sampling evaluations. hyperparameters: pd.DataFrame the hyperparameters values that were acquired, all keys of configuration-space
should appear as columns.
objectives_names: List[str] the name of the objectives that were acquired objectives_evaluations: np.array values of recorded objectives, must have shape
(num_evals, num_seeds, num_fidelities, num_objectives)
-
configuration_space:
Dict
-
hyperparameters:
DataFrame
-
objectives_names:
List
[str
]
-
objectives_evaluations:
array
- top_k_hyperparameter_configurations(k, mode, objective)[source]
Returns the best k hyperparameter configurations. :type k:
int
:param k: The number of top hyperparameters to return. :type mode:str
:param mode: ‘min’ or ‘max’, indicating the type of optimization problem. :type objective:str
:param objective: The objective to consider for ranking hyperparameters. :rtype:List
[Dict
[str
,Any
]] :returns: List of hyperparameters in order.
-
configuration_space:
- class syne_tune.optimizer.schedulers.transfer_learning.TransferLearningMixin(config_space, transfer_learning_evaluations, metric_names, **kwargs)[source]
Bases:
object
- top_k_hyperparameter_configurations_per_task(transfer_learning_evaluations, num_hyperparameters_per_task, mode, metric)[source]
Returns the best hyperparameter configurations for each task. :type transfer_learning_evaluations:
Dict
[str
,TransferLearningTaskEvaluations
] :param transfer_learning_evaluations: Set of candidates to choose from. :type num_hyperparameters_per_task:int
:param num_hyperparameters_per_task: The number of top hyperparameters per task to return. :type mode:str
:param mode: ‘min’ or ‘max’, indicating the type of optimization problem. :type metric:str
:param metric: The metric to consider for ranking hyperparameters. :rtype:Dict
[str
,List
[Dict
[str
,Any
]]] :returns: Dict which maps from task name to list of hyperparameters in order.
- class syne_tune.optimizer.schedulers.transfer_learning.BoundingBox(scheduler_fun, config_space, metric, transfer_learning_evaluations, mode=None, num_hyperparameters_per_task=1)[source]
Bases:
TransferLearningMixin
,TrialScheduler
Simple baseline that computes a bounding-box of the best candidate found in previous tasks to restrict the search space to only good candidates. The bounding-box is obtained by restricting to the min-max of the best numerical hyperparameters and restricting to the set of the best candidates on categorical parameters. Reference:
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning.Valerio Perrone, Huibin Shen, Matthias Seeger, Cédric Archambeau, Rodolphe Jenatton.NeurIPS 2019.scheduler_fun
is used to create the scheduler to be used here, feeding it with the modified config space. Any additional scheduler arguments (such aspoints_to_evaluate
) should be encoded inside this function. Example:from syne_tune.optimizer.baselines import RandomSearch def scheduler_fun(new_config_space: Dict[str, Any], mode: str, metric: str): return RandomSearch(new_config_space, metric, mode) bb_scheduler = BoundingBox(scheduler_fun, ...)
Here,
bb_scheduler
represents random search, where the hyperparameter ranges are restricted to contain the best evalutions of previous tasks, as provided bytransfer_learning_evaluations
.- Parameters:
scheduler_fun (
Callable
[[dict
,str
,str
],TrialScheduler
]) – Maps tuple of configuration space (dict), mode (str), metric (str) to a scheduler. This is required since the final configuration space is known only after computing a bounding-box.config_space (
Dict
[str
,Any
]) – Initial configuration space to consider, will be updated to the bounding of the best evaluations of previous tasksmetric (
str
) – Objective name to optimize, must be present in transfer learning evaluations.mode (
Optional
[str
]) – Mode to be considered, default to “min”.transfer_learning_evaluations (
Dict
[str
,TransferLearningTaskEvaluations
]) – Dictionary from task name to offline evaluations.num_hyperparameters_per_task (
int
) – Number of the best configurations to use per task when computing the bounding box, defaults to 1.
- suggest(trial_id)[source]
Returns a suggestion for a new trial, or one to be resumed
This method returns
suggestion
of typeTrialSuggestion
(unless there is no config left to explore, and None is returned).If
suggestion.spawn_new_trial_id
isTrue
, a new trial is to be started with configsuggestion.config
. Typically, this new trial is started from scratch. But ifsuggestion.checkpoint_trial_id
is given, the trial is to be (warm)started from the checkpoint written for the trial with this ID. The new trial has IDtrial_id
.If
suggestion.spawn_new_trial_id
isFalse
, an existing and currently paused trial is to be resumed, whose ID issuggestion.checkpoint_trial_id
. If this trial has a checkpoint, we start from there. In this case,suggestion.config
is optional. If not given (default), the config of the resumed trial does not change. Otherwise, its config is overwritten bysuggestion.config
(seeHyperbandScheduler
withtype="promotion"
for an example why this can be useful).Apart from the HP config, additional fields can be appended to the dict, these are passed to the trial function as well.
- Parameters:
trial_id (
int
) – ID for new trial to be started (ignored if existing trial to be resumed)- Return type:
Optional
[TrialSuggestion
]- Returns:
Suggestion for a trial to be started or to be resumed, see above. If no suggestion can be made, None is returned
- on_trial_add(trial)[source]
Called when a new trial is added to the trial runner.
Additions are normally triggered by
suggest
.- Parameters:
trial (
Trial
) – Trial to be added
- on_trial_complete(trial, result)[source]
Notification for the completion of trial.
Note that
on_trial_result()
is called with the same result before. However, if the scheduler only uses one final report from each trial, it may ignoreon_trial_result()
and just useresult
here.- Parameters:
trial (
Trial
) – Trial which is completingresult (
Dict
[str
,Any
]) – Result dictionary
- on_trial_remove(trial)[source]
Called to remove trial.
This is called when the trial is in PAUSED or PENDING state. Otherwise, call
on_trial_complete()
.- Parameters:
trial (
Trial
) – Trial to be removed
- on_trial_error(trial)[source]
Called when a trial has failed.
- Parameters:
trial (
Trial
) – Trial for which error is reported.
- on_trial_result(trial, result)[source]
Called on each intermediate result reported by a trial.
At this point, the trial scheduler can make a decision by returning one of
SchedulerDecision.CONTINUE
,SchedulerDecision.PAUSE
, orSchedulerDecision.STOP
. This will only be called when the trial is currently running.- Parameters:
trial (
Trial
) – Trial for which results are reportedresult (
Dict
[str
,Any
]) – Result dictionary
- Return type:
str
- Returns:
Decision what to do with the trial
- class syne_tune.optimizer.schedulers.transfer_learning.RUSHScheduler(config_space, transfer_learning_evaluations, metric, type='stopping', points_to_evaluate=None, custom_rush_points=None, num_hyperparameters_per_task=1, **kwargs)[source]
Bases:
TransferLearningMixin
,HyperbandScheduler
A transfer learning variation of Hyperband which uses previously well-performing hyperparameter configurations as an initialization. The best hyperparameter configuration of each individual task provided is evaluated. The one among them which performs best on the current task will serve as a hurdle and is used to prune other candidates. This changes the standard successive halving promotion as follows. As usual, only the top-performing fraction is promoted to the next rung level. However, these candidates need to be at least as good as the hurdle configuration to be promoted. In practice this means that much fewer candidates can be promoted. Reference:
A resource-efficient method for repeated HPO and NAS.Giovanni Zappella, David Salinas, Cédric Archambeau.AutoML workshop @ ICML 2021.Additional arguments on top of parent class
HyperbandScheduler
.- Parameters:
transfer_learning_evaluations (
Dict
[str
,TransferLearningTaskEvaluations
]) – Dictionary from task name to offline evaluations.points_to_evaluate (
Optional
[List
[dict
]]) – If given, these configurations are evaluated aftercustom_rush_points
and configurations inferred fromtransfer_learning_evaluations
. These points are not used to prune any configurations.custom_rush_points (
Optional
[List
[dict
]]) – If given, these configurations are evaluated first, in addition to top performing configurations from other tasks and also serve to preemptively prune underperforming configurationsnum_hyperparameters_per_task (
int
) – The number of top hyperparameter configurations to consider per task. Defaults to 1