syne_tune.blackbox_repository package

class syne_tune.blackbox_repository.BlackboxOffline(df_evaluations, configuration_space, fidelity_space=None, objectives_names=None, seed_col=None)[source]

Bases: Blackbox

A blackbox obtained given offline evaluations. Each row of the dataframe should contain one evaluation given a fixed configuration, fidelity and seed. The columns must correspond to the provided configuration and fidelity space, by default all columns that are prefixed by "metric_" are assumed to be metrics but this can be overridden by providing metric columns.

Additional arguments on top of parent class Blackbox:

Parameters:

  • df_evaluations (DataFrame) – Data frame with evaluations data

  • seed_col (Optional[str]) – optional, can be used when multiple seeds are recorded

hyperparameter_objectives_values(predict_curves=False)[source]

If predict_curves is False, the shape of X is (num_evals * num_seeds * num_fidelities, num_hps + 1), the shape of y is (num_evals * num_seeds * num_fidelities, num_objectives). This can be reshaped to (num_fidelities, num_seeds, num_evals, *). The final column of X is the fidelity value (only a single fidelity attribute is supported).

If predict_curves is True, the shape of X is (num_evals * num_seeds, num_hps), the shape of y is (num_evals * num_seeds, num_fidelities * num_objectives). The latter can be reshaped to (num_seeds, num_evals, num_fidelities, num_objectives).

Returns:

a tuple of two dataframes (X, y), where X contains hyperparameters values and y contains objective values, this is used when fitting a surrogate model.

syne_tune.blackbox_repository.deserialize(path)[source]
Parameters:

  • path (str) – where to find blackbox serialized information (at least data.csv.zip and configspace.json)

  • groupby_col – separate evaluations into a list of blackbox with different task if the column is provided

Return type:

Union[Dict[str, BlackboxOffline], BlackboxOffline]

Returns:

list of blackboxes per task, or single blackbox in the case of a single task

syne_tune.blackbox_repository.load_blackbox(name, skip_if_present=True, s3_root=None, generate_if_not_found=True, yahpo_kwargs=None, ignore_hash=True)[source]
Parameters:

  • name (str) –

    name of a blackbox present in the repository, see blackbox_list() to get list of available blackboxes. Syne Tune currently provides the following blackboxes evaluations:

    • ”nasbench201”: 15625 multi-fidelity configurations of computer vision architectures evaluated on 3 datasets. NAS-Bench-201: Extending the scope of reproducible neural architecture search. Dong, X. and Yang, Y. 2020.

    • ”fcnet”: 62208 multi-fidelity configurations of MLP evaluated on 4 datasets. Tabular benchmarks for joint architecture and hyperparameter optimization. Klein, A. and Hutter, F. 2019.

    • ”lcbench”: 2000 multi-fidelity Pytorch model configurations evaluated on many datasets. Reference: Auto-PyTorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. Lucas Zimmer, Marius Lindauer, Frank Hutter. 2020.

    • ”icml-deepar”: 2420 single-fidelity configurations of DeepAR forecasting algorithm evaluated on 10 datasets. A quantile-based approach for hyperparameter transfer learning. Salinas, D., Shen, H., and Perrone, V. 2021.

    • ”icml-xgboost”: 5O00 single-fidelity configurations of XGBoost evaluated on 9 datasets. A quantile-based approach for hyperparameter transfer learning. Salinas, D., Shen, H., and Perrone, V. 2021.

    • ”yahpo-*”: Number of different benchmarks from YAHPO Gym. Note that these blackboxes come with surrogates already, so no need to wrap them into SurrogateBlackbox

  • skip_if_present (bool) – skip the download if the file locally exists

  • s3_root (Optional[str]) – S3 root directory for blackbox repository. Defaults to S3 bucket name of SageMaker session

  • generate_if_not_found (bool) – If the blackbox file is not present locally or on S3, should it be generated using its conversion script?

  • yahpo_kwargs (Optional[dict]) – For a YAHPO blackbox (name == "yahpo-*"), these are additional arguments to instantiate_yahpo

  • ignore_hash (bool) – do not check if hash of currently stored files matches the pre-computed hash. Be careful with this option. If hashes do not match, results might not be reproducible.

Return type:

Union[Dict[str, Blackbox], Blackbox]

Returns:

blackbox with the given name, download it if not present.

syne_tune.blackbox_repository.blackbox_list()[source]
Return type:

List[str]

Returns:

list of blackboxes available

syne_tune.blackbox_repository.add_surrogate(blackbox, surrogate=None, configuration_space=None, predict_curves=None, separate_seeds=False, fit_differences=None)[source]

Fits a blackbox surrogates that can be evaluated anywhere, which can be useful for supporting interpolation/extrapolation.

Parameters:

  • blackbox (Blackbox) – the blackbox must implement hyperparameter_objectives_values() so that input/output are passed to estimate the model

  • surrogate – the model that is fitted to predict objectives given any configuration. Possible examples: KNeighborsRegressor(n_neighbors=1), MLPRegressor() or any estimator obeying Scikit-learn API. The model is fit on top of pipeline that applies basic feature-processing to convert rows in X to vectors. We use configuration_space to deduce the types of columns in X (categorical parameters are one-hot encoded).

  • configuration_space (Optional[dict]) – configuration space for the resulting blackbox surrogate. The default is blackbox.configuration_space. But note that if blackbox is tabular, the domains in blackbox.configuration_space are typically categorical even for numerical parameters.

  • predict_curves (Optional[bool]) – If True, the surrogate uses multivariate regression to predict metric curves over fidelities. If False, fidelity is used as input. The latter can lead to inconsistent predictions along fidelity and is typically more expensive. If not given, the default value is False if blackbox is of type BlackboxOffline, otherwise True.

  • separate_seeds (bool) – If True, seeds in blackbox map to seeds in the surrogate blackbox, which fits different models to each seed. If False, the data from blackbox is merged for all seeds, and the surrogate represents a single seed. The latter provides more data for the surrogate model to be fit, but the variation between seeds is lost in the surrogate. Defaults to False.

  • fit_differences (Optional[List[str]]) – Names of objectives which are cumulative sums. For these objectives, the y data is transformed to finite differences before fitting the model. This is recommended for elapsed_time objectives.

Returns:

a blackbox where the output is obtained through the fitted surrogate

class syne_tune.blackbox_repository.BlackboxRepositoryBackend(blackbox_name, elapsed_time_attr, max_resource_attr=None, seed=None, support_checkpointing=True, dataset=None, surrogate=None, surrogate_kwargs=None, add_surrogate_kwargs=None, config_space_surrogate=None, **simulatorbackend_kwargs)[source]

Bases: _BlackboxSimulatorBackend

Allows to simulate a blackbox from blackbox-repository, selected by blackbox_name. See examples/launch_simulated_benchmark.py for an example on how to use. If you want to add a new dataset, see the Adding a new dataset section of syne_tune/blackbox_repository/README.md.

In each result reported to the simulator backend, the value for key elapsed_time_attr must be the time since the start of the evaluation. For example, if resource (or fidelity) equates to epochs trained, this would be the time from start of training until the end of the epoch. If the blackbox contains this information in a column, elapsed_time_attr should be its key.

If this backend is used with pause-and-resume multi-fidelity scheduling, it needs to track at which resource level each trial is paused. Namely, once a trial is resumed, all results for resources smaller or equal to that level are ignored, which simulates the situation that training is resumed from a checkpoint. This feature relies on result to be passed to pause_trial(). If this is not done, the backend cannot know from which resource level to resume a trial, so it starts the trial from scratch (which is equivalent to no checkpointing). The same happens if support_checkpointing is False.

Note

If the blackbox maintains cumulative time (elapsed_time), this is different from what SimulatorBackend requires for elapsed_time_attr, if a pause-and-resume scheduler is used. Namely, the backend requires the time since the start of the last recent resume. This conversion is done here internally in _run_job_and_collect_results(), which is called for each resume. This means that the field elapsed_time_attr is not what is received from the blackbox table, but instead what the backend needs.

max_resource_attr plays the same role as in HyperbandScheduler. If given, it is the key in a configuration config for the maximum resource. This is used by schedulers which limit each evaluation by setting this argument (e.g., promotion-based Hyperband).

If seed is given, entries of the blackbox are queried for this seed. Otherwise, a seed is drawn at random for every trial, but the same seed is used for all _run_job_and_collect_results() calls for the same trial. This is important for pause and resume scheduling.

Parameters:

  • blackbox_name (str) – Name of a blackbox, must have been registered in blackbox repository.

  • elapsed_time_attr (str) – Name of the column containing cumulative time

  • max_resource_attr (Optional[str]) – See above

  • seed (Optional[int]) – If given, this seed is used for all trial evaluations. Otherwise, seed is sampled at random for each trial. Only relevant for blackboxes with multiple seeds

  • support_checkpointing (bool) – If False, the simulation does not do checkpointing, so resumed trials are started from scratch. Defaults to True

  • dataset (Optional[str]) – Selects different versions of the blackbox (typically, the same ML model has been trained on different datasets)

  • surrogate (Optional[str]) – Optionally, a model that is fitted to predict objectives given any configuration. Examples: “KNeighborsRegressor”, “MLPRegressor”, “XGBRegressor”, which would enable using the corresponding scikit-learn estimator, see also make_surrogate(). The model is fit on top of pipeline that applies basic feature-processing to convert hyperparameter rows in X to vectors. The configuration_space hyperparameter types are used to deduce the types of columns in X (for instance, categorical hyperparameters are one-hot encoded).

  • surrogate_kwargs (Optional[dict]) – Arguments for the scikit-learn estimator, for instance {"n_neighbors": 1} can be used if surrogate="KNeighborsRegressor" is chosen. If blackbox_name is a YAHPO blackbox, then surrogate_kwargs is passed as yahpo_kwargs to load_blackbox(). In this case, surrogate is ignored (YAHPO always uses surrogates).

  • config_space_surrogate (Optional[dict]) – If surrogate is given, this is the configuration space for the surrogate blackbox. If not given, the space of the original blackbox is used. However, its numerical parameters have finite domains (categorical or ordinal), which is usually not what we want for a surrogate.

  • simulatorbackend_kwargs – Additional arguments to parent SimulatorBackend

property blackbox: Blackbox
class syne_tune.blackbox_repository.UserBlackboxBackend(blackbox, elapsed_time_attr, max_resource_attr=None, seed=None, support_checkpointing=True, **simulatorbackend_kwargs)[source]

Bases: _BlackboxSimulatorBackend

Version of _BlackboxSimulatorBackend, where the blackbox is given as explicit Blackbox object. See examples/launch_simulated_benchmark.py for an example on how to use.

Additional arguments on top of parent _BlackboxSimulatorBackend:

Parameters:

blackbox (Blackbox) – Blackbox to be used for simulation

property blackbox: Blackbox

Subpackages

Submodules