syne_tune.blackbox_repository.repository module

syne_tune.blackbox_repository.repository.blackbox_list()[source]
Return type:

List[str]

Returns:

list of blackboxes available

syne_tune.blackbox_repository.repository.load_blackbox(name, custom_repo_id=None, yahpo_kwargs=None, local_files_only=False, force_download=False, **snapshot_download_kwargs)[source]
Parameters:

  • name (str) –

    name of a blackbox present in the repository, see blackbox_list() to get list of available blackboxes. Syne Tune currently provides the following blackboxes evaluations:

    • ”nasbench201”: 15625 multi-fidelity configurations of computer vision architectures evaluated on 3 datasets. NAS-Bench-201: Extending the scope of reproducible neural architecture search. Dong, X. and Yang, Y. 2020.

    • ”fcnet”: 62208 multi-fidelity configurations of MLP evaluated on 4 datasets. Tabular benchmarks for joint architecture and hyperparameter optimization. Klein, A. and Hutter, F. 2019.

    • ”lcbench”: 2000 multi-fidelity Pytorch model configurations evaluated on many datasets. Reference: Auto-PyTorch: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL. Lucas Zimmer, Marius Lindauer, Frank Hutter. 2020.

    • ”icml-deepar”: 2420 single-fidelity configurations of DeepAR forecasting algorithm evaluated on 10 datasets. A quantile-based approach for hyperparameter transfer learning. Salinas, D., Shen, H., and Perrone, V. 2021.

    • ”icml-xgboost”: 5O00 single-fidelity configurations of XGBoost evaluated on 9 datasets. A quantile-based approach for hyperparameter transfer learning. Salinas, D., Shen, H., and Perrone, V. 2021.

    • ”yahpo-*”: Number of different benchmarks from YAHPO Gym. Note that these blackboxes come with surrogates already, so no need to wrap them into SurrogateBlackbox

    • ”hpob_*”: ca. 6.34 million evaluations distributed on 16 search spaces and 101 datasets. HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML. S. Arango, H. Jomaa, M. Wistuba, J. Grabocka, 2021.

    • ”tabrepo-*”: TabRepo contains the predictions and metrics of 1530 models evaluated on 211 classification and regression datasets. TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications. D. Salinas, N. Erickson, 2024.

  • custom_repo_id (Optional[str]) – custom hugging face repoid to use, default to Syne Tune hub

  • yahpo_kwargs (Optional[dict]) – For a YAHPO blackbox (name == "yahpo-*"), these are additional arguments to instantiate_yahpo

  • local_files_only (bool) – whether to use local files with no internet check on the Hub

  • force_download (bool) – forces files to be downloaded

  • snapshot_download_kwargs – keyword arguments for snapshot_download (other than local_files_only and force_download)

Return type:

Union[Dict[str, Blackbox], Blackbox]

Returns:

blackbox with the given name, download it if not present.

syne_tune.blackbox_repository.repository.check_blackbox_local_files(tgt_folder)[source]

checks whether the file of the blackbox name are present in repository_path

Return type:

bool