syne_tune.blackbox_repository.blackbox module

class syne_tune.blackbox_repository.blackbox.Blackbox(configuration_space, fidelity_space=None, objectives_names=None)[source]

Bases: object

Interface designed to be compatible with

Parameters:

  • configuration_space (Dict[str, Any]) – Configuration space of blackbox.

  • fidelity_space (Optional[dict]) – Fidelity space for blackbox, optional.

  • objectives_names (Optional[List[str]]) – Names of the metrics, by default consider all metrics prefixed by "metric_" to be metrics

objective_function(configuration, fidelity=None, seed=None)[source]

Returns an evaluation of the blackbox.

First perform data check and then call _objective_function() that should be overriden in the child class.

Parameters:

  • configuration (Dict[str, Any]) – configuration to be evaluated, should belong to configuration_space

  • fidelity (Union[dict, Number, None]) – not passing a fidelity is possible if either the blackbox does not have a fidelity space or if it has a single fidelity in its fidelity space. In the latter case, all fidelities are returned in form of a tensor with shape (num_fidelities, num_objectives).

  • seed (Optional[int]) – Only used if the blackbox defines multiple seeds

Return type:

Union[Dict[str, float], ndarray]

Returns:

dictionary of objectives evaluated or tensor with shape (num_fidelities, num_objectives) if no fidelity was given.

hyperparameter_objectives_values(predict_curves=False)[source]

If predict_curves is False, the shape of X is (num_evals * num_seeds * num_fidelities, num_hps + 1), the shape of y is (num_evals * num_seeds * num_fidelities, num_objectives). This can be reshaped to (num_fidelities, num_seeds, num_evals, *). The final column of X is the fidelity value (only a single fidelity attribute is supported).

If predict_curves is True, the shape of X is (num_evals * num_seeds, num_hps), the shape of y is (num_evals * num_seeds, num_fidelities * num_objectives). The latter can be reshaped to (num_seeds, num_evals, num_fidelities, num_objectives).

Return type:

Tuple[DataFrame, DataFrame]

Returns:

a tuple of two dataframes (X, y), where X contains hyperparameters values and y contains objective values, this is used when fitting a surrogate model.

property fidelity_values: array | None
Returns:

Fidelity values; or None if the blackbox has none

fidelity_name()[source]

Can only be used for blackboxes with a single fidelity attribute.

Return type:

str

Returns:

Name of fidelity attribute (must be single one)

configuration_space_with_max_resource_attr(max_resource_attr)[source]

It is best practice to have one attribute in the configuration space to represent the maximum fidelity value used for evaluation (e.g., the maximum number of epochs).

Parameters:

max_resource_attr (str) – Name of new attribute for maximum resource

Return type:

Dict[str, Any]

Returns:

Configuration space augmented by the new attribute

syne_tune.blackbox_repository.blackbox.from_function(configuration_space, eval_fun, fidelity_space=None, objectives_names=None)[source]

Helper to create a blackbox from a function, useful for test or to wrap-up real blackbox functions.

Parameters:

  • configuration_space (Dict[str, Any]) – Configuration space for blackbox

  • eval_fun (Callable) – Function that returns dictionary of objectives given configuration and fidelity

  • fidelity_space (Optional[dict]) – Fidelity space for blackbox

  • objectives_names (Optional[List[str]]) – Objectives returned by blackbox

Return type:

Blackbox

Returns:

Resulting blackbox wrapping eval_fun