syne_tune.experiments package

class syne_tune.experiments.ExperimentResult(name, results, metadata, tuner, path)

Bases: object

Wraps results dataframe and provides retrieval services.

Parameters:

• name (str) – Name of experiment

• results (DataFrame) – Dataframe containing results of experiment

• metadata (Dict[str, Any]) – Metadata stored along with results

• tuner (Tuner) – Tuner object stored along with results

• path (Path) – local path where the experiment is stored

name: str

results: DataFrame

metadata: Dict[str, Any]

tuner: Tuner

path: Path

creation_date()

Returns:

Timestamp when Tuner was created

plot_hypervolume(metrics_to_plot=None, reference_point=None, figure_path=None, **plt_kwargs)

Plot best hypervolume value as function of wallclock time

Parameters:

• reference_point (Optional[ndarray]) – Reference point for hypervolume calculations. If None, the maximum values of each metric is used.

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• plt_kwargs – Arguments to matplotlib.pyplot.plot()

plot(metric_to_plot=0, figure_path=None, **plt_kwargs)

Plot best metric value as function of wallclock time

Parameters:

• metric_to_plot (Union[str, int]) – Indicates which metric to plot, can be the index or a name of the metric. default to 0 - first metric defined

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• plt_kwargs – Arguments to matplotlib.pyplot.plot()

plot_trials_over_time(metric_to_plot=0, figure_path=None, figsize=None)

Plot trials results over as function of wallclock time

Parameters:

• metric_to_plot (Union[str, int]) – Indicates which metric to plot, can be the index or a name of the metric. default to 0 - first metric defined

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• figsize – width and height of figure

metric_mode()

Return type:

Union[str, List[str]]

metric_names()

Return type:

List[str]

entrypoint_name()

Return type:

str

best_config(metric=0)

Return the best config found for the specified metric :type metric: Union[str, int] :param metric: Indicates which metric to use, can be the index or a name of the metric.

default to 0 - first metric defined in the Scheduler

Return type:

Dict[str, Any]

Returns:

Configuration corresponding to best metric value

syne_tune.experiments.load_experiment(tuner_name, load_tuner=False, local_path=None, experiment_name=None)

Load results from an experiment

Parameters:

• tuner_name (str) – Name of a tuning experiment previously run

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

• local_path (Optional[str]) – Path containing the experiment to load. If not specified, ~/{SYNE_TUNE_FOLDER}/ is used.

• experiment_name (Optional[str]) – If given, this is used as first directory.

Return type:

ExperimentResult

Returns:

Result object

syne_tune.experiments.get_metadata(path_filter=None, root=PosixPath('/home/docs/syne-tune'))

Load meta-data for a number of experiments

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• root (Path) – Root path for experiment results. Default is experiment_path()

Return type:

Dict[str, dict]

Returns:

Dictionary from tuner name to metadata dict

syne_tune.experiments.list_experiments(path_filter=None, experiment_filter=None, root=PosixPath('/home/docs/syne-tune'), load_tuner=False)

List experiments for which results are found

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• experiment_filter (Optional[Callable[[ExperimentResult], bool]]) – Filter on ExperimentResult, optional

• root (Path) – Root path for experiment results. Default is result of experiment_path()

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

Return type:

List[ExperimentResult]

Returns:

List of result objects

syne_tune.experiments.load_experiments_df(path_filter=None, experiment_filter=None, root=PosixPath('/home/docs/syne-tune'), load_tuner=False)

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• experiment_filter (Optional[Callable[[ExperimentResult], bool]]) – Filter on ExperimentResult

• root (Path) – Root path for experiment results. Default is experiment_path()

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

Return type:

DataFrame

Returns:

Dataframe that contains all evaluations reported by tuners according to the filter given. The columns contain trial-id, hyperparameter evaluated, metrics reported via Reporter. These metrics are collected automatically:

• st_worker_time (indicating time spent in the worker when report was seen)

• time (indicating wallclock time measured by the tuner)

• decision decision taken by the scheduler when observing the result

• status status of the trial that was shown to the tuner

• config_{xx} configuration value for the hyperparameter {xx}

• tuner_name named passed when instantiating the Tuner

• entry_point_name, entry_point_path name and path of the entry point that was tuned

syne_tune.experiments.hypervolume_indicator_column_generator(metrics_and_modes, reference_point=None, increment=1)

Returns generator for new dataframe column containing the best hypervolume indicator as function of wall-clock time, based on the metrics in metrics_and_modes (metric names correspond to column names in the dataframe). For a metric with mode == "max", we use its negative.

This mapping is used to create the dataframe_column_generator argument of plot(). Since the current implementation is not incremental and quite slow, if you plot results for single-fidelity HPO methods, it is strongly recommended to also use one_result_per_trial=True:

results = ComparativeResults(...)
dataframe_column_generator = hypervolume_indicator_column_generator(
    metrics_and_modes
)
plot_params = PlotParameters(
    metric="hypervolume_indicator",
    mode="max",
)
results.plot(
    benchmark_name=benchmark_name,
    plot_params=plot_params,
    dataframe_column_generator=dataframe_column_generator,
    one_result_per_trial=True,
)

Parameters:

• metrics_and_modes (List[Tuple[str, str]]) – List of (metric, mode), see above

• reference_point (Optional[ndarray]) – Reference point for hypervolume computation. If not given, a default value is used

• increment (int) – If > 1, the HV indicator is linearly interpolated, this is faster. Defaults to 1 (no interpolation)

Returns:

Dataframe column generator

Submodules

• syne_tune.experiments.experiment_result module
  • ExperimentResult
    • ExperimentResult.name
    • ExperimentResult.results
    • ExperimentResult.metadata
    • ExperimentResult.tuner
    • ExperimentResult.path
    • ExperimentResult.creation_date()
    • ExperimentResult.plot_hypervolume()
    • ExperimentResult.plot()
    • ExperimentResult.plot_trials_over_time()
    • ExperimentResult.metric_mode()
    • ExperimentResult.metric_names()
    • ExperimentResult.entrypoint_name()
    • ExperimentResult.best_config()
  • load_experiment()
  • get_metadata()
  • list_experiments()
  • load_experiments_df()
• syne_tune.experiments.multiobjective module
  • hypervolume_indicator_column_generator()
• syne_tune.experiments.util module
  • filter_final_row_per_trial()