syne_tune.experiments.experiment_result module

class syne_tune.experiments.experiment_result.ExperimentResult(name, results, metadata, tuner, path)

Bases: object

Wraps results dataframe and provides retrieval services.

Parameters:

• name (str) – Name of experiment

• results (DataFrame) – Dataframe containing results of experiment

• metadata (Dict[str, Any]) – Metadata stored along with results

• tuner (Tuner) – Tuner object stored along with results

• path (Path) – local path where the experiment is stored

name: str

results: DataFrame

metadata: Dict[str, Any]

tuner: Tuner

path: Path

creation_date()

Returns:

Timestamp when Tuner was created

plot_hypervolume(metrics_to_plot=None, reference_point=None, figure_path=None, **plt_kwargs)

Plot best hypervolume value as function of wallclock time

Parameters:

• reference_point (Optional[ndarray]) – Reference point for hypervolume calculations. If None, the maximum values of each metric is used.

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• plt_kwargs – Arguments to matplotlib.pyplot.plot()

plot(metric_to_plot=0, figure_path=None, **plt_kwargs)

Plot best metric value as function of wallclock time

Parameters:

• metric_to_plot (Union[str, int]) – Indicates which metric to plot, can be the index or a name of the metric. default to 0 - first metric defined

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• plt_kwargs – Arguments to matplotlib.pyplot.plot()

plot_trials_over_time(metric_to_plot=0, figure_path=None, figsize=None)

Plot trials results over as function of wallclock time

Parameters:

• metric_to_plot (Union[str, int]) – Indicates which metric to plot, can be the index or a name of the metric. default to 0 - first metric defined

• figure_path (Optional[str]) – If specified, defines the path where the figure will be saved. If None, the figure is shown

• figsize – width and height of figure

metric_mode()

Return type:

Union[str, List[str]]

metric_names()

Return type:

List[str]

entrypoint_name()

Return type:

str

best_config(metric=0)

Return the best config found for the specified metric :type metric: Union[str, int] :param metric: Indicates which metric to use, can be the index or a name of the metric.

default to 0 - first metric defined in the Scheduler

Return type:

Dict[str, Any]

Returns:

Configuration corresponding to best metric value

syne_tune.experiments.experiment_result.download_single_experiment(tuner_name, s3_bucket=None, experiment_name=None)

Downloads results from S3 of a tuning experiment

Parameters:

• tuner_name (str) – Name of tuner to be retrieved.

• s3_bucket (Optional[str]) – If not given, the default bucket for the SageMaker session is used

• experiment_name (Optional[str]) – If given, this is used as first directory.

syne_tune.experiments.experiment_result.load_experiment(tuner_name, download_if_not_found=True, load_tuner=False, local_path=None, experiment_name=None)

Load results from an experiment

Parameters:

• tuner_name (str) – Name of a tuning experiment previously run

• download_if_not_found (bool) – If True, fetch results from S3 if not found locally

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

• local_path (Optional[str]) – Path containing the experiment to load. If not specified, ~/{SYNE_TUNE_FOLDER}/ is used.

• experiment_name (Optional[str]) – If given, this is used as first directory.

Return type:

ExperimentResult

Returns:

Result object

syne_tune.experiments.experiment_result.get_metadata(path_filter=None, root=PosixPath('/home/docs/syne-tune'))

Load meta-data for a number of experiments

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• root (Path) – Root path for experiment results. Default is experiment_path()

Return type:

Dict[str, dict]

Returns:

Dictionary from tuner name to metadata dict

syne_tune.experiments.experiment_result.list_experiments(path_filter=None, experiment_filter=None, root=PosixPath('/home/docs/syne-tune'), load_tuner=False)

List experiments for which results are found

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• experiment_filter (Optional[Callable[[ExperimentResult], bool]]) – Filter on ExperimentResult, optional

• root (Path) – Root path for experiment results. Default is result of experiment_path()

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

Return type:

List[ExperimentResult]

Returns:

List of result objects

syne_tune.experiments.experiment_result.load_experiments_df(path_filter=None, experiment_filter=None, root=PosixPath('/home/docs/syne-tune'), load_tuner=False)

Parameters:

• path_filter (Optional[Callable[[str], bool]]) – If passed then only experiments whose path matching the filter are kept. This allows rapid filtering in the presence of many experiments.

• experiment_filter (Optional[Callable[[ExperimentResult], bool]]) – Filter on ExperimentResult

• root (Path) – Root path for experiment results. Default is experiment_path()

• load_tuner (bool) – Whether to load the tuner in addition to metadata and results

Return type:

DataFrame

Returns:

Dataframe that contains all evaluations reported by tuners according to the filter given. The columns contain trial-id, hyperparameter evaluated, metrics reported via Reporter. These metrics are collected automatically:

• st_worker_time (indicating time spent in the worker when report was seen)

• time (indicating wallclock time measured by the tuner)

• decision decision taken by the scheduler when observing the result

• status status of the trial that was shown to the tuner

• config_{xx} configuration value for the hyperparameter {xx}

• tuner_name named passed when instantiating the Tuner

• entry_point_name, entry_point_path name and path of the entry point that was tuned