syne_tune.experiments.visualization.plot_per_trial module

class syne_tune.experiments.visualization.plot_per_trial.MultiFidelityParameters(rung_levels, multifidelity_setups)[source]

Bases: object

Parameters configuring the multi-fidelity version of TrialsOfExperimentResults.

multifidelity_setups contains names of setups which are multi-fidelity, the remaining ones are single-fidelity. It can also be a dictionary, mapping a multi-fidelity setup name to True if this is a pause-and-resume method (these are visualized differently), False otherwise (early stopping method).

Parameters:

  • rung_levels (List[int]) – See above. Positive integers, increasing

  • multifidelity_setups (Union[List[str], Dict[str, bool]]) – See above

rung_levels: List[int]
multifidelity_setups: Union[List[str], Dict[str, bool]]
check_params(setups)[source]
class syne_tune.experiments.visualization.plot_per_trial.TrialsOfExperimentResults(experiment_names, setups, metadata_to_setup, plot_params=None, multi_fidelity_params=None, benchmark_key='benchmark', seed_key='seed', with_subdirs='*', datetime_bounds=None, download_from_s3=False, s3_bucket=None)[source]

Bases: object

This class loads, processes, and plots metric results for single experiments, where the curves for different trials have different colours.

Compared to ComparativeResults, each subfigure uses data from a single experiment (one benchmark, one seed, one setup). Both benchmark and seed need to be chosen in plot(). If there are different setups, they give rise to subfigures.

If plot_params.subplots is not given, the arrangement is one row with columns corresponding to setups, and setup names as titles. Specify plot_params.subplots in order to change this arrangement (e.g., to have more than one row). Setups can be selected by using plot_params.subplots.subplot_indices. Also, if plot_params.subplots.titles is not given, we use setup names, and each subplot gets its own title (plot_params.subplots.title_each_figure is ignored).

For plot_params, we use the same PlotParameters as in ComparativeResults, but some fields are not used here (title, aggregate_mode, show_one_trial, subplots.legend_no, subplots.xlims).

Parameters:

  • experiment_names (Tuple[str, ...]) – Tuple of experiment names (prefixes, without the timestamps)

  • setups (Iterable[str]) – Possible values of setup names

  • metadata_to_setup (Union[Callable[[Dict[str, Any]], Optional[str]], Dict[str, Callable[[Dict[str, Any]], Optional[str]]]]) – See above

  • plot_params (Optional[PlotParameters]) – Parameters controlling the plot. Can be overwritten in plot(). See PlotParameters

  • multi_fidelity_params (Optional[MultiFidelityParameters]) – If given, we use a special variant tailored to multi-fidelity methods (see plot()).

  • benchmark_key (Optional[str]) – Key for benchmark in metadata files. Defaults to “benchmark”. If this is None, there is only a single benchmark, and all results are merged together

  • seed_key (str) – Key for seed in metadata files. Defaults to “seed”.

  • with_subdirs (Union[str, List[str], None]) – See above. Defaults to “*”

  • datetime_bounds (Union[Tuple[Optional[str], Optional[str]], Dict[str, Tuple[Optional[str], Optional[str]]], None]) – See above

  • download_from_s3 (bool) – Should result files be downloaded from S3? This is supported only if with_subdirs

  • s3_bucket (Optional[str]) – Only if download_from_s3 == True. If not given, the default bucket for the SageMaker session is used

plot(benchmark_name=None, seed=0, plot_params=None, file_name=None)[source]

Creates a plot, whose subfigures should metric data from single experiments. In general:

  • Each trial has its own color, which is cycled through periodically. The cycling depends on the largest rung level for the trial. This is to avoid neighboring curves to have the same color

For single-fidelity methods (default, multi_fidelity_params not given):

  • The learning curve for a trial ends with ‘o’. If it reports only once at the end, this is all that is shown for the trial

For multi-fidelity methods:

  • Learning curves are plotted in contiguous chunks of execution. For pause and resume setups (those in ``multi_fidelity_params.pause_resume_setups), they are interrupted. Each chunk starts at the epoch after resume and ends at the epoch where the trial is paused

  • Values at rung levels are marked as ‘o’. If this is the furthest the trial got to, the marker is ‘D’ (diamond)

Results for different setups are plotted as subfigures, either using the setup in plot_params.subplots, or as columns of a single row.

Parameters:

  • benchmark_name (Optional[str]) – Name of benchmark for which to plot results. Not needed if there is only one benchmark

  • seed (int) – Seed number. Defaults to 0

  • plot_params (Optional[PlotParameters]) – Parameters controlling the plot. Values provided here overwrite values provided at construction.

  • file_name (Optional[str]) – If given, the figure is stored in a file of this name