syne_tune.experiments.visualization.results_utils module

syne_tune.experiments.visualization.results_utils.create_index_for_result_files(experiment_names, metadata_to_setup, metadata_to_subplot=None, metadata_keys=None, metadata_subplot_level=False, benchmark_key='benchmark', with_subdirs='*', datetime_bounds=None, seed_key=None)[source]

Helper function for ComparativeResults.

Runs over all result directories for experiments of a comparative study. For each experiment, we read the metadata file, extract the benchmark name (key benchmark_key), and use metadata_to_setup, metadata_to_subplot to map the metadata to setup name and subplot index. If any of the two return None, the result is not used. Otherwise, we enter (result_path, setup_name, subplot_no) into the list for benchmark name. Here, result_path is the result path for the experiment, without the experiment_path() prefix. The index returned is the dictionary from benchmark names to these list. It allows loading results specifically for each benchmark, and we do not have to load and parse the metadata files again.

If benchmark_key is None, the returned index is a dictionary with a single element only, and the metadata files need not contain an entry for benchmark name.

Result files have the path f"{experiment_path()}{ename}/{patt}/{ename}-*/", where path is from with_subdirs, and ename from experiment_names. The default is with_subdirs="*". If with_subdirs is None, result files have the path f"{experiment_path()}{ename}-*/". This is an older convention, which makes it harder to sync files from S3, it is not recommended.

If metadata_keys is given, it contains a list of keys into the metadata. In this case, a nested dictionary metadata_values is returned, where metadata_values[benchmark_name][key][setup_name] contains a list of metadata values for this benchmark, key in metadata_keys, and setup name. In this case, if metadata_subplot_level is True and metadata_to_subplot is given, metadata_values has the structure metadata_values[benchmark_name][key][setup_name][subplot_no]. This should be set if different subplots share the same setup names.

If datetime_bounds is given, it contains a tuple of strings (lower_time, upper_time), or a dictionary mapping experiment names (from experiment_names) to such tuples. Both strings are time-stamps in the format ST_DATETIME_FORMAT (example: “2023-03-19-22-01-57”), and each can be None as well. This serves to filter out any result whose time-stamp does not fall within the interval (both sides are inclusive), where None means the interval is open on that side. This feature is useful to filter out results of erroneous attempts.

If seed_key is given, the returned index is a dictionary with keys (benchmark_name, seed), where seed is the value corresponding to seed_key in the metadata dict. This mode is needed for plots focusing on a single experiment.

Parameters:

  • experiment_names (Tuple[str, ...]) – Tuple of experiment names (prefixes, without the timestamps)

  • metadata_to_setup (Union[Callable[[Dict[str, Any]], Optional[str]], Dict[str, Callable[[Dict[str, Any]], Optional[str]]]]) – See above

  • metadata_to_subplot (Optional[Callable[[Dict[str, Any]], Optional[int]]]) – See above. Optional

  • metadata_keys (Optional[List[str]]) – See above. Optional

  • metadata_subplot_level (bool) – See above. Defaults to False

  • benchmark_key (Optional[str]) – Key for benchmark in metadata files. Defaults to “benchmark”

  • with_subdirs (Union[str, List[str], None]) – See above. Defaults to “*”

  • datetime_bounds (Union[Tuple[Optional[str], Optional[str]], Dict[str, Tuple[Optional[str], Optional[str]]], None]) – See above

  • seed_key (Optional[str]) – See above

Return type:

Union[Dict[str, Any], Dict[Tuple[str, int], Any]]

Returns:

Dictionary; entry “index” for index (see above); entry “setup_names” for setup names encountered; entry “metadata_values” see metadata_keys

syne_tune.experiments.visualization.results_utils.load_results_dataframe_per_benchmark(experiment_list)[source]

Helper function for ComparativeResults.

Loads time-stamped results for all experiments in experiments_list and returns them in a single dataframe with additional columns “setup_name”, “suplot_no”, “tuner_name”, whose values are constant across data for one experiment, allowing for later grouping.

Parameters:

experiment_list (List[Tuple[str, str, int]]) – Information about experiments, see create_index_for_result_files()

Return type:

Optional[DataFrame]

Returns:

Dataframe with all results combined

syne_tune.experiments.visualization.results_utils.download_result_files_from_s3(experiment_names, s3_bucket=None)[source]

Downloads result files from S3. This works only if the result objects on S3 have prefixes f"{s3_experiment_path(s3_bucket)}{ename}/", where ename is in experiment_names. Only files with names ST_METADATA_FILENAME and ST_RESULTS_DATAFRAME_FILENAME are downloaded.

Parameters:

  • experiment_names (Tuple[str, ...]) – Tuple of experiment names (prefixes, without the timestamps)

  • s3_bucket (Optional[str]) – If not given, the default bucket for the SageMaker session is used