syne_tune.blackbox_repository.conversion_scripts.scripts.lcbench.api module

class syne_tune.blackbox_repository.conversion_scripts.scripts.lcbench.api.Benchmark(data_dir, cache=False, cache_dir='cached/')

Bases: object

API for TabularBench.

query(dataset_name, tag, config_id)

Query a run.

Keyword arguments: dataset_name – str, the name of the dataset in the benchmark tag – str, the tag you want to query config_id – int, an identifier for which run you want to query, if too large will query the last run

query_best(dataset_name, tag, criterion, position=0)

Query the n-th best run. “Best” here means achieving the largest value at any epoch/step,

Keyword arguments: dataset_name – str, the name of the dataset in the benchmark tag – str, the tag you want to query criterion – str, the tag you want to use for the ranking position – int, an identifier for which position in the ranking you want to query

get_queriable_tags(dataset_name=None, config_id=None)

Returns a list of all queriable tags

get_dataset_names()

Returns a list of all availabe dataset names like defined on openml

get_openml_task_ids()

Returns a list of openml task ids

get_number_of_configs(dataset_name)

Returns the number of configurations for a dataset

get_config(dataset_name, config_id)

Returns the configuration of a run specified by dataset name and config id

plot_by_name(dataset_names, x_col, y_col, n_configs=10, show_best=False, xscale='linear', yscale='linear', criterion=None)

Plot multiple datasets and multiple runs.

Keyword arguments: dataset_names – list x_col – str, tag to plot on x-axis y_col – str, tag to plot on y-axis n_configs – int, number of configs to plot for each dataset show_best – bool, weather to show the n_configs best (according to query_best()) xscale – str, set xscale, options as in matplotlib: “linear”, “log”, “symlog”, “logit”, … yscale – str, set yscale, options as in matplotlib: “linear”, “log”, “symlog”, “logit”, … criterion – str, tag used as criterion for query_best()