benchmarking.utils package
- benchmarking.utils.get_cost_model_for_batch_size(params, batch_size_key, batch_size_range)[source]
Returns cost model depending on the batch size only.
- Parameters:
params (
Dict
[str
,Any
]) – Command line argumentsbatch_size_key (
str
) – Name of batch size entry in configbatch_size_range (
Tuple
[int
,int
]) – (lower, upper) for batch size, both sides are inclusive
- Returns:
Cost model (or None if dependencies cannot be imported)
- class benchmarking.utils.StoreSearcherStatesCallback[source]
Bases:
TunerCallback
Stores list of searcher states alongside a tuning run. The list is extended by a new state whenever the
TuningJobState
has changed compared to the last recently added one.This callback is useful to create meaningful unit tests, by sampling a given searcher alongside a realistic experiment.
Works only for
ModelBasedSearcher
searchers. For other searchers, nothing is stored.- on_trial_result(trial, status, result, decision)[source]
Called when a new result (reported by a trial) is observed
The arguments here are inputs or outputs of
scheduler.on_trial_result
(called just before).- Parameters:
trial (
Trial
) – Trial whose report has been receivedstatus (
str
) – Status of trial beforescheduler.on_trial_result
has been calledresult (
Dict
) – Result dict receiveddecision (
str
) – Decision returned byscheduler.on_trial_result
- property states