syne_tune.backend.simulator_backend.simulator_backend module

class syne_tune.backend.simulator_backend.simulator_backend.SimulatorConfig(delay_on_trial_result=0.05, delay_complete_after_final_report=0.05, delay_complete_after_stop=0.05, delay_start=0.05, delay_stop=0.05)

Bases: object

Configures the simulator:

Parameters:

• delay_on_trial_result (float) – Time from report called on worker to result registered at backend, defaults to DEFAULT_DELAY

• delay_complete_after_final_report (float) – Time from final report called on worker to job completion being registered at backend. Defaults to DEFAULT_DELAY

• delay_complete_after_stop (float) – Time from stop signal received at worker to job completion being registered at backend. Defaults to DEFAULT_DELAY

• delay_start (float) – Time from start command being sent at backend and job starting on the worker (which is free). Defaults to DEFAULT_DELAY

• delay_stop (float) – Time from stop signal being sent at backend to signal received at worker (which is running). Defaults to DEFAULT_DELAY

delay_on_trial_result: float = 0.05

delay_complete_after_final_report: float = 0.05

delay_complete_after_stop: float = 0.05

delay_start: float = 0.05

delay_stop: float = 0.05

class syne_tune.backend.simulator_backend.simulator_backend.SimulatorBackend(entry_point, elapsed_time_attr, simulator_config=None, tuner_sleep_time=5.0, debug_resource_attr=None)

Bases: LocalBackend

This simulator backend drives experiments with tabulated training evaluation functions, which return their computation time rather than spend it. To this end, time (on the tuning instance) is simulated using a time_keeper and an event priority queue in _simulator_state.

Time is advanced both by run() waiting, and by non-negligible computations during the tuning loop (in particular, we take care of scheduler.suggest and scheduler.on_trial_result there).

When the entry_point script is executed, we wait for all results to be returned. In each result, the value for key elapsed_time_attr contains the time since start of the script. These values are used to place worker events on the simulated timeline (represented by simulator_state). NOTE: If a trial is resumed, the elapsed_time value contains the time since start of the last recent resume, NOT the cumulative time used by the trial.

Each method call starts by advancing time by what was spent outside, since the last recent call to the backend. Then, all events in simulator_state are processed whose time is before the current time in time_keeper. The method ends by time_keeper.mark_exit().

Note

In this basic version of the simulator backend, we still call a Python main function as a subprocess, which returns the requested metrics by looking them up or running a surrogate. This is flexible, but has the overhead of loading a table at every call. For fast and convenient simulations, use :BlackboxRepositoryBackend after bringing your tabulated data or surrogate benchmark into the blackbox repository.

Parameters:

• entry_point (str) – Python main file to be tuned (this should return all results directly, and report elapsed time in the elapsed_time_attr field

• elapsed_time_attr (str) – See above

• simulator_config (Optional[SimulatorConfig]) – Parameters for simulator, optional

• tuner_sleep_time (float) – Effective sleep time in run(). This information is needed in SimulatorCallback. Defaults to DEFAULT_SLEEP_TIME

property time_keeper: SimulatedTimeKeeper

start_trial(config, checkpoint_trial_id=None)

Start new trial with new trial ID

Parameters:

• config (Dict) – Configuration for new trial

• checkpoint_trial_id (Optional[int]) – If given, the new trial starts from the checkpoint written by this previous trial

Return type:

Trial

Returns:

New trial, which includes new trial ID

fetch_status_results(trial_ids)

Parameters:

trial_ids (List[int]) – Trials whose information should be fetched.

Return type:

(Dict[int, Tuple[Trial, str]], List[Tuple[int, dict]])

Returns:

A tuple containing 1) a dictionary from trial-id to Trial and status information; 2) a list of (trial-id, results) pairs for each new result emitted since the last call. The list of results is sorted by the worker time-stamp.

busy_trial_ids()

Returns list of ids for currently busy trials

A trial is busy if its status is in_progress or stopping. If the execution setup is able to run n_workers jobs in parallel, then if this method returns a list of size n, the tuner may start n_workers - n new jobs.

Return type:

List[Tuple[int, str]]

Returns:

List of (trial_id, status)