benchmarking.benchmark_definitions.finetune_transformer_glue module

benchmarking.benchmark_definitions.finetune_transformer_glue.finetune_transformer_glue_benchmark(sagemaker_backend=False, choose_model=False, dataset='rte', model_type='bert-base-cased', num_train_epochs=3, train_valid_fraction=0.7, random_seed=31415927, **kwargs)[source]

This benchmark consists of fine-tuning a Hugging Face transformer model, selected from the zoo, on one of the GLUE benchmarks:

Wang etal.
GLUE: A Multi-task Benchmark and Analysis Platform for Natural
Language Understanding
ICLR 2019
Parameters:

  • sagemaker_backend (bool) – Use SageMaker backend? This affects the choice of instance type. Defaults to False

  • choose_model (bool) – Should tuning involve selecting the best pre-trained model from PRETRAINED_MODELS? If so, the configuration space is extended by another choice variable. Defaults to False

  • dataset (str) – Name of GLUE task, from TASK2METRICSMODE. Defaults to “rte”

  • model_type (str) – Pre-trained model to be used. If choose_model is set, this is the model used in the first evaluation. Defaults to “bert-base-cased”

  • num_train_epochs (int) – Maximum number of epochs for fine-tuning. Defaults to 3

  • train_valid_fraction (float) – The original training set is split into training and validation part, this is the fraction of the training part

  • random_seed (int) – Random seed for training script

  • kwargs – Overwrites default params in RealBenchmarkDefinition object returned

Return type:

RealBenchmarkDefinition

benchmarking.benchmark_definitions.finetune_transformer_glue.finetune_transformer_glue_all_benchmarks(sagemaker_backend=False, model_type='bert-base-cased', num_train_epochs=3, train_valid_fraction=0.7, random_seed=31415927, **kwargs)[source]
Return type:

Dict[str, RealBenchmarkDefinition]