Running

Train

class farm.train.EarlyStopping(metric='loss', save_dir=None, mode='min', patience=0, min_delta=0.001, min_evals=0)[source]

Bases: object

__init__(metric='loss', save_dir=None, mode='min', patience=0, min_delta=0.001, min_evals=0)[source]

Can be used to control early stopping with a Trainer class. Any object can be used instead which implements the method check_stopping and and provides the attribute save_dir :param save_dir: the directory where to save the final best model, if None, no saving. :param metric: name of dev set metric to monitor (default: loss) to get extracted from the 0th head or a function that extracts a value from the trainer dev evaluation result. NOTE: this is different from the metric to get specified for the processor which defines how to calculate one or more evaluation matric values from prediction/target sets, while this specifies the name of one particular such metric value or a method to calculate that value from the result returned from a processor metric. :param mode: “min” or “max” :param patience: how many evaluations to wait after the best evaluation to stop :param min_delta: minimum difference to a previous best value to count as an improvement. :param min_evals: minimum number of evaluations to wait before using eval value

check_stopping(eval_result)[source]

Provide the evaluation value for the current evaluation. Returns true if stopping should occur. This will save the model, if necessary. :param eval: the current evaluation result :return: a tuple (stopprocessing, savemodel, evalvalue) indicating if processing should be stopped and if the current model should get saved and the evaluation value used.

class farm.train.Trainer(model, optimizer, data_silo, epochs, n_gpu, device, lr_schedule=None, evaluate_every=100, evaluator_dev=None, evaluator_test=None, use_amp=None, grad_acc_steps=1, local_rank=-1, early_stopping=None, log_learning_rate=False, checkpoint_on_sigterm=False, checkpoint_every=None, checkpoint_root_dir=None, checkpoints_to_keep=3, from_epoch=0, from_step=0)[source]

Bases: object

Handles the main model training procedure. This includes performing evaluation on the dev set at regular intervals during training as well as evaluation on the test set at the end of training.

__init__(model, optimizer, data_silo, epochs, n_gpu, device, lr_schedule=None, evaluate_every=100, evaluator_dev=None, evaluator_test=None, use_amp=None, grad_acc_steps=1, local_rank=-1, early_stopping=None, log_learning_rate=False, checkpoint_on_sigterm=False, checkpoint_every=None, checkpoint_root_dir=None, checkpoints_to_keep=3, from_epoch=0, from_step=0)[source]
Parameters
  • optimizer – An optimizer object that determines the learning strategy to be used during training

  • data_silo (DataSilo) – A DataSilo object that will contain the train, dev and test datasets as PyTorch DataLoaders

  • epochs (int) – How many times the training procedure will loop through the train dataset

  • n_gpu (int) – The number of gpus available for training and evaluation.

  • device – The device on which the train, dev and test tensors should be hosted. Choose from “cpu” and “cuda”.

  • lr_schedule – An optional scheduler object that can regulate the learning rate of the optimizer

  • evaluate_every (int) – Perform dev set evaluation after this many steps of training.

  • evaluator_dev (Evaluator, None or False) – Evaluator for dev set. Options: None (Default) => will init a new evaluator, if there’s a dev set in the DataSilo Evaluator Object => use the manually supplied evaluator False => Don’t use any evaluator

  • evaluator_test (Evaluator, None or False) – Evaluator for test set. Options: None (Default) => will init a new evaluator, if there’s a test set in the DataSilo Evaluator Object => use the manually supplied evaluator False => Don’t use any evaluator

  • use_amp (str) – Whether to use automatic mixed precision with Apex. One of the optimization levels must be chosen. “O1” is recommended in almost all cases.

  • grad_acc_steps (int) – TODO

  • local_rank (int) – TODO

  • early_stopping (EarlyStopping) – an initialized EarlyStopping object to control early stopping and saving of best models.

  • log_learning_rate (bool) – Whether to log learning rate to Mlflow

  • checkpoint_on_sigterm (bool) – save a checkpoint for the Trainer when a SIGTERM signal is sent. The checkpoint can be used to resume training. It is useful in frameworks like AWS SageMaker with Spot instances where a SIGTERM notifies to save the training state and subsequently the instance is terminated.

  • checkpoint_every (int) – save a train checkpoint after this many steps of training.

  • checkpoint_root_dir (Path) – the Path of directory where all train checkpoints are saved. For each individual checkpoint, a subdirectory with the name epoch_{epoch_num}_step_{step_num} is created.

  • checkpoints_to_keep (int) – maximum number of train checkpoints to save.

  • from_epoch (int) – the epoch number to start the training from. In the case when training resumes from a saved checkpoint, it is used to fast-forward training to the last epoch in the checkpoint.

  • from_step (int) – the step number to start the training from. In the case when training resumes from a saved checkpoint, it is used to fast-forward training to the last step in the checkpoint.

classmethod create_or_load_checkpoint(data_silo, checkpoint_root_dir, resume_from_checkpoint='latest', **kwargs)[source]

Try loading a saved Trainer checkpoint. If no checkpoint found, it creates a new instance of Trainer.

Parameters
  • data_silo (DataSilo) – A DataSilo object that will contain the train, dev and test datasets as PyTorch DataLoaders

  • checkpoint_root_dir (Path) – Path of the directory where all train checkpoints are saved. Each individual checkpoint is stored in a sub-directory under it.

  • resume_from_checkpoint (str) – the checkpoint name to start training from, e.g., “epoch_1_step_4532”. It defaults to “latest”, using the checkpoint with the highest train steps.

train()[source]

Perform the training procedure.

backward_propagate(loss, step)[source]
adjust_loss(loss)[source]
log_params()[source]

Eval

class farm.eval.Evaluator(data_loader, tasks, device, report=True)[source]

Bases: object

Handles evaluation of a given model over a specified dataset.

__init__(data_loader, tasks, device, report=True)[source]
Parameters
  • data_loader (DataLoader) – The PyTorch DataLoader that will return batches of data from the evaluation dataset

  • label_maps

  • device – The device on which the tensors should be processed. Choose from “cpu” and “cuda”.

  • metrics – The list of metrics which need to be computed, one for each prediction head.

  • metrics – list

  • report (bool) – Whether an eval report should be generated (e.g. classification report per class).

eval(model, return_preds_and_labels=False)[source]

Performs evaluation on a given model.

Parameters
  • model (AdaptiveModel) – The model on which to perform evaluation

  • return_preds_and_labels (bool) – Whether to add preds and labels in the returned dicts of the

Return all_results

A list of dictionaries, one for each prediction head. Each dictionary contains the metrics and reports generated during evaluation.

Rtype all_results

list of dicts

static log_results(results, dataset_name, steps, logging=True, print=True, num_fold=None)[source]

Infer

class farm.infer.Inferencer(model, processor, batch_size=4, gpu=False, name=None, return_class_probs=False)[source]

Bases: object

Loads a saved AdaptiveModel from disk and runs it in inference mode. Can be used for a model with prediction head (down-stream predictions) and without (using LM as embedder).

Example usage:

# down-stream inference
basic_texts = [
    {"text": "Schartau sagte dem Tagesspiegel, dass Fischer ein Idiot sei"},
    {"text": "Martin Müller spielt Handball in Berlin"},
]
model = Inferencer.load(your_model_dir)
model.inference_from_dicts(dicts=basic_texts)
# LM embeddings
model.extract_vectors(dicts=basic_texts)
__init__(model, processor, batch_size=4, gpu=False, name=None, return_class_probs=False)[source]

Initializes Inferencer from an AdaptiveModel and a Processor instance.

Parameters
  • model (AdaptiveModel) – AdaptiveModel to run in inference mode

  • processor (Processor) – A dataset specific Processor object which will turn input (file or dict) into a Pytorch Dataset.

  • batch_size (int) – Number of samples computed once per batch

  • gpu (bool) – If GPU shall be used

  • name (string) – Name for the current Inferencer model, displayed in the REST API

  • return_class_probs (bool) – either return probability distribution over all labels or the prob of the associated label

Returns

An instance of the Inferencer.

classmethod load(model_name_or_path, batch_size=4, gpu=False, task_type=None, return_class_probs=False, strict=True, max_seq_len=256)[source]

Load an Inferencer incl. all relevant components (model, tokenizer, processor …) either by

  1. specifying a public name from transformers’ model hub (https://huggingface.co/models)

  2. or pointing to a local directory it is saved in.

Parameters
  • model_name_or_path (str) – Local directory or public name of the model to load.

  • batch_size (int) – Number of samples computed once per batch

  • gpu (bool) – If GPU shall be used

  • task_type – Type of task the model should be used for. Currently supporting: “embeddings”, “question_answering”, “text_classification”. More coming soon…

  • task_type – str

  • strict (bool) – whether to strictly enforce that the keys loaded from saved model match the ones in the PredictionHead (see torch.nn.module.load_state_dict()). Set to False for backwards compatibility with PHs saved with older version of FARM.

Returns

An instance of the Inferencer.

save(path)[source]
inference_from_file(file, max_processes=128)[source]

Run down-stream inference on samples created from an input file. The file should be in the same format as the ones used during training (e.g. squad style for QA, tsv for doc classification …) as the same processor will be used for conversion .

Parameters
  • file (str) – path of the input file for Inference

  • max_processes (int) – the maximum size of multiprocessing.Pool. Set to value of 1 to disable multiprocessing.

inference_from_dicts(dicts, rest_api_schema=False, max_processes=128)[source]

Runs down-stream inference on samples created from input dictionaries. The format of the input dicts depends on the task:

QA: [{“qas”: [“What is X?”], “context”: “Some context containing the answer”}] Classification / NER: [{“text”: “Some input text”}]

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • rest_api_schema (bool) – Whether input dicts use the format that complies with the FARM REST API. Currently only used for QA to switch from squad to a more useful format in production. While input is almost the same, output contains additional meta data(offset, context..)

  • max_processes – the maximum size of multiprocessing.Pool. Set to value of 1 to disable multiprocessing. For very small number of dicts, time incurred in spawning processes could outweigh performance boost, eg, in the case of HTTP APIs for Inference. For such cases, multiprocessing can be disabled using this param.

Returns

dict of predictions

extract_vectors(dicts, extraction_strategy='cls_token', extraction_layer=-1)[source]

Converts a text into vector(s) using the language model only (no prediction head involved).

Example:

basic_texts = [{“text”: “Some text we want to embed”}, {“text”: “And a second one”}] result = inferencer.extract_vectors(dicts=basic_texts)

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • extraction_strategy (str) – Strategy to extract vectors. Choices: ‘cls_token’ (sentence vector), ‘reduce_mean’ (sentence vector), reduce_max (sentence vector), ‘per_token’ (individual token vectors)

  • extraction_layer – number of layer from which the embeddings shall be extracted. Default: -1 (very last layer).

Type

int

Returns

dict of predictions

class farm.infer.FasttextInferencer(model, name=None)[source]

Bases: object

__init__(model, name=None)[source]

Initialize self. See help(type(self)) for accurate signature.

classmethod load(load_dir, batch_size=4, gpu=False)[source]
extract_vectors(dicts, extraction_strategy='reduce_mean')[source]

Converts a text into vector(s) using the language model only (no prediction head involved).

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • extraction_strategy (str) – Strategy to extract vectors. Choices: ‘reduce_mean’ (mean sentence vector), ‘reduce_max’ (max per embedding dim), ‘CLS’

Returns

dict of predictions

Experiment

farm.experiment.load_experiments(file)[source]
farm.experiment.run_experiment(args)[source]
farm.experiment.get_adaptive_model(lm_output_type, prediction_heads, layer_dims, model, device, embeds_dropout_prob, class_weights=None)[source]
farm.experiment.validate_args(args)[source]
farm.experiment.save_model()[source]
farm.experiment.load_model()[source]

Metrics

farm.metrics.register_metrics(name, implementation)[source]
farm.metrics.simple_accuracy(preds, labels)[source]
farm.metrics.acc_and_f1(preds, labels)[source]
farm.metrics.f1_macro(preds, labels)[source]
farm.metrics.pearson_and_spearman(preds, labels)[source]
farm.metrics.compute_metrics(metric, preds, labels)[source]
farm.metrics.squad_EM(preds, labels)[source]
farm.metrics.squad_f1(preds, labels)[source]
farm.metrics.squad_f1_single(pred, label)[source]
farm.metrics.squad(preds, labels)[source]

File utils

Utilities for working with the local dataset cache. This file is adapted from the AllenNLP library at https://github.com/allenai/allennlp Copyright by the AllenNLP authors.

farm.file_utils.url_to_filename(url, etag=None)[source]

Convert url into a hashed filename in a repeatable way. If etag is specified, append its hash to the url’s, delimited by a period.

farm.file_utils.filename_to_url(filename, cache_dir=None)[source]

Return the url and etag (which may be None) stored for filename. Raise EnvironmentError if filename or its stored metadata do not exist.

farm.file_utils.cached_path(url_or_filename, cache_dir=None)[source]

Given something that might be a URL (or might be a local path), determine which. If it’s a URL, download the file and cache it, and return the path to the cached file. If it’s already a local path, make sure the file exists and then return the path.

farm.file_utils.split_s3_path(url)[source]

Split a full s3 path into the bucket name and path.

farm.file_utils.s3_request(func)[source]

Wrapper function for s3 requests in order to create more helpful error messages.

farm.file_utils.s3_etag(url)[source]

Check ETag on S3 object.

farm.file_utils.s3_get(url, temp_file)[source]

Pull a file directly from S3.

farm.file_utils.http_get(url, temp_file, proxies=None)[source]
farm.file_utils.get_from_cache(url, cache_dir=None)[source]

Given a URL, look for the corresponding dataset in the local cache. If it’s not there, download it. Then return the path to the cached file.

farm.file_utils.read_set_from_file(filename)[source]

Extract a de-duped collection (set) of text from a file. Expected file format is one item per line.

farm.file_utils.get_file_extension(path, dot=True, lower=True)[source]
farm.file_utils.read_config(path, flattend=False)[source]
farm.file_utils.unnestConfig(config, flattened=False)[source]

This function creates a list of config files for evaluating parameters with different values. If a config parameter is of type list this list is iterated over and a config object without lists is returned. Can handle lists inside any number of parameters.

Can handle shallow or nested (one level) configs