Running

Train

class farm.train.WrappedDataParallel(module, device_ids=None, output_device=None, dim=0)[source]

Bases: torch.nn.parallel.data_parallel.DataParallel

A way of adapting attributes of underlying class to parallel mode. See: https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html#dataparallel

Gets into recursion errors. Workaround see: https://discuss.pytorch.org/t/access-att-of-model-wrapped-within-torch-nn-dataparallel-maximum-recursion-depth-exceeded/46975

class farm.train.Trainer(optimizer, data_silo, epochs, n_gpu, device, warmup_linear=0.1, evaluate_every=100, evaluator_dev=None, evaluator_test=None, fp16=False, grad_acc_steps=1, local_rank=-1)[source]

Bases: object

Handles the main model training procedure. This includes performing evaluation on the dev set at regular intervals during training as well as evaluation on the test set at the end of training.

__init__(optimizer, data_silo, epochs, n_gpu, device, warmup_linear=0.1, evaluate_every=100, evaluator_dev=None, evaluator_test=None, fp16=False, grad_acc_steps=1, local_rank=-1)[source]
Parameters
  • optimizer – An optimizer object that determines the learning strategy to be used during training

  • data_silo (DataSilo) – A DataSilo object that will contain the train, dev and test datasets as PyTorch DataLoaders

  • epochs (int) – How many times the training procedure will loop through the train dataset

  • n_gpu (int) – The number of gpus available for training and evaluation.

  • device – The device on which the train, dev and test tensors should be hosted. Choose from “cpu” and “cuda”.

  • warmup_linear – TODO

  • evaluate_every (int) – Perform dev set evaluation after this many steps of training.

  • evaluator_dev (Evaluator) – The dev set Evaluator object.

  • evaluator_test (Evaluator) – The test set Evaluator object.

  • fp16 (bool) – Whether to use floating point 16 mode.

  • grad_acc_steps – TODO

train(model)[source]

Perform the training procedure.

backward_propagate(loss, step)[source]
adjust_loss(loss)[source]
log_params()[source]

Eval

class farm.eval.Evaluator(data_loader, tasks, device, classification_report=True)[source]

Bases: object

Handles evaluation of a given model over a specified dataset.

__init__(data_loader, tasks, device, classification_report=True)[source]
Parameters
  • data_loader (DataLoader) – The PyTorch DataLoader that will return batches of data from the evaluation dataset

  • label_maps

  • device – The device on which the tensors should be processed. Choose from “cpu” and “cuda”.

  • metrics – The list of metrics which need to be computed, one for each prediction head.

  • metrics – list

  • classification_report (bool) – Whether a report on the classification performance should be generated.

eval(model)[source]

Performs evaluation on a given model.

Parameters

model (AdaptiveModel) – The model on which to perform evaluation

Return all_results

A list of dictionaries, one for each prediction head. Each dictionary contains the metrics and reports generated during evaluation.

Rtype all_results

list of dicts

static log_results(results, dataset_name, steps, logging=True, print=True)[source]

Infer

class farm.infer.Inferencer(model, processor, batch_size=4, gpu=False, name=None, return_class_probs=False)[source]

Bases: object

Loads a saved AdaptiveModel from disk and runs it in inference mode. Can be used for a model with prediction head (down-stream predictions) and without (using LM as embedder).

Example usage:

# down-stream inference
basic_texts = [
    {"text": "Schartau sagte dem Tagesspiegel, dass Fischer ein Idiot sei"},
    {"text": "Martin Müller spielt Handball in Berlin"},
]
model = Inferencer.load(your_model_dir)
model.inference_from_dicts(dicts=basic_texts)
# LM embeddings
model.extract_vectors(dicts=basic_texts)
__init__(model, processor, batch_size=4, gpu=False, name=None, return_class_probs=False)[source]

Initializes Inferencer from an AdaptiveModel and a Processor instance.

Parameters
  • model (AdaptiveModel) – AdaptiveModel to run in inference mode

  • processor (Processor) – A dataset specific Processor object which will turn input (file or dict) into a Pytorch Dataset.

  • batch_size (int) – Number of samples computed once per batch

  • gpu (bool) – If GPU shall be used

  • name (string) – Name for the current Inferencer model, displayed in the REST API

  • return_class_probs (bool) – either return probability distribution over all labels or the prob of the associated label

Returns

An instance of the Inferencer.

classmethod load(load_dir, batch_size=4, gpu=False, embedder_only=False, return_class_probs=False, strict=True)[source]

Initializes Inferencer from directory with saved model.

Parameters
  • load_dir (str) – Directory where the saved model is located.

  • batch_size (int) – Number of samples computed once per batch

  • gpu (bool) – If GPU shall be used

  • embedder_only (bool) – If true, a faster processor (InferenceProcessor) is loaded. This should only be used for extracting embeddings (no downstream predictions).

  • strict (bool) – whether to strictly enforce that the keys loaded from saved model match the ones in the PredictionHead (see torch.nn.module.load_state_dict()). Set to False for backwards compatibility with PHs saved with older version of FARM.

Returns

An instance of the Inferencer.

inference_from_file(file, use_multiprocessing=True)[source]
inference_from_dicts(dicts, rest_api_schema=False, use_multiprocessing=True)[source]

Runs down-stream inference using the prediction head.

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • rest_api_schema (bool) – whether conform to the schema used for dicts in the HTTP API for Inference.

  • use_multiprocessing – time incurred in spawning processes could outweigh performance boost for very small number of dicts, eg, HTTP APIs for inference. This flags allows to disable multiprocessing for such cases.

Returns

dict of predictions

extract_vectors(dicts, extraction_strategy='cls_token', extraction_layer=-1)[source]

Converts a text into vector(s) using the language model only (no prediction head involved).

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • extraction_strategy (str) – Strategy to extract vectors. Choices: ‘cls_token’ (sentence vector), ‘reduce_mean’ (sentence vector), reduce_max (sentence vector), ‘per_token’ (individual token vectors)

  • extraction_layer – number of layer from which the embeddings shall be extracted. Default: -1 (very last layer).

Type

int

Returns

dict of predictions

class farm.infer.FasttextInferencer(model, name=None)[source]

Bases: object

__init__(model, name=None)[source]

Initialize self. See help(type(self)) for accurate signature.

classmethod load(load_dir, batch_size=4, gpu=False, embedder_only=True)[source]
extract_vectors(dicts, extraction_strategy='reduce_mean')[source]

Converts a text into vector(s) using the language model only (no prediction head involved).

Parameters
  • dicts ([dict]) – Samples to run inference on provided as a list of dicts. One dict per sample.

  • extraction_strategy (str) – Strategy to extract vectors. Choices: ‘reduce_mean’ (mean sentence vector), ‘reduce_max’ (max per embedding dim), ‘CLS’

Returns

dict of predictions

Experiment

farm.experiment.load_experiments(file)[source]
farm.experiment.run_experiment(args)[source]
farm.experiment.get_adaptive_model(lm_output_type, prediction_heads, layer_dims, model, device, embeds_dropout_prob, class_weights=None)[source]
farm.experiment.validate_args(args)[source]
farm.experiment.save_model()[source]
farm.experiment.load_model()[source]

Metrics

farm.metrics.simple_accuracy(preds, labels)[source]
farm.metrics.acc_and_f1(preds, labels)[source]
farm.metrics.f1_macro(preds, labels)[source]
farm.metrics.pearson_and_spearman(preds, labels)[source]
farm.metrics.compute_metrics(metric, preds, labels)[source]
farm.metrics.squad_EM(preds, labels)[source]
farm.metrics.squad_f1(preds, labels)[source]
farm.metrics.squad_f1_single(pred, label)[source]
farm.metrics.squad(preds, labels)[source]

File utils

Utilities for working with the local dataset cache. This file is adapted from the AllenNLP library at https://github.com/allenai/allennlp Copyright by the AllenNLP authors.

farm.file_utils.url_to_filename(url, etag=None)[source]

Convert url into a hashed filename in a repeatable way. If etag is specified, append its hash to the url’s, delimited by a period.

farm.file_utils.filename_to_url(filename, cache_dir=None)[source]

Return the url and etag (which may be None) stored for filename. Raise EnvironmentError if filename or its stored metadata do not exist.

farm.file_utils.cached_path(url_or_filename, cache_dir=None)[source]

Given something that might be a URL (or might be a local path), determine which. If it’s a URL, download the file and cache it, and return the path to the cached file. If it’s already a local path, make sure the file exists and then return the path.

farm.file_utils.split_s3_path(url)[source]

Split a full s3 path into the bucket name and path.

farm.file_utils.s3_request(func)[source]

Wrapper function for s3 requests in order to create more helpful error messages.

farm.file_utils.s3_etag(url)[source]

Check ETag on S3 object.

farm.file_utils.s3_get(url, temp_file)[source]

Pull a file directly from S3.

farm.file_utils.http_get(url, temp_file, proxies=None)[source]
farm.file_utils.get_from_cache(url, cache_dir=None)[source]

Given a URL, look for the corresponding dataset in the local cache. If it’s not there, download it. Then return the path to the cached file.

farm.file_utils.read_set_from_file(filename)[source]

Extract a de-duped collection (set) of text from a file. Expected file format is one item per line.

farm.file_utils.get_file_extension(path, dot=True, lower=True)[source]
farm.file_utils.read_config(path, flattend=False)[source]
farm.file_utils.unnestConfig(config, flattened=False)[source]

This function creates a list of config files for evaluating parameters with different values. If a config parameter is of type list this list is iterated over and a config object without lists is returned. Can handle lists inside any number of parameters.

Can handle shallow or nested (one level) configs