farm.metrics.register_metrics(name, implementation)[source]
farm.metrics.simple_accuracy(preds, labels)[source]
farm.metrics.acc_and_f1(preds, labels)[source]
farm.metrics.f1_macro(preds, labels)[source]
farm.metrics.pearson_and_spearman(preds, labels)[source]
farm.metrics.compute_metrics(metric, preds, labels)[source]
farm.metrics.squad_EM(preds, labels)[source]
farm.metrics.squad_f1(preds, labels)[source]
farm.metrics.squad_f1_single(pred, label)[source]
farm.metrics.squad(preds, labels)[source]

File utils

Utilities for working with the local dataset cache. This file is adapted from the AllenNLP library at Copyright by the AllenNLP authors.

farm.file_utils.url_to_filename(url, etag=None)[source]

Convert url into a hashed filename in a repeatable way. If etag is specified, append its hash to the url’s, delimited by a period.

farm.file_utils.filename_to_url(filename, cache_dir=None)[source]

Return the url and etag (which may be None) stored for filename. Raise EnvironmentError if filename or its stored metadata do not exist.

farm.file_utils.cached_path(url_or_filename, cache_dir=None)[source]

Given something that might be a URL (or might be a local path), determine which. If it’s a URL, download the file and cache it, and return the path to the cached file. If it’s already a local path, make sure the file exists and then return the path.


Split a full s3 path into the bucket name and path.


Wrapper function for s3 requests in order to create more helpful error messages.


Check ETag on S3 object.

farm.file_utils.s3_get(url, temp_file)[source]

Pull a file directly from S3.

farm.file_utils.http_get(url, temp_file, proxies=None)[source]
farm.file_utils.get_from_cache(url, cache_dir=None)[source]

Given a URL, look for the corresponding dataset in the local cache. If it’s not there, download it. Then return the path to the cached file.


Extract a de-duped collection (set) of text from a file. Expected file format is one item per line.

farm.file_utils.get_file_extension(path, dot=True, lower=True)[source]
farm.file_utils.read_config(path, flattend=False)[source]
farm.file_utils.unnestConfig(config, flattened=False)[source]

This function creates a list of config files for evaluating parameters with different values. If a config parameter is of type list this list is iterated over and a config object without lists is returned. Can handle lists inside any number of parameters.

Can handle shallow or nested (one level) configs