foundry.foundry_dataset

module foundry.foundry_dataset

class FoundryDataset

Representation of an individual dataset. Provides access to metadata as well as functions to instantiate data into memory in different formats.

Args:

  • dataset_name (str): name of dataset (equivalent to source_id in MDF)

  • splits List[FoundrySplit]: list of splits in the dataset

  • use_globus (bool): if True, use Globus to download the data else try HTTPS

  • interval (int): How often to wait before checking Globus transfer status

  • parallel_https (int): Number of files to download in parallel if using HTTPS

  • verbose (bool): Produce more debug messages to screen

Desired functions: - Get as pandas - Get as tensorflow dataset - Get as pytorch dataset - Get file list - Set metadata - Attach datafiles - Validate against schema - Get citation

method __init__

__init__(
    dataset_name: str,
    datacite_entry: dict,
    transfer_client: Any,
    foundry_schema: FoundrySchema,
    use_globus: bool = False,
    interval: int = 10,
    parallel_https: int = 4,
    verbose: bool = False,
    forge_client: Forge = None,
    local_cache_dir: str = None
)

method clear_dataset_cache

clear_dataset_cache()

Deletes the cached data for this specific datset

method get_as_dict

get_as_dict(split: str = None, as_hdf5: bool = False)

Returns the data from the dataset as a dictionary

Arguments:

  • split (string): Split to create dataset on.

  • **Default: ** None

Returns: (dict) Dictionary of all the data from the specified split

method get_as_tensorflow

get_as_tensorflow(split: str = None)

Convert Foundry Dataset to a Tensorflow Sequence

Arguments:

  • split (string): Split to create Tensorflow Sequence on.

  • **Default: ** None

Returns: (TensorflowSequence) Tensorflow Sequence of all the data from the specified split

method get_as_torch

get_as_torch(split: str = None)

Returns the data from the dataset as a TorchDataset

Arguments:

  • split (string): Split to create PyTorch Dataset on.

  • **Default: ** None

Returns: (TorchDataset) PyTorch Dataset of all the data from the specified split

method get_citation

get_citation() → str

method validate_metadata

validate_metadata(metadata)

Validate the JSON message against the FoundryDataset model

Arguments:

  • metadata (dict): Metadata information provided by the user.

Raises:

  • ValidationError: if metadata supplied by user does not meet the specificiation of a FoundryDataset object.

This file was automatically generated via lazydocs.

Previousfoundry.foundry_cacheNextfoundry.https_download