> For the complete documentation index, see [llms.txt](https://ai-materials-and-chemistry.gitbook.io/foundry-1.0/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai-materials-and-chemistry.gitbook.io/foundry-1.0/loading-and-using/examples.md).
# Using Datasets
## Scientific Examples
We have [example notebooks](https://github.com/MLMI2-CSSI/foundry/tree/main/examples) using material science data that illustrate how to load data or publish datasets using Foundry. These notebooks are compatible with Google Colab and with Jupyter Notebook. However, some of the datasets are quite large and cannot be loaded into Google Colab without the pro version.
## Quickstart
### Creating a Foundry Client
The Foundry client provides access to all of the methods described here for listing, loading, and publishing datasets and models. The code below will create a Foundry client. If you'd like to use https instead of Globus when downloading data, specify that here with `Globus=False`.
```python
from foundry import Foundry
f = Foundry()
```
{% hint style="success" %}
If you are running your script on cloud resources (e.g. Google Colab, Binder), see: [Using Foundry on Cloud Computing Resources](/foundry-1.0/loading-and-using/examples.md#using-foundry-on-cloud-computing-resources)
{% endhint %}
### Listing Datasets
To show all available Foundry datasets, you can use the Foundry [`list()` method](/foundry-1.0/classes-and-methods/foundry.foundry.md) as follows. The method returns a pandas DataFrame with details on the available datasets.
```python
f.list()
```
### Searching Datasets
The Foundry client can be used to search for datasets using a `source_id` or a digital object identifier (DOI) e.g. here `"foundry_wei_atom_locating_benchmark"` or `"10.18126/e73h-3w6n"`*.* You can retrieve the `source_id`from the [`list()` method](/foundry-1.0/loading-and-using/examples.md#listing-datasets).
```python
from foundry import Foundry
f = Foundry()
dataset_doi = '10.18126/e73h-3w6n'
datasets = f.search(dataset_doi)
```
The search method will return a list of `FoundryDataset` objects. If searching by DOI, you can expect that it will be the first result and select that `FoundryDataset` object by specifying the first index.
```
dataset = datasets[0]
```
### Finding Datasets
Instead of using `search`, you can use the `get_dataset()` method. This method returns a FoundryDataset associated with the given name or DOI.
```
from foundry import Foundry
f = Foundry()
dataset_doi = '10.18126/e73h-3w6n'
dataset = f.get_dataset(dataset_doi)
```
### Loading Datasets
To load the dataset, you can use the `get_as_dict()` method appended to the `FoundryDataset` object. From the example above, this looks like:
```
dataset.get_as_dict()
```
This will remotely load the metadata (e.g., data location, data keys, etc.) and download the data to local storage if it is not already cached.
{% hint style="success" %}
All datasets are accessible via HTTPS and Globus by authenticated download. HTTPS is the default. Read about the [FoundryDataset object](/foundry-1.0/classes-and-methods/foundry.foundry_dataset.md) to use Globus.
{% endhint %}
### Viewing Metadata
To learn more about the dataset, you can access the metadata. The `get_metadata_by_doi()` method returns a JSON object with all the metadata.
f.get_metadata_by_doi('10.18126/e73h-3w6n')
The image below is what the metadata looks like when the dataset variable is printed in a notebook. This table contains the dataset's [metadata](/foundry-1.0/publishing/describing-datasets.md#descriptive-metadata).
This is an example of accessing data in a specified [split](/foundry-1.0/publishing/describing-datasets.md#splits) that is defined for the dataset, here we use `train`.
```python
imgs = dataset['train']['input']['imgs']
coords = dataset['train']['input']['coords']
# Show some images with coordinate overlays
import matplotlib.pyplot as plt
n_images = 3
offset = 150
key_list = list(res['train']['input']['imgs'].keys())[0+offset:n_images+offset]
fig, axs = plt.subplots(1, n_images, figsize=(20,20))
for i in range(n_images):
axs[i].imshow(imgs[key_list[i]])
axs[i].scatter(coords[key_list[i]][:,0],
coords[key_list[i]][:,1], s = 20, c = 'r', alpha=0.5)
```
## Using Foundry on Cloud Computing Resources
Foundry works with common cloud computing providers (e.g., the NSF sponsored Jetstream and Google Colab). On these resources, simply add the following arguments to use a cloud-compatible authentication flow.
```python
f = Foundry(no_browser=True, no_local_server=True)
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://ai-materials-and-chemistry.gitbook.io/foundry-1.0/loading-and-using/examples.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.