Stream a dataset into training

Keep training datasets on Usenet and read them straight into standard tooling — the Hugging Face datasets library, pandas, PyTorch — without staging a copy on every machine that trains.

The training box needs pip install datasets s3fs and a scoped read-only key. No Usenet account, no UsenetShare license, no SDK of ours.

Why this shape works

A dataset is read many times by many machines and is usually far larger than any one of them wants to store. The normal answer is to copy it to each runner first, which costs the copy every time and a disk big enough to hold it.

Streaming removes both. Reads are served range-on-demand, so a run touches the bytes it uses. Two formats make that genuinely efficient rather than merely possible:

Parquet — column projection and filter pushdown happen before the fetch. Read three columns from a wide table and only those columns cross the network.

WebDataset (.tar) — designed for sequential shard reading, which is the access pattern streaming is best at.

jsonl, csv, plain text and loose files all work through the matching datasets builder. They just cannot prune bytes the way Parquet can.

Start at the AI & Datasets page in the app for the full picture, including performance notes and dataset cards.

Publishing a dataset

Publish the folder exactly as any other share — see Upload and publish. Two things make it better to consume:

Shard it. Parquet shards or WebDataset tars, not one enormous file. Shards are what let readers parallelise and skip.

Include a README.md dataset card in the folder. It travels with the data, so whoever consumes it later has the provenance, licence and column meanings without asking you.

Consuming it on a training box

1. Open the share in UsenetShare and choose Connect to AI / ML.

2. Download the kit. It carries a scoped, read-only, revocable key limited to that one dataset.

3. On the training box:

   pip install datasets s3fs
   

4. Run the kit's snippet.

The shape of what it configures, for orientation — the kit fills in your real endpoint and key:

from datasets import load_dataset

ds = load_dataset(
    "parquet",
    data_files="s3://<share-id>/train/*.parquet",
    storage_options={
        "key": "<scoped access key>",
        "secret": "<scoped secret>",
        "client_kwargs": {"endpoint_url": "http://<endpoint>:8086"},
    },
    streaming=True,
)

Because it is fsspec underneath, pandas.read_parquet, pyarrow.dataset and anything else that accepts an fsspec URL work the same way.

When you do want a local copy

Streaming is not always right. A run that reads the same data twenty times is better off with it local, and a first-party CLI does that:

uss sync ls   <share-id>            # what is in it
uss sync pull <share-id> ./data     # fetch it all locally
uss sync push ./outputs             # publish results, get a share link

rclone and the aws CLI work equally well for the pull — it is ordinary S3. See S3 object storage.

uss sync push is the useful one for pipelines: a CI job or an agent can publish its artifacts and hand back a share link, with no bucket to provision.

Requirements and limits

On the UsenetShare machine: the S3 gateway enabled, and reachable from the training box — which means a Public endpoint set if they are different machines. See S3 object storage.

Read-only. Training reads; it cannot write back into a share. Publish outputs as a new share instead.

Connections are shared with downloading, half each by default, with a banner to change the split. Several runners streaming at once share the same provider limit — that is the ceiling to plan around, and a second provider account raises it.

Random tiny reads are the weak case. A shuffle that reads scattered single records pays a round trip each time. Shard-level shuffling, which is what WebDataset expects, is the pattern that performs.

Next

• AI & Datasets page in the app — formats, performance, dataset cards
• /api-docs/datasets — the wire reference and copy-paste recipes
• S3 object storage
• Upload and publish