# ML Datasets That Train AI at the Edge

> Deploy machine learning datasets with automatic versioning and lineage tracking. Stream training data efficiently and track experiments across teams.

Canonical: https://datasets.do/

## In code

```ts
import { Datasets } from 'datasets.do'

// Collect training data from events
const dataset = Datasets({ type: 'ChurnPrediction', version: '2024-01' })

on.Customer.cancels($ => {
  dataset.add.Example({ features: $.Customer.data, label: 'churned' })
})

on.Model.training($ => {
  dataset.stream.Batches({ size: 1024 }).forEach(batch => $.Model.train(batch))
})

dataset`compare performance across last 5 versions`
```

## Frequently asked questions

### How does datasets.do differ from data warehouses?

Datasets.do is optimized for ML workflows with automatic versioning (track every dataset iteration), feature lineage tracking (know provenance of every column), schema evolution (add features without breaking models), and streaming access (TBps throughput for distributed training). Data warehouses are designed for analytical queries, not ML training. Datasets.do stores raw features, transformed features, and labels together with automatic format conversion (Parquet, TFRecord, CSV). Integrated experiment tracking links datasets to model versions, hyperparameters, and performance metrics. Global edge deployment enables training on regional data without cross-region transfer—comply with data residency while training globally distributed models.

### Can I version datasets and track experiments?

Yes. Every dataset change creates a new immutable version with automatic snapshots. Track which dataset versions were used to train which models with full lineage graphs. Compare model performance across dataset versions to identify data quality issues or distribution shifts. Semantic versioning (major.minor.patch) for breaking vs non-breaking changes. Time-travel queries access historical dataset states: "show me training data as it existed on 2024-01-15". Automatic data quality monitoring detects schema violations, null rates, outliers, and drift. Experiment tracking records hyperparameters, metrics, artifacts, and logs linked to specific dataset versions. Reproducibility guaranteed—recreate any training run exactly.

### How does feature lineage and governance work?

Automatic lineage tracking records complete data provenance from source systems through transformations to final features. Visual lineage graphs show which tables, columns, and transformations produced each feature. Impact analysis identifies downstream datasets and models affected by upstream data changes. Feature store integration shares features across teams with consistent definitions and computation logic. Data governance policies enforce access controls (PII protection, GDPR compliance), retention policies (auto-delete after N days), and quality constraints (required fields, value ranges). Automatic PII detection flags sensitive features. Audit logs track who accessed which features when for compliance reporting.

### What about performance for large-scale training?

Stream billions of records at TBps speeds with parallel data loading across distributed training clusters. Automatic sharding distributes dataset across edge locations close to training nodes—eliminate cross-region data transfer. Smart caching pre-loads next batches while current batch trains. Format optimization converts data to most efficient format for your framework (TensorFlow, PyTorch, JAX). Compressed storage reduces costs by 10x while maintaining fast decompression. Handle datasets from MBs to 100TB+ with automatic partitioning. Random access patterns supported for non-sequential sampling strategies. Zero-copy streaming minimizes memory overhead and maximizes GPU utilization.

### Can I integrate with existing ML tools and platforms?

Yes. Native integrations with TensorFlow, PyTorch, JAX, Hugging Face, scikit-learn, and all major ML frameworks. MLflow and Weights & Biases integration for experiment tracking. Kubeflow and SageMaker connectors for orchestration. DVC and MLflow for model versioning. Jupyter notebook extensions for interactive data exploration. REST API and Python SDK for custom integrations. Export to standard formats (Parquet, CSV, TFRecord, JSONL) for any tool. Use datasets.do as your central feature store and training data repository while leveraging existing ML infrastructure for training, serving, and monitoring.

## Part of Platform.do

datasets.do is one of the primitives of Platform.do, in the Data layer (records, vocabulary and search). Every .do shares one grammar, Business-as-Code, and one SDK.

- What is Business-as-Code? https://platform.do/business-as-code
- Platform.do: https://platform.do/
- Docs: https://docs.platform.do