# Delivery specification

## Formats

| format | use | note |
|---|---|---|
| Parquet (zstd, row groups 128k) | default bulk | ~5× smaller than CSV; column pruning works |
| CSV (UTF-8, `,`, header) | spot checks, Excel, legacy loaders | one file per session per symbol |
| DuckDB / SQLite file | single-file research copy | opt-in, built on request |

CSV is about **5.0×** the Parquet bytes on the shipped sample (14.6 MB vs
2.9 MB for 68,250 rows). Size a CSV order at five times the Parquet figure.

## Layout

Two equivalent layouts; pick one at order time.

```
by-symbol/                      by-day/
  SPY/                            2020-03-16/
    SPY-2008.parquet                SPY.parquet
    SPY-2009.parquet                NVDA.parquet
    ...                             ...
  NVDA/                          2020-03-17/
    ...                             ...
```

- Partition unit: one Parquet per symbol per **calendar year**, or one per
  session in `by-day`. Nothing bigger than 1 GB per object.
- A `manifest.jsonl` accompanies every order: object key, rows, bytes,
  SHA-256, min/max session, schema version.
- `SHA256SUMS` at the archive root, plus a detached signature on the manifest.
- Schema version is in the path (`v1/`); a field change is a new version, never
  an in-place rewrite.

## Size

Measured on the eight deep symbols actually held:

| item | value |
|---|---|
| rows | 69,856,552 |
| Parquet | 1.72 GB |
| CSV equivalent | ~8.6 GB |
| compressed tarball, whole order | ~1.7 GB |

The full 2008–2025 history for those eight names fits on a thumb drive. A
1,500-name broad order is quoted after the universe is fixed, because row count
scales with the name list and expiry calendar, not with a flat per-day figure.

## Transfer

- Cloudflare R2 bucket, S3-compatible, presigned URLs; `aws s3 sync` or `rclone`
  both work and resume.
- Single-order cap of 50 concurrent object fetches; no per-day request cap.
- HTTP `Range` supported, so Parquet footers and single columns can be pulled
  without the full object.
- API for incremental work: `GET /v1/chains/{symbol}?from=&to=&cursor=`,
  1,000 rows per page, ETag on every page, `next_cursor` until exhausted.
- No FTP. SFTP on request for buyers with a fixed ingest box.

## Freshness and corrections

- Broad mirror top-up runs after the 16:00 ET close; files land by 18:30 ET.
- Corrections arrive as a dated `corrections/` overlay that supersedes the base
  row on `(contract, quote_date)`. Loaders must apply overlays last; the base
  file is never rewritten under you.
- Every delivery carries the upstream ingest timestamp per row
  (`ingest_utc`), so staleness is auditable.

## Licensing and redistribution

Upstream mirrors are public and redistributable for research with attribution;
the delivery ships `NOTICE.md` naming each source. Exchange fees and
redistribution inside a commercial product are the buyer's to confirm. The
licensed top-up (see `COVERAGE.md`) carries a vendor licence and is quoted
separately.

## What you get to check first

The free sample in `data/sample/` is the exact production schema, produced by
the same pipeline. `pipeline/verify.py` recomputes counts, checksums, parity and
the split case before you commit to an order.