Large Data
How large datasets can support Worlds without placing raw archives onchain.
Separate archive from workload
A Model may reference terabytes of source material without loading the full archive on every tick. The expected pipeline is:
RAW ARCHIVE → VERSIONED TRANSFORM → RUNTIME DATASET → ACTIVE WORKING SET → MODEL
Raw and derived bytes remain offchain. The Blueprint or World manifest commits to the exact datasets, transforms, versions, and hashes used.
Cost drivers
Archive size alone does not determine runtime cost. The important measures include:
- bytes read and transferred per run;
- preprocessing frequency;
- active memory and storage;
- CPU/GPU time;
- checkpoint size and cadence;
- verifier data and recomputation requirements.
A large archive can be inexpensive when each run uses a small indexed slice. A smaller dataset can be expensive when every tick performs heavy computation.
Reproducibility
Published results must identify the data snapshot used. Any data required to verify an economically relevant result must remain retrievable through the Season, challenge window, and settlement finality period.
Data minimization, partitioning, caching, and precomputed derived features may reduce cost, but none may silently change the declared Model or evidence trail.