Data preparation

The primary data preparation tooling used in this project is based in the radiance tool developed by Jump’s Firedancer team. It is rapidely developing, and active development for this project is currently based out of this repository and branch: Radiance Triton.

The radiance tool utilises the rocksdb snapshots that have been generated by Warehouse nodes. From these snapshots a CAR file per epoch is generated. This CAR file then needs to be processed by Filecoin tools such as split-and-commp which generates the details needed for making a Filecoin deal.

Currently, this tool is being tested from the following warehouse archives:

  • Solana Foundation (public)
    • gs://mainnet-beta-ledger-us-ny5
    • gs://mainnet-beta-ledger-europe-fr2
    • gs://mainnet-beta-ledger-asia-sg1
  • Triton One (private)

If you have warehouse nodes generating rocksdb archive snapshots, please contact (even if they can’t be made publicly available). We would like to have you generate CAR files for verification purposes.

CAR files

CAR file generation produces a CAR containing a DAG. This DAG is reproducible and follows the structure of Epoch -> Block -> Transaction see schema. The CAR file generation is deterministic, so even if you use different rocksdb source snapshots you should end up with the same CAR output. This allows comparison between different providers.

The data generation flow is illustrated below:

radiance datagen flow

Generating an epoch car file

Once you have downloaded rocksdb ledger archives you can run the Radiance tool to generate a car file for an epoch. Make sure you have all the slots available in rocksdb ledger archive for the epoch. You may need to download multiple ledger snapshots in order to have a full set of slots available. Once you know you have a rocksdb that covers all the slots for the epoch run the radiance tool like follows:

radiance car create2 107 --db=46223992/rocksdb --out=/storage/car/

This will produce a car file called containing all the blocks and transactions for that epoch.