ethereum-etl

Python

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Last updated Jul 8, 2026

3.1k

Stars

904

Forks

153

Issues

Stars/day

Attention Score

View on GitHub ↗ Open in github.dev

Topics

#aws #bigquery #blockchain-analytics #csv #erc20 #erc20-tokens #erc721 #ethereum #etl #export #gcp #google-cloud #sql #transaction

Language breakdown

Python 99.2%

Shell 0.6%

HTML 0.2%

Dockerfile 0.0%

▸ Files click to expand

README

Ethereum ETL

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl exportblocksand_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl exporttokentransfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, tokentransfers continually to console (Reference):

> pip3 install ethereum-etl[streaming] > ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \ --provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call
> pip3 install -e . > python3 ethereumetl.py

Rust Implementation

For improved performance, check out ethereum-etl.rs - a Rust rewrite that is ~1.4x faster than this Python version while maintaining full CLI compatibility.

Useful Links

Schema

Command Reference

Documentation

Public Datasets in BigQuery

Exporting the Blockchain

Querying in Amazon Athena

Querying in Google BigQuery

Querying in Kaggle

Airflow DAGs

Postgres ETL

Ethereum 2.0 ETL

Running Tests

> pip3 install -e .[dev,streaming] > export ETHEREUMETLRUNSLOWTESTS=True > export PROVIDERURL=<yourprovider_uri> > pytest -vv

Running Tox Tests

> pip3 install tox > tox

Running in Docker

Install Docker: https://docs.docker.com/get-docker/

Build a docker image

> docker build -t ethereum-etl:latest . > docker image ls
Run a container out of the image

> docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io
Run streaming to console or Pub/Sub

> docker build -t ethereum-etl:latest . > echo "Stream to console" > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt > echo "Stream to Pub/Sub" > docker run -v /pathtocredentialsfile/:/ethereum-etl/ --env GOOGLEAPPLICATIONCREDENTIALS=/ethereum-etl/credentialsfile.json ethereum-etl:latest stream --start-block 500000 --output projects/project>/topics/cryptoethereum

If running on an Apple M1 chip add the --platform linux/x86_64 option to the build and run commands e.g.:

docker build --platform linux/x86_64 -t ethereum-etl:latest .
docker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000

Projects using Ethereum ETL

Google - Public BigQuery Ethereum datasets
Nansen - Analytics platform for Ethereum

🔗 More in this category