kay-ou
kay-ou
SimTradeData
Python

SimTradeData is a utility library supporting SimTradeDesk, SimTradeLab and simtradeML with reliable, high-quality simulated trading data for model training, backtesting, and performance evaluation.

Last updated Jul 3, 2026
82
Stars
28
Forks
4
Issues
0
Stars/day
Attention Score
42
View on GitHub β†— Open in github.dev
Topics
#backtesting#data-engineering#financial-data#python3
Language breakdown
No language data available.
β–Έ Files click to expand
README

English | δΈ­ζ–‡ | Deutsch

SimTradeData - Quantitative Trading Data Downloader

Python 3.10+ License: AGPL-3.0 Version DuckDB Parquet Code Style: Black Poetry

BaoStock + Mootdx + EastMoney + yfinance Multi-Source | China A-Shares + US Stocks | PTrade Compatible | DuckDB + Parquet Storage

SimTradeData is an efficient data download tool designed for SimTradeLab. It supports China A-shares (BaoStock, Mootdx, EastMoney) and US stocks (yfinance) from multiple data sources, automatically orchestrating each source's strengths. Data is stored in DuckDB as intermediate storage and exported to Parquet format, with efficient incremental updates and querying.

Recommended Combo: SimTradeData + SimTradeLab

Fully PTrade Compatible | A-Shares + US Stocks | 20x+ Backtesting Speedup

SimTradeLab

No PTrade Strategy Code Changes Needed | Ultra-Fast Local Backtesting | Zero-Cost Solution

Key Features

Efficient Storage Architecture

  • DuckDB Intermediate Storage: High-performance columnar database with SQL queries and incremental updates
  • Parquet Export Format: Highly compressed, cross-platform compatible, ideal for large-scale data analysis
  • Automatic Incremental Updates: Intelligently detects existing data, only downloads new records

Comprehensive Data Coverage

  • Market Data: OHLCV daily bars with limit-up/down prices and previous close
  • Valuation Metrics: PE/PB/PS/PCF/Turnover Rate/Total Shares/Float Shares
  • Financial Data: 23 quarterly financial indicators + automatic TTM calculation
  • Corporate Actions: Dividends, bonus shares, rights offerings (with forward adjustment factors)
  • Metadata: Stock info, trading calendar, index constituents, ST/suspension status
  • US Stock Support: 6,000+ US common stocks, S&P 500 / NASDAQ-100 index constituents

Data Quality Assurance

  • Auto-Validation: Data integrity validation before writes
  • Export-Time Calculation: Limit prices, TTM metrics computed at export for consistency
  • Detailed Logging: Comprehensive error logs and warnings

Generated Data Structure

data/
β”œβ”€β”€ cn.duckdb          # DuckDB database - A-shares (download source)
β”œβ”€β”€ us.duckdb             # DuckDB database - US stocks (download source)
└── export/                      # Exported Parquet files (by market)
    β”œβ”€β”€ cn/                      # A-shares export
    β”‚   β”œβ”€β”€ stocks/              # Daily bars (one file per stock)
    β”‚   β”‚   β”œβ”€β”€ 000001.SZ.parquet
    β”‚   β”‚   └── 600519.SS.parquet
    β”‚   β”œβ”€β”€ exrights/            # Corporate action events
    β”‚   β”œβ”€β”€ fundamentals/        # Quarterly financials (with TTM)
    β”‚   β”œβ”€β”€ valuation/           # Valuation metrics (daily)
    β”‚   β”œβ”€β”€ metadata/            # Metadata
    β”‚   └── manifest.json
    └── us/                      # US stocks export
        β”œβ”€β”€ stocks/
        β”‚   β”œβ”€β”€ AAPL.US.parquet
        β”‚   └── MSFT.US.parquet
        β”œβ”€β”€ exrights/
        β”œβ”€β”€ fundamentals/
        β”œβ”€β”€ valuation/
        β”œβ”€β”€ metadata/
        └── manifest.json

Prerequisites

  • Python: 3.10 or higher
  • Poetry: Installation guide
  • Network: Required for downloading data from BaoStock/Mootdx/EastMoney/yfinance (China mainland network recommended for A-share data)

Quick Start

Option 1: Download Pre-Built Data (Recommended)

Download the latest data from Releases:

  • A-shares: data-cn-v* release β†’ extract to data/cn/
  • US stocks: data-us-v* release β†’ extract to data/us/
# A-shares
mkdir -p /path/to/SimTradeLab/data/cn
tar -xzf simtradelab-data-cn-*.tar.gz -C /path/to/SimTradeLab/data/cn/

US stocks

mkdir -p /path/to/SimTradeLab/data/us
tar -xzf simtradelab-data-us-*.tar.gz -C /path/to/SimTradeLab/data/us/

Option 2: Download Data Yourself

1. Install Dependencies

# Clone the project
git clone https://github.com/kay-ou/SimTradeData.git
cd SimTradeData

Install dependencies

poetry install

Activate virtual environment

poetry shell

2. Download Data

Recommended: Unified Download Command

A single command downloads all data, using the TDX daily package for the fast OHLCV path and then orchestrating Mootdx and BaoStock for their respective strengths:

# Daily/production CN refresh (recommended)

TDX: OHLCV full history and daily package import


Mootdx: corporate actions, bulk financials, trading calendar, benchmark index


BaoStock: valuation metrics, ST/suspension status, index constituents

poetry run python scripts/download.py --tdx-download --skip-mootdx-ohlcv

Fallback full orchestration without the TDX package

poetry run python scripts/download.py

Use an already-downloaded TDX ZIP file

poetry run python scripts/download.py --tdx-source data/downloads/hsjday.zip --skip-mootdx-ohlcv

Check data status

poetry run python scripts/download.py --status

Skip financial data (faster)

poetry run python scripts/download.py --tdx-download --skip-mootdx-ohlcv --skip-fundamentals

Run Mootdx phase only

poetry run python scripts/download.py --source mootdx

Run BaoStock phase only

poetry run python scripts/download.py --source baostock

Data Source Division of Labor

| Data Type | Source | Reason | |-----------|--------|--------| | OHLCV Market Data (daily/full history) | TDX Daily Package | Fastest, ~500MB bulk import of full history and latest daily data | | OHLCV Market Data (fallback/backfill) | Mootdx | Per-symbol fallback when the TDX package is unavailable | | Corporate Actions (XDXR) | Mootdx | More complete data | | Bulk Financial Data | Mootdx | One ZIP = all stocks, far better than per-stock queries | | Valuation PE/PB/PS/Turnover | BaoStock | Exclusive data | | ST/Suspension Status | BaoStock | Exclusive data | | Index Constituents | BaoStock | Exclusive data | | Trading Calendar | Mootdx | Provides exchange calendar data | | Benchmark Index | Mootdx | Provides benchmark series |

Using Individual Data Sources

# BaoStock (includes valuation data, but slower)
poetry run python scripts/download_efficient.py
poetry run python scripts/download_efficient.py --skip-fundamentals
poetry run python scripts/download_efficient.py --valuation-only  # Valuation + status only

Mootdx (faster, but no valuation data)

poetry run python scripts/download_mootdx.py
poetry run python scripts/download_mootdx.py --skip-fundamentals

EastMoney Complementary Data (Money Flow, Dragon Tiger Board, Margin Trading)

# Download last 30 days of complementary data (requires existing market data)
poetry run python scripts/downloaddailyextras.py

Specify number of days (LHB API only retains ~30 days, run regularly)

poetry run python scripts/downloaddailyextras.py --days 7

US Stock Data (yfinance)

Free US stock data via yfinance, no API key required:

# Full download (6,000+ US stocks with OHLCV + financials + valuation + metadata)
poetry run python scripts/download_us.py

Specific symbols (small-scale testing)

poetry run python scripts/download_us.py --symbols AAPL,MSFT,GOOGL

Market data only (skip time-consuming per-stock financials and metadata)

poetry run python scripts/download_us.py --skip-fundamentals --skip-metadata

Specify start date

poetry run python scripts/download_us.py --start-date 2020-01-01

US stock ticker format: AAPL.US (consistent with A-shares 600000.SS using {code}.{market}), stored in a separate database data/us.duckdb.

TDX Official Data Package (Fastest Way to Get Full Historical Data)

# Auto-download official TDX Shanghai/Shenzhen/Beijing daily data package (~500MB)
poetry run python scripts/downloadtdxday.py

Force re-download

poetry run python scripts/downloadtdxday.py --force-download

Use an already-downloaded file

poetry run python scripts/downloadtdxday.py --file hsjday.zip

3. Export to Parquet

# Export A-shares β†’ data/export/cn/
poetry run python scripts/export_parquet.py

Export US stocks β†’ data/export/us/

poetry run python scripts/export_parquet.py --market us

Custom output directory

poetry run python scripts/export_parquet.py --market cn --output /custom/path

4. Release to GitHub (Maintainer)

# Release A-shares data
bash scripts/release_data.sh --market cn

Release US stock data

bash scripts/release_data.sh --market us

Specify version

bash scripts/release_data.sh --market cn 1.3.0

5. Use in SimTradeLab

# Copy exported data to SimTradeLab data directory
rsync -a data/export/cn/ /path/to/SimTradeLab/data/cn/
rsync -a data/export/us/ /path/to/SimTradeLab/data/us/

Project Architecture

SimTradeData/
β”œβ”€β”€ scripts/
β”‚   β”œβ”€β”€ download.py                # Unified download entry (recommended for A-shares)
β”‚   β”œβ”€β”€ download_efficient.py      # BaoStock download script
β”‚   β”œβ”€β”€ download_mootdx.py         # Mootdx (TDX API) download script
β”‚   β”œβ”€β”€ downloaddailyextras.py   # EastMoney complementary data download script
β”‚   β”œβ”€β”€ downloadtdxday.py        # TDX official daily data package download/import
β”‚   β”œβ”€β”€ download_us.py             # US stock download script (yfinance)
β”‚   β”œβ”€β”€ importtdxday.py          # TDX .day file import script
β”‚   β”œβ”€β”€ export_parquet.py          # Parquet export script
β”‚   └── release_data.sh            # GitHub Release publishing script
β”œβ”€β”€ simtradedata/
β”‚   β”œβ”€β”€ router/
β”‚   β”‚   β”œβ”€β”€ smart_router.py      # SmartRouter - smart data source routing
β”‚   β”‚   β”œβ”€β”€ route_config.py      # Route table configuration
β”‚   β”‚   └── exceptions.py        # Router exceptions
β”‚   β”œβ”€β”€ fetchers/
β”‚   β”‚   β”œβ”€β”€ base_fetcher.py      # Base Fetcher class
β”‚   β”‚   β”œβ”€β”€ baostock_fetcher.py  # BaoStock data fetching
β”‚   β”‚   β”œβ”€β”€ unified_fetcher.py   # BaoStock unified fetching (optimized)
β”‚   β”‚   β”œβ”€β”€ mootdx_fetcher.py    # Mootdx basic data fetching
β”‚   β”‚   β”œβ”€β”€ mootdxunifiedfetcher.py  # Mootdx unified data fetching
β”‚   β”‚   β”œβ”€β”€ mootdxaffairfetcher.py   # Mootdx financial data fetching
β”‚   β”‚   β”œβ”€β”€ eastmoney_fetcher.py # EastMoney complementary data fetching
β”‚   β”‚   └── yfinance_fetcher.py  # yfinance US stock data fetching
β”‚   β”œβ”€β”€ processors/
β”‚   β”‚   └── data_splitter.py     # Data stream splitting
β”‚   β”œβ”€β”€ writers/
β”‚   β”‚   └── duckdb_writer.py     # DuckDB write and export
β”‚   β”œβ”€β”€ validators/
β”‚   β”‚   └── data_validator.py    # Data quality validation
β”‚   β”œβ”€β”€ config/
β”‚   β”‚   β”œβ”€β”€ field_mappings.py    # A-share field mapping config
β”‚   β”‚   β”œβ”€β”€ usfieldmappings.py # US stock field mapping config
β”‚   β”‚   └── mootdxfinvaluemap.py  # Mootdx financial field mapping
β”‚   └── utils/
β”‚       β”œβ”€β”€ code_utils.py        # Stock code conversion
β”‚       └── ttm_calculator.py    # Quarterly range calculation
β”œβ”€β”€ data/                        # Data directory (gitignored)
β”‚   β”œβ”€β”€ cn.duckdb      # A-shares DuckDB source
β”‚   β”œβ”€β”€ us.duckdb         # US stocks DuckDB source
β”‚   └── export/                  # Parquet exports
β”‚       β”œβ”€β”€ cn/                  # A-shares export
β”‚       └── us/                  # US stocks export
└── docs/                        # Documentation
    β”œβ”€β”€ PTRADEPARQUETFORMAT.md # Parquet format specification
    └── PTradeAPImini_Reference.md

Core Modules

1. SmartRouter - Smart Data Source Router

  • Unified data access API, automatically selects the best data source by data type and market
  • Static priority + health-aware: auto fallback to backup sources when primary fails
  • Integrates Phase 1 circuit breaker, skips unhealthy sources
from simtradedata.router import SmartRouter


with SmartRouter() as router:     # Auto-selects best source: mootdx β†’ eastmoney β†’ baostock     df = router.getdailybars("600000.SS", "2024-01-01", "2024-12-31")


# Single-source data also goes through router for unified API     mf = router.getmoneyflow("600000.SS", "2024-01-01", "2024-12-31")


# US stocks auto-route to yfinance     us = router.getdailybars("AAPL.US", "2024-01-01", "2024-12-31")

2. UnifiedDataFetcher - Unified Data Fetching

  • Single API call fetches market, valuation, and status data
  • Reduces API calls by 33%
2. DuckDBWriter - Data Storage and Export
  • Efficient incremental writes (upsert)
  • Computes limit prices and TTM metrics at export time
  • Forward-fills quarterly data to daily frequency
3. DataSplitter - Data Stream Splitting
  • Routes unified data to appropriate tables by type

Data Field Reference

stocks/ - Daily Stock Bars

| Field | Description | |-------|-------------| | date | Trading date | | open/high/low/close | OHLC prices | | highlimit/lowlimit | Limit-up/down prices (computed at export) | | preclose | Previous close price | | volume | Trading volume (shares) | | money | Trading amount (CNY) |

valuation/ - Valuation Metrics (Daily)

| Field | Description | |-------|-------------| | pettm/pb/psttm/pcf | Valuation ratios | | roe/roettm/roa/roattm | Profitability metrics (forward-filled from quarterly reports) | | naps | Net asset per share (computed at export) | | totalshares/afloats | Total shares / float shares | | turnover_rate | Turnover rate |

fundamentals/ - Financial Data (Quarterly)

Contains 23 financial indicators and their TTM versions. See PTRADEPARQUET_FORMAT.md for details.

Configuration

Edit scripts/download_efficient.py:

# Date range
START_DATE = "2017-01-01"
END_DATE = None  # None = current date

Output directory

OUTPUT_DIR = "data"

Batch size

BATCH_SIZE = 20

Documentation

| Document | Description | |----------|-------------| | PTRADEPARQUET_FORMAT.md | Parquet data format specification | | PTradeAPImini_Reference.md | PTrade API reference |

Notes

Data Source Comparison

| Feature | BaoStock | Mootdx API | EastMoney | TDX Official Package | yfinance (US) | |---------|----------|------------|-----------|---------------------|---------------| | Market | A-shares | A-shares | A-shares | A-shares | US stocks | | Speed | Slower | Fast | Fast | Fastest (bulk download) | Medium | | Valuation Data | Yes (PE/PB/PS etc.) | No | No | No | Yes (computed) | | Financial Data | Yes (per-stock query) | Yes (bulk ZIP, faster) | No | No | Yes (per-stock query) | | Money Flow | No | No | Yes (exclusive) | No | No | | Dragon Tiger Board | No | No | Yes (exclusive) | No | No | | Margin Trading | No | No | Yes (exclusive) | No | No | | History Start | 2015 | 2015 | 2015 | Full history | Full history | | API Key | Not required | Not required | Not required | N/A | Not required |

Recommended: Use scripts/download.py --tdx-download --skip-mootdx-ohlcv for daily/production CN refreshes. TDX handles OHLCV, Mootdx supplements corporate actions/financials/calendar/benchmark, and BaoStock handles valuation/status.

Incremental Update Mechanism

  • Market Data: Checks for new trading days; skips in seconds when no new data
  • Financial Data: Incremental checks based on remote file hash; only downloads changed quarters
  • Index Constituents: Tracks downloaded months; only downloads new months
  • Interrupt Recovery: Financial data progress and data are committed in the same transaction; resumes after interruption

Incremental Update Workflow

# 1. Incremental download (fetches only new data, automatically skips existing)
poetry run python scripts/download.py --tdx-download --skip-mootdx-ohlcv

2. Export to Parquet

poetry run python scripts/export_parquet.py              # CN β†’ data/export/cn/
poetry run python scripts/export_parquet.py --market us  # US β†’ data/export/us/

3. Optional: export a client delta package for downstream local merge/rebuild

poetry run python scripts/export_parquet.py --delta --base-version 2026-06-20 --target-version 2026-06-22

Step 1 imports the latest TDX daily package, skips Mootdx OHLCV after that fast path, and refreshes the remaining Mootdx/BaoStock datasets. When there are no new trading days, existing rows are skipped quickly. Delta export contains changed symbol-table rows and a manifest for the requested version window. First install, periodic reconciliation, and failed delta recovery still use the full Parquet export.

For production scheduling, run the daily script later in the trading day (for example after 22:30). It uses the same TDX-backed fast path by default. The recommended production setting is three bounded attempts, each covering download plus pre-release integrity, so transient upstream delays can recover without publishing incomplete data:

DOWNLOADATTEMPTS=3 INTEGRITYSTRICT=1 bash scripts/run_daily.sh

The integrity gate verifies that active stocks have latest market and valuation data, validates the exported manifest, and writes JSON reports under logs/daily/. If all bounded attempts fail, keep the last known-good release and inspect the latest log or alert before running manual recovery.

Data Quality

  • Data sourced from BaoStock free data service
  • For research and educational purposes only

Testing

# Unit tests (no network required)
poetry run pytest tests/ -v

SmartRouter routing and fallback tests

poetry run pytest tests/router/ -v

SmartRouter live integration test (requires network)

poetry run python scripts/testsmartrouter_live.py

Version History

See CHANGELOG.md for the full version history.

Latest: v1.2.0 (2026-03-13) - Smart Data Source Router

Related Links

  • SimTradeLab: https://github.com/kay-ou/SimTradeLab
  • BaoStock: http://baostock.com/
  • Mootdx: https://github.com/mootdx/mootdx
  • EastMoney: https://www.eastmoney.com/
  • yfinance: https://github.com/ranaroussi/yfinance

πŸ’– Sponsor

If this project helps you, consider sponsoring!

| WeChat Pay | Alipay | |:---:|:---:| | WeChat Pay | Alipay |

License

This project is licensed under AGPL-3.0. See the LICENSE file for details.

Status: Production Ready | Version: v1.2.0 | Last Updated: 2026-03-13

πŸ”— More in this category
public-apis
public-apis/public-apis
A collective list of free APIs
β˜… 447.8k
 EbookFoundation
EbookFoundation/free-programming-books
:books: Freely available programming books
β˜… 391.5k
 vinta
vinta/awesome-python
An opinionated list of Python frameworks, libraries, tools, and resources
β˜… 307.0k
 tensorflow
tensorflow/tensorflow
An Open Source Machine Learning Framework for Everyone
β˜… 196.3k
 donnemartin
donnemartin/system-design-primer
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
β˜… 356.6k
 TheAlgorithms
TheAlgorithms/Python
All Algorithms implemented in Python
β˜… 222.5k
 NousResearch
NousResearch/hermes-agent
The agent that grows with you
β˜… 212.4k
 Significant-Gravitas
Significant-Gravitas/AutoGPT
AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
β˜… 185.4k
 huggingface
huggingface/transformers
πŸ€— Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
β˜… 162.4k
 yt-dlp
yt-dlp/yt-dlp
A feature-rich command-line audio/video downloader
β˜… 176.2k
 langgenius
langgenius/dify
Production-ready platform for agentic workflow development.
β˜… 148.4k
 anthropics
anthropics/skills
Public repository for Agent Skills
β˜… 159.3k

Β© 2026 GitRepoTrend Β· kay-ou/SimTradeData Β· Updated daily from GitHub