AurumQ-RL · A股量化强化学习选股开源项目

AurumQ-RL · An Open-Source Reinforcement Learning Stock-Selection Framework for the China A-Share Market

中文：一份面向 A 股的因子工程 + 强化学习选股参考实现，附完整的迭代史、消融实验、生产化决策与教训。

English: A factor-engineering + reinforcement-learning stock-picking reference implementation for China A-shares, shipped with the full iteration history, ablations, productionization decisions, and lessons learned.

📊 China A-share · 🤖 PPO/A2C/SAC · 🚀 GPU Train + CPU Infer · 📈 Alpha101 + GTJA Alpha191 + Main-Force + Hot-Money + Northbound · 🧪 26 Phases of Open Experiments

---

摘要 / Abstract

中文. AurumQ-RL 是一个针对 A 股市场特有微观结构（T+1、±10% 涨跌停、主板/科创/创业/北交分层、ST 风险警示、申万一级行业、龙虎榜、北向、游资席位、筹码分布）做工程化封装的强化学习选股开源项目。仓库内含：(1) 一个 polars-native 的因子计算引擎，覆盖 105 个 WorldQuant Alpha101 + 191 个国泰君安 Alpha191（合计 296 个量价因子）外加 11 个 A 股私有因子族（mf, mfp, hm, hk, inst, mg, cyq, senti, sh, fund, ind, mkt, gtja, tech, cmf, zt）；(2) Stable-Baselines3 PPO 训练栈，针对 RTX 4070 12 GB 做了 GPU 化重构（per-stock 编码器 + CUDA-resident rollout buffer + 索引化观测）；(3) 完整的 14-phase 训练栈演化史，从最初 11% GPU 利用率到 1M-step 隔夜训练；(4) 26-phase 模型实验史，覆盖奖励重设计、长 panel 消融、rank-z 假设检验、SHAP 剪枝、事件衰减编码等关键转折；(5) ONNX 导出 + CPU 推理生产管线。核心实证发现：(a) rank-z 跨截面归一化会在长 panel 训练中销毁 5-6 bps 的跨年因子幅度信号；(b) 5 年训练窗口已是 plateau，2018-2019 数据零边际贡献；(c) Strategy D（top-K 仓位按分数加权）能与任何基模型叠加 +7-10 bps 的 meany；(d) 二值事件标志直接进 LayerNorm 是 −33% 准确率回归的元凶，必须用 exp-decay τ=10d 编码；(e) cyq 筹码因子的回填 vs 实采分布漂移导致 1.5× T-1 hit 回归，根因是 z-score 不抹平时序 regime 跳变。当前生产状态（2026-05-11）：10 个 modelversion 同时在 Celery Beat 18:50-19:00 排程，新进 best 是 path5long（H1 校准 meany +0.02882，T1_hit 55.8%）。

English. AurumQ-RL is an open-source RL stock-selection framework engineered around the China A-share market's unique microstructure (T+1 settlement; ±10% daily limit per board; main-board vs ChiNext/STAR/BSE segmentation; ST risk-warning; SW Tier-1 industry; Dragon-Tiger List; Northbound; hot-money seats; chip distribution). The repository ships (1) a polars-native factor engine covering 105 WorldQuant Alpha101 + 191 Guotai Junan Alpha191 (296 price-volume factors total) plus 11 private A-share factor families (mf, mfp, hm, hk, inst, mg, cyq, senti, sh, fund, ind, mkt, gtja, tech, cmf, zt); (2) a Stable-Baselines3 PPO training stack with a GPU-vectorized rewrite for the RTX 4070 12 GB (per-stock encoder + CUDA-resident rollout buffer + index-only observations); (3) the complete 14-phase training-stack evolution from 11 % GPU utilization at bring-up to overnight 1M-step training; (4) the 26-phase modeling experiment history covering reward redesign, long-panel ablations, the rank-z hypothesis test, SHAP-based pruning, and event-decay encoding; (5) ONNX export + CPU inference production pipeline. Key empirical findings. (a) Per-day cross-sectional rank-z destroys 5-6 bps of cross-year factor amplitude signal in long-panel training. (b) Five-year training window is the plateau; 2018-2019 data contribute nothing. (c) Strategy D top-K score-weighted sizing compounds +7-10 bps meany onto any base model. (d) Binary event flags fed directly into LayerNorm cause a −33 % accuracy regression and must be encoded as exp-decay with τ=10d. (e) Chip-distribution (cyq) backfill versus real-sample distribution shift drove a 1.5× T-1 hit regression — root cause is that cross-section z-score does not equalize mid-stream temporal regime shifts. Current production state (2026-05-11): 10 model versions co-scheduled in the Celery Beat 18:50-19:00 budget; the new best is path5long (H1 calibrated meany +0.02882, T1hit 55.8 %).

---

目录 / Table of Contents

中文 · English · Phase Timeline

中文导览

• §1 引言：为什么 A 股市场需要专门的 RL 框架
• §2 系统总览：数据契约、因子前缀、宇宙过滤
• §3 因子库：296 量价因子 + 13 个 A 股私有因子族
• §4 训练方法演进：14 个 Phase 的工程史
• §5 模型实验史：26 个 Phase 的研究决策
• §6 监督学习对照赛道（paris 侧 P0/P2/Path 1-6）
• §7 实证发现：6 条改变方向的结论
• §8 生产流水线：每日 18:30-19:00 评分预算
• §9 工程教训：从踩坑到守则
• §10 上手与复现
• §11 路线图、引用、许可
• §12 研究范式分类与未来方向

English TOC

• §1 Introduction: Why A-shares Need a Dedicated RL Framework
• §2 System Overview: Data Contract, Factor Prefixes, Universe Filter
• §3 Factor Library: 296 Price-Volume + 13 Private A-share Families
• §4 Training-Stack Evolution: 14 Phases of Engineering
• §5 Modeling Experiment History: 26 Phases of Research Decisions
• §6 Supervised-Learning Companion Track (paris side P0/P2/Path 1-6)
• §7 Empirical Findings: Six Conclusions That Changed Direction
• §8 Production Pipeline: Daily 18:30-19:00 Scoring Budget
• §9 Engineering Lessons: From Pitfalls to Operating Rules
• §10 Quick Start and Reproduction
• §11 Roadmap, Citation, License
• §12 Research Paradigms and Future Directions

Phase Timeline

Phase 0       Synthetic pipeline-up                 (~pre-2026-04-29)
Phase 1       First real-data alpha101 PPO          2026-04-29..30
Phase 2       Combined 355-col panel + wider net    2026-04-30
Phase 3       R1/R2/R3 smoke-round tuning           2026-05-01 morning
Phase 4       fps scaling, IPC ceiling discovery    2026-05-01 noon
Phase 5       Realizations → GPU framework redesign 2026-05-01 PM
Phase 6/7     GPU-vectorized framework + 50k smoke  2026-05-01 evening
Phase 8       GPURolloutBuffer (CUDA-resident)      2026-05-01 evening
Phase 9       IndexOnlyRolloutBuffer + n_steps=1024 2026-05-01 late evening
Phase 10      Optimizer orphan + LayerNorm + dual pooling  2026-05-01 night
Phase 11/12   bf16 autocast / target_kl=0.10 (eliminated)
Phase 13      PPO SGD perf-probe                    2026-05-01 late night
Phase 14      TF32 + unique-date + 1M overnight     2026-05-02 → 03 early
Phase 15      RL serving integration (champion)     2026-05-02..03
Phase 16-19   Eval correction + multi-seed ensemble 2026-05-03
Phase 20      Long-data PPO                         2026-05-05
Phase 21      V2 forward_10d REJECTED               2026-05-05..06
Phase 22      Main-wave reward redesign             2026-05-06
Phase 23      Episode targets cleanup               2026-05-06
Phase 24/25   Tech-factor + importance-weight REJECT 2026-05-07
Phase 26A-G   cyq fix + event-decay tech (26F prod) 2026-05-07..08
SL companion (paris side, AurumQ repo)
P0            Wave label ablation (Method A wins)   2026-05-09
P2            5-seed ensemble (wavet3lgbm_v2)     2026-05-09
P3            PPO residual ALGORITHM_SPEC v2        2026-05-09
Path 1-6      Multi-path SL exploration              2026-05-10
Long-panel    Hybrid / path1long / path5long      2026-05-10..11

---

§1 引言 — 为什么 A 股市场需要专门的 RL 框架

§1 Introduction — Why A-shares Need a Dedicated RL Framework

1.1 市场微观结构 / Market Microstructure

中文. A 股市场和欧美/港股/加密的微观结构差异远大于表面上的"也是股票"。本项目所有训练、回测、因子计算一律仅在 A 股主板 + 非 ST + 未退市（约 3000 只）上跑，这是 CLAUDE.md 的硬约束。理由是 regime homogeneity：

• 板别价格限制差异：主板 ±10%、ST/*ST ±5%、科创板 ±20%、创业板 ±20%、北交所 ±30%。把这五个池子混进同一个训练 batch，模型必须额外学一个"我现在在哪个板"的元变量，跨年泛化崩溃。
• T+1 结算：当日买入次日才能卖。RL 训练时的"动作-奖励"延迟和欧美/加密 T+0 完全不同；用 t→t+1 的同日 PnL 等价于看穿未来。
• 集合竞价 vs 连续竞价：9:15-9:30 集合竞价是 A 股最重要的"标价"事件之一，stkauction.bid* 字段在 kpllist 里需要单独抓，常规 pctchg/amount 不适用。
• 披露节奏：主板财报 + 业绩预告时点高度同步，跨年信号同质性强；科创板/北交所披露规则差异显著。
• 流动性结构：主板日均成交额 vs 北交所差 1-2 个数量级，分位归一化（rank-z）会把北交所的微弱信号放大到和主板同尺度，污染监督信号。

English. A-share microstructure differs from US/HK/crypto markets in ways that defeat naive "stocks are stocks" transfer. All training, backtest, and factor computation are restricted by hard constraint (CLAUDE.md) to main-board, non-ST, non-delisted stocks (~3000 names). Reasons:

• Board-specific price limits: main-board ±10 %, ST/*ST ±5 %, STAR/ChiNext ±20 %, BSE ±30 %. Mixing these into a single training batch forces the model to learn a "which board am I on" meta-variable, destroying cross-year generalization.
• T+1 settlement: bought today cannot be sold until tomorrow. The action-reward lag in RL training is fundamentally different from T+0 US/crypto. Using same-day t→t+1 PnL is look-ahead.
• Call auction vs continuous trading: the 09:15–09:30 call auction is one of the most informative tape events in A-shares. stkauction.bid* fields in kpllist need special extraction; standard pctchg/amount do not apply.
• Disclosure cadence: main-board earnings + pre-announcement timing is tightly synchronized, giving cross-year signal homogeneity; STAR/BSE differ.
• Liquidity structure: main-board daily turnover vs BSE differs by 1–2 orders of magnitude; rank-z normalization would inflate BSE micro-signals to the same scale as main-board, polluting supervision.

1.2 为什么 RL 而不是因子排序 / Why RL Rather Than Linear Alpha Aggregation

中文. 传统量化做法把 alpha101/gtja191 当 96 个排名分数线性加权，找最优权重向量。问题：

• 截面 IC 是低维度量：300 个因子 × 200 天 = 6 万样本估 300 个权重，过拟合容易，跨年泛化差。
• 非线性交互被忽略：alpha026 × cyqwinning_ratio 在 30% 行业暴露上限下的表现可能远超两者线性和，线性模型抓不到。
• 奖励/成本不进训练目标：传统模型最小化 IC residual，但真实回报是「扣除 30bp 双边费 + T+1 不能反手 + 单行业 30% 上限」之后的实现收益，目标函数和评估指标不一致。
• 状态依赖动作：今天选哪 50 只取决于全市场截面分布，不是任意 200 只独立打分加总。

RL 用神经网络直接学映射「当前因子截面 + 持仓状态 → top-K 动作」，把成本、流动性、约束都搬进环境。代价是：样本效率低、调参成本高、可解释性差。本项目实事求是地认为「RL 不一定比线性 alpha 加权强」，因此并行维护一条监督学习（SL）赛道作为对照（见 §6）；事实证明 SL 赛道的 path5long 当前是综合最佳（H1 +0.02882），但 RL 赛道的 phase15e150kgrandchampion 在 Sharpe 维度（OOS +6.27）仍是不可替代的多样性来源。

English. Conventional quant treats alpha101/gtja191 as ~96 ranking scores to linearly combine. Problems:

• Cross-sectional IC is a low-dimensional statistic. 300 factors × 200 days = 60k samples to estimate 300 weights — overfitting prone, poor cross-year generalization.
• Nonlinear interactions go unmodeled. alpha026 × cyqwinning_ratio under a 30 %-industry-cap can dominate either factor alone; linear models cannot capture this.
• Cost/constraint are absent from the training objective. Traditional models minimize IC residual; realized return is net of 30bp round-trip cost, T+1 inability to reverse, and 30 % single-industry exposure cap. Objective and evaluation diverge.
• State-dependent action. Today's top-50 picks depend on the full market cross-section, not 200 independent scores summed.

RL learns the mapping current factor cross-section + holdings → top-K action directly, moving cost / liquidity / constraints into the environment. Tradeoffs: poor sample efficiency, expensive hyperparameter tuning, weak interpretability. This project honestly does NOT assume RL is always better than linear alpha aggregation — we maintain a parallel supervised-learning (SL) track as control (§6). Today the SL track's path5long is the all-around best (H1 +0.02882), but the RL track's phase15e150kgrandchampion remains an irreplaceable Sharpe-dimension diversity contributor (OOS +6.27).

1.3 项目定位 / Scope and Boundaries

中文. 本项目是：

• ✅ RL 选股算法 + Gymnasium 环境的参考实现
• ✅ A 股微观结构（T+1 / 涨跌停 / ST / 板别 / 行业暴露上限）的工程化封装
• ✅ 多源因子的消费者（按列名前缀识别，输入有什么就用什么）
• ✅ 离线训练 → ONNX 导出 → CPU 推理的端到端流水线
• ✅ 完整的研究决策记录：每个 phase 都记录了 stack 变更 / 量化证据 / 拒绝原因，便于复现和审计

不是：

• ❌ 实盘交易系统（无券商接口、无下单 API）
• ❌ 数据采集工具（不内置任何数据 API key；用户自己 pipeline 写 Parquet）
• ❌ 因子计算库（因子计算可选——aurumq_rl.factors 提供 296 个 polars 实现，但用户可以完全自己算）
• ❌ 高频交易（日频选股，T+1 持仓）

English. This project is:

• ✅ Reference implementation of an RL stock-picking algorithm + Gymnasium environment
• ✅ Engineering encapsulation of A-share microstructure (T+1 / price limits / ST / board / industry cap)
• ✅ A consumer of multi-source factors (prefix-recognized; uses whatever is in your Parquet)
• ✅ End-to-end offline-train → ONNX-export → CPU-infer pipeline
• ✅ Complete research decision log: every phase records stack diff, quantitative evidence, rejection reason — auditable and reproducible

Is NOT:

• ❌ A live trading system (no broker adapter, no order API)
• ❌ A data-ingestion tool (no API keys bundled; you write the Parquet)
• ❌ A factor-computation library (optional — aurumq_rl.factors offers 296 polars implementations, but you can roll your own)
• ❌ High-frequency trading (daily-frequency stock picking under T+1)

1.4 AurumQ 生态 / The AurumQ Ecosystem

中文. 本项目是 AurumQ 平台的一个开源子模块，整个生态分三层：

| 层 | 项目 | 职责 | |---|---|---| | 策略 DSL | AQML | .aqml 声明式策略，可读 / 可验证 / AI 可生成的筛选 + 打分 + 风控规则 | | 因子 + RL | AurumQ-RL (本项目) | 因子工程、A 股约束、PPO/A2C/SAC、ONNX 推理 | | 平台 | AurumQ (闭源) | Web 平台 + REST API + 模拟盘 + 风控引擎 + AI 投研 |

典型工作流：先用 AQML 写策略意图 → 用 AurumQ-RL 把因子列和约束塞进模型训练 → 训出的 .onnx 回到 AurumQ 平台跑模拟盘和实时排程。

English. This project is an open-source submodule of the AurumQ platform; the ecosystem has three layers:

| Layer | Project | Role | |---|---|---| | Strategy DSL | AQML | .aqml declarative strategy — human-readable, validatable, AI-generatable screening + scoring + risk rules | | Factors + RL | AurumQ-RL (this repo) | Factor engineering, A-share constraints, PPO/A2C/SAC, ONNX inference | | Platform | AurumQ (proprietary) | Web platform + REST API + paper trading + risk engine + AI research |

Typical workflow: write strategy intent in AQML → feed factor columns and constraints into AurumQ-RL for training → the resulting .onnx returns to the AurumQ platform for paper trading and real-time scheduling.

---

§2 系统总览

§2 System Overview

2.1 数据契约 / The Data Contract

中文. 项目对外契约就一句话：给我一份 Parquet，我就能训练。Parquet 必含：

• ts_code (str): Tushare 风格代码 XXXXXX.SH/SZ/BJ
• trade_date (date): 交易日
• close (float): 收盘价
• pct_chg (float): 涨跌幅（小数形式，+10% = 0.10，不是 10.0）
• vol (float): 成交量（== 0 视为停牌）
• 因子列（至少包含一组前缀）：alpha / gtja / mf / mfp / hm / hk / inst / mg / cyq / senti / sh / fund / ind / mkt / tech / cmf / zt_*

可选字段（提供则使用，不提供则自动降级）：

• is_st (bool): ST 标记，缺则按全 False 处理
• dayssinceipo (int): 上市以来交易日数（用于新股 60 日保护）
• industry_code (int): 申万一级行业编码（用于 30% 行业暴露上限）
• ishs300 / iszz500 (bool): 是否成分股，按 trade_date 历史变更（支持「2024-01 在 300、2024-06 调出」的时变性）

数据怎么来不是本项目关心的事，三种取数方式：

• 用 scripts/generate_synthetic.py 一键生成 10 MB 合成数据 demo
• 用 scripts/exportfactorpanel.py 从 PostgreSQL 自己的数据仓库抽取（含 SQL 模板，含 HS300/ZZ500 成员标志支持）
• 自己用任何工具（pandas / DuckDB / Spark）造一份满足契约的 Parquet 写入

English. The single contract is: give me a Parquet, I will train. Required columns:

• ts_code (str): Tushare-style code XXXXXX.SH/SZ/BJ
• trade_date (date)
• close (float)
• pct_chg (float): decimal form, +10 % = 0.10, NOT 10.0
• vol (float): 0 means suspended
• Factor columns under at least one prefix: alpha / gtja / mf / mfp / hm / hk / inst / mg / cyq / senti / sh / fund / ind / mkt / tech / cmf / zt_*

Optional fields (auto-fallback when absent): isst, dayssinceipo, industrycode, ishs300, iszz500 (time-varying by trade_date).

Three ways to get data: (1) synthetic demo (generatesynthetic.py), (2) export from your own PG warehouse (exportfactor_panel.py), (3) BYO Parquet from any tool that meets the contract.

2.2 因子前缀识别 / Factor-Prefix Auto-Discovery

中文. data_loader.py 通过列名前缀识别因子组，输入 Parquet 中存在的前缀就被自动加载，不存在的自动跳过。这套设计的核心是：项目本身不知道你给的是哪些因子，只要列名前缀对得上就一律纳入观测。

| 前缀 | 含义 | 推荐维度 | 输入数据要求 | |---|---|---|---| | alpha_* | WorldQuant Alpha101（项目自带 105 个实现，含 6 个自定义补充）| 105 | 日频 OHLCV + amount | | gtja_* | 国泰君安 Alpha191（项目自带 191 个实现）| 191 | 日频 OHLCV + vwap + amount + 基准指数 OHLC | | mf* | Money Flow Velocity — 主力资金流速（4 档累计筹码）| 14 + 6 log 变体 | 4 档资金流分档 | | mfp* | Main Force Position — 主力筹码持仓（与 mf 互补，不要混用）| 12 | 主力净持仓时序表 | | hm_* | Hot Money — 主流游资席位 | 6 | 龙虎榜游资席位日成交明细 | | hk_* | Northbound — 北向资金真实持股 | 4 | 北向持股日表（港股通名单内） | | inst_* | Institutional — 龙虎榜机构净买入 | 3 | 龙虎榜机构席位明细 | | mg_* | Margin — 融资融券 | 3 | 融资融券日表 | | cyq* | Chip Distribution — 筹码分布 | 3 | Tushare cyqperf 表 | | senti_* | Sentiment — 涨停板情绪 | 3 | 涨停板池 + 热度榜 | | sh_* | Shareholder — 股东户数 + 大股东增减持 | 2 | 股东数据 | | fund_* | Fundamentals — 基本面 PE/PB/ROE/营收增速 | 4 | 基本面表 | | ind_* | Industry — 申万行业相对强度 | 2 | 行业指数 | | mkt_* | Market — 大盘 + 拥挤度 | 2 | 指数日表 | | tech_* | Technical — 上游算好的 MA/KDJ/MACD/Bollinger（v1.1 后 30 列）| 30 | 已 z-score 的 OHLCV 技术指标 | | cmf_* | Chaikin Money Flow — 60d/120d 累计资金流 | 2 | 量价资金流派生 | | zt_* | 涨停板 stats — 30d/60d 涨停频次、首板、连板 | 6 | 涨停板池 + 历史 |

总维度灵活：纯 Alpha101 = 105 维 / Alpha101 + GTJA191 = 296 维 / 全部 17 前缀 ≈ 360 维 / 自定义任意子集。

StockPickingConfig.n_factors 决定取前 N 个因子（按字母序），多余的丢弃，不足的报错。

English. data_loader.py recognizes factor groups by column-name prefix; whatever prefixes are present in your Parquet get loaded, whatever is absent gets skipped. The design assumes the project does not know which factors you have — match the prefix, get included as observation.

(See the Chinese table above for the 17-prefix breakdown. Total flexible: pure alpha101 = 105 dims / alpha+gtja = 296 / all 17 prefixes ≈ 360.)

2.3 宇宙过滤 / Universe Filtering

中文. 默认 UniverseFilter.MAINBOARDNON_ST 应用六道 AND 闸门：

• data_ok: 当日有日线数据 AND vol > 0（剔除停牌）
• main_board: 60[0135]\d{3}.SH ∪ 00[0123]\d{3}.SZ（剔除 300/688/689*/4xx/8xx/9xx）
• listed: dayssinceipo ≥ 60（新股 60 日保护）
• notdelisted: stockinfo.delist_date IS NULL
• notst: 按当日 isst == False 逐日过滤（C3:不再按当前名称删除整段历史,避免幸存者偏差;无 is_st 列时才按行级名称回退）
• not_suspended: 当日非停牌（vol > 0 也涵盖此条件）

应用顺序很重要：先 dataok 再 mainboard，避免在停牌日按板别 regex 算返回值时遇到 NaN/Null 行的 regex 失败。

如要自定义：

from aurumqrl.dataloader import UniverseFilter, load_panel
全市场（仅排 ST + 停牌）
panel = loadpanel("data.parquet", universefilter=UniverseFilter.ALLNONST)
只跑沪深 300
panel = loadpanel("data.parquet", universefilter=UniverseFilter.HS300)

English. Default UniverseFilter.MAINBOARDNONST applies six AND-gates: dataok (has bar + vol > 0), mainboard (regex 60[0135]\d{3}.SH ∪ 00[0123]\d{3}.SZ), listed (dayssinceipo ≥ 60), notdelisted, notst (per-date isst = False; C3: no whole-history name-based drop — that is survivorship bias; row-level name is only a fallback when isst is missing), notsuspended. Ordering matters: dataok before mainboard to avoid regex on NaN. Alternative filters: ALLNONST, HS300, ZZ500, or supply your own callable.

2.4 模块架构 / Module Architecture

中文. 仓库布局：

aurumq-rl/
├── src/aurumq_rl/
│   ├── env.py                  # StockPickingEnv (Gymnasium)
│   ├── gpu_env.py              # GPU-vectorized env (Phase 6+)
│   ├── portfolioweightenv.py # 连续权重组合环境（马科维茨扩展）
│   ├── data_loader.py          # Parquet → numpy/cuda 面板（多前缀识别）
│   ├── policies/
│   │   ├── perstockencoder.py    # Deep Sets 风格 per-stock 编码器
│   │   └── shared_policy.py        # 早期 flat MLP（保留参考）
│   ├── rollout/
│   │   ├── gpurolloutbuffer.py   # CUDA-resident rollout buffer
│   │   └── indexonlybuffer.py    # 索引化观测（Phase 9+）
│   ├── inference.py            # ONNX CPU 推理
│   ├── onnx_export.py          # SB3 → ONNX 导出
│   ├── price_limits.py         # 板别动态涨跌停
│   ├── reward_functions.py     # Return / Sharpe / Sortino / Mean-Variance / MainWaveHold
│   ├── mainwavelabels.py     # Phase 22 — MA5/MA10 死叉 + 5d cap 持仓回报标签
│   ├── metrics.py              # 训练指标 JSONL 读写
│   ├── wandb_integration.py    # 实验跟踪（默认离线）
│   ├── sb3_callbacks.py        # SB3 callbacks (WandbMetricsCallback, GpuSamplerCallback, …)
│   ├── gpu_monitor.py          # pynvml-based GPU 采样
│   └── factors/                # polars-native 因子库
│       ├── alpha101/            # WorldQuant Alpha101 (105 因子，10 模块)
│       ├── gtja191/             # 国泰君安 Alpha191 (191 因子，10 batch 文件)
│       ├── ops.py              # 25+ 通用算子（tssum / tscorr / csrank / decay_linear / regbeta / ...)
│       ├── registry.py          # ALPHA101REGISTRY + GTJA191REGISTRY
│       └── _docs.py             # markdown 文档生成器
├── scripts/
│   ├── train.py                # 训练入口 V1（CLI）
│   ├── trainv2.py             # 训练入口 V2（Phase 21+ Dict obs，被 Phase 22 V1 mainwave 回退后保留）
│   ├── infer.py                # 推理入口（CLI）
│   ├── eval_backtest.py        # 测试集 IC / Sharpe / 等权净值曲线
│   ├── evalmainwavev1.py   # Phase 22 V1 mainwave 评估（含 holdreturn / win_rate / drawdown）
│   ├── compare_rewards.py      # 多 reward 类型对比训练
│   ├── exportfactorpanel.py  # PG → Parquet 数据抽取（含 SQL 模板）
│   ├── generate_synthetic.py   # 合成 demo 数据生成
│   ├── ossdownloadresumable.py  # HEAD + Range-based resumable downloader
│   └── reference_data/         # alpha101 / gtja191 reference parquet 重建脚本
├── web/                        # Next.js 16 dashboard（runs/ 可视化）
├── data/
│   ├── README.md               # 数据格式 + 列名约定
│   └── synthetic_demo.parquet  # 10 MB 开箱即跑
├── docs/
│   ├── ARCHITECTURE.md
│   ├── FACTORS.md              # 因子前缀约定 + 列名规范
│   ├── TRAINING_HISTORY.md     # 14 phase 完整训练栈演化（1350 行）
│   ├── factor_library/         # 296 篇因子 markdown 文档
│   ├── phase26/                # Phase 26A-G 实验报告
│   ├── SCHEMA.md
│   ├── TRAINING.md
│   └── INFERENCE.md
├── tests/                      # 1386+ 测试，含因子 parity 与 docs 验证
├── handoffs/                   # 跨机器（4070 ↔ ECS）交接日志
└── examples/
    └── quickstart.py           # 端到端示例

English. Repository layout (see Chinese tree above for full structure). Three key code surfaces:

• src/aurumqrl/env.py and gpuenv.py — the Gymnasium StockPickingEnv and its later GPU-vectorized counterpart
• src/aurumqrl/policies/perstock_encoder.py — the Deep Sets-style permutation-equivariant policy that became the architectural breakthrough in Phase 5
• src/aurumqrl/mainwave_labels.py — Phase 22's MA5/MA10 death-cross + 5d-cap hold-return reward that broke the 5.72 % random baseline for the first time

2.5 硬件与训练资源约束 / Hardware & Training-Resource Constraints

中文. 项目对硬件有两条红线：

• 本地 ECS（8C14G）严禁运行训练。PyTorch 安装即占 ~3 GB RSS，训练时 OOM 必杀；7-worker ProcessPoolExecutor 曾把整台主机 OOM-killed + 重启。训练只能在 GPU 实例（推荐本地 RTX 4070+ 或云端 RTX 4090 / A10 / V100）。
• maxworkers=3 是硬上限（对所有 ProcessPoolExecutor / ThreadPoolExecutor），PostgreSQL sharedbuffers=2GB，内存余量 < 4 GB 时 PG 会被 OOM。

实证训练成本（i7-13700K + RTX 4070 12 GB + 64 GB DDR5）：

| 配置 | 因子数 | 训练步数 | wall time | 备注 | |---|---:|---:|---|---| | smoke (Phase 0) | 16 | 1k | 90s | 合成数据，CPU 即可 | | Phase 1 | 16 | 100k | ~50 min | alpha101 short panel，n_envs=8, fps 333 | | Phase 7 | 64 | 50k | ~7 min | GPU framework smoke, fps 1490 | | Phase 10 | 64 | 1M | ~8h（隔夜）| LayerNorm + dual pooling + bf16, fps 326 | | Phase 14 | 64 | 1M | ~6h（隔夜）| TF32 + unique-date, fps 460 | | Phase 16a (prod) | 343 | 300k | ~5h | 6 seeds 并行外推 | | Phase 22 (main_wave) | 343 | 300k | ~8h | 3-run 隔夜对照 (A/B/C) | | Phase 26F-v3 (prod) | 361 | 300k | ~5h | 3 seeds × 1 config |

English. Two hard rules:

• The local 8-core 14 GB ECS is FORBIDDEN for training. PyTorch installation alone occupies ~3 GB RSS; training will OOM-kill the host. A 7-worker ProcessPoolExecutor once OOM-killed and rebooted the box. Train only on a GPU instance (local RTX 4070+ or cloud RTX 4090 / A10 / V100).
• maxworkers=3 is a hard ceiling for all ProcessPoolExecutor / ThreadPoolExecutor. PostgreSQL sharedbuffers=2GB; PG OOMs when host free RAM < 4 GB.

Measured training cost on i7-13700K + RTX 4070 12 GB + 64 GB DDR5 (see Chinese table above for the 8-config breakdown spanning smoke runs to overnight 1M-step phases).

---

§3 因子库

§3 Factor Library

3.1 自带因子计算引擎 / The Built-in Factor Engine

中文. src/aurumqrl/factors/ 是 polars-native 实现的 296 个量价因子（105 alpha101 + 191 gtja191）。每个因子一篇 markdown 文档在 docs/factorlibrary/，含原文公式 + Polars 实现说明 + 引用。

| Family | 实现 | quality_flag=0 (clean) | =1 (best-effort) | =2 (stub) | |---|---|---|---|---| | alpha101 | 101/101 + 6 自定义 | 88 | 13 | 0 | | gtja191 | 191/191 | 177 | 12 | 2 |

quality_flag 语义：

• 0 (clean)：完整 + 数值稳定 + 跨平台 parity（如 alpha001、gtja159）
• 1 (best-effort)：实现合理但存在已知边界情况（如 alpha017 在窗口=2 时 std=0 触发 NaN，已用 fillnull 处理但未触发 inf）
• 2 (stub)：实现存在但等价于占位（如 gtja115/189 没有可靠数据源对应的 sdpe_ttm 字段）

注册表用法：

import aurumq_rl.factors.alpha101  # registers 107
import aurumq_rl.factors.gtja191   # registers 191
from aurumqrl.factors.registry import ALPHA101REGISTRY, GTJA191_REGISTRY
panel = pl.read_parquet("ohlcv.parquet")  # 需含 OHLCV + vwap + amount df = panel.withcolumns([fn(panel).alias(name) for name, fn in ALPHA101REGISTRY.items()])

通用算子 ops.py 提供 25+ 个跨家族复用的算子：tssum / tscorr / csrank / decaylinear / regbeta / tsargmax / tsargmin / tsmin / tsmax / tsrank / tsdelta / tsdelay / tsstd / tsskew / tskurt / indneutralize / scale / signed_power / sign，所有都是 polars expr-aware，可在懒求值 graph 里组合。

English. src/aurumqrl/factors/ ships 296 polars-native price-volume factors (105 alpha101 + 191 gtja191), one markdown doc per factor under docs/factorlibrary/ with original formula + Polars implementation notes + references. qualityflag ∈ {0 clean, 1 best-effort, 2 stub} per the Chinese table above. The common-operator module ops.py provides 25+ Polars-aware operators (tssum, tscorr, csrank, decaylinear, regbeta, …) that compose lazily.

3.2 A 股私有因子族 / Private A-share Factor Families

中文. 这是项目 11 个 A 股私有因子族，必须从用户自己的数据仓库算好后写进 Parquet。它们对应的中国市场原始数据源在欧美市场没有等价物：

3.2.1 mf_* 主力资金流速 (Money Flow Velocity, 14 + 6 cols)

由用户上游 scripts/computemfpanel.py 输出，14 个基础列 + 6 个 _log 变体。例：

• mfnet{1d,3d,5d,10d,20d,60d} — 主力净流入累计（元）
• mfbuyshare_main — 主力买入占比（SHAP rank 7，Path 4 模型里第 7 重要的特征）
• mfnetaccel520 — 5d / 20d 流入加速度
• mfnet5damountratio — 5d 净流入 / 5d 成交额
• mfnet{1d,3d,5d,10d,20d,60d}log — sign-preserving log1p 变体（2026-05-08 加入），公式 sign(x) · ln(1 + |x|/totalamount)，把原始 std=1.5×10⁸ 的"元"量级压到 std=0.040 的"无量纲"量级

HUGETAIL 事故：原始 mfnet*d 标准差从 1e8 到 1e10，跨 tscode 的尺度差异极大（一只大盘股一天净流入 10 亿元，一只小盘股 1 万元）。Phase 24 在 dataloader.crosssectionzscore 里 z-score 之后仍然有量级，因为 polars 默认 ddof=1 在 3000-stock 截面下分母被极端值拉爆。修复：上游加 _log 变体后训练直接吃压缩量级，跨年泛化恢复。

3.2.2 mfp_* 主力筹码持仓 (Main Force Position, 12 cols)

由 src/aurumq/factors/mainforce.py 输出，与 mf* 互补但完全独立：

• mfpelgbuyratio20d — 超大单买入占比 20 日
• mfplgbuyratio20d — 大单买入占比 20 日
• mfpmainnetcumpct — 主力净流入累计百分位
• mfpmainnetvolatility20d — 主力净流入波动 20 日

Phase 16 关键事故：mfp 前缀曾被静默从 aurumqrl.dataloader.FACTORCOL_PREFIXES 漏掉，12 列输入完全没进训练。修复后 16a 跑出 adj Sharpe +1.593（vs 之前 plateau +1.165）。教训：前缀注册表是 single source of truth，prefix-glob 漏一个前缀等于静默丢一族特征。

3.2.3 cyq_* 筹码分布 (Chip Distribution, 3 cols)

源自 Tushare cyq_perf 表，3 列：

• cyqwinningratio — 当前价位上的获利筹码占比
• cyqconcentration70 — 70% 筹码集中在多少价位区间
• cyqcostdistance — 当前价 vs 平均成本距离

Phase 26 关键事故：cyq 是 A 股独有的因子（券商内部模型 + Tushare 加工），但 Tushare 历史只能回填到 2025-10-20，更早数据是 cyqperf v1.0 用合成方法补的。结果：训练集（含合成回填）cyqcost_distance std = 0.197；OOS 集（全部实采）std = 0.066，3× 压缩。跨截面 z-score 不抹平这种 mid-stream regime shift —— 模型学到的是合成数据的尺度，到真实数据上全错。Phase 26C2 切换到 v1.2 修正 cyq（bulk API 重新回填）后，T-1 lift 从 1.47× 反弹到 2.61×（甚至超过原始 23A 的 2.38×，且收敛快 4 倍）。

3.2.4 hm_* 主流游资席位 (Hot Money Seats, 6 cols)

源自 Tushare toplist + topinst：

• hmnet5d / hmnet20d / hmnet60d — 游资席位累计净买入
• hmrecentactive / hmseatcount30d / hmtop3_concentration — 活跃席位 / 30 日席位数 / 前 3 名集中度

结构性 hard wall：Tushare 龙虎榜数据 ≥ 2023-08-16 才存在，2018-2023.8 的 hm* 永远是 NULL。Phase 20 长 panel 训练时 LightGBM 的 usemissing=True 自动处理，但 RL 训练时观测向量必须填 0（不能 NaN）。修复：dataloader.fillmissingwithzerotrack_mask 同时填 0 + 写 mask，模型可选择性地学到"这一列 mask=1 时无效"。

3.2.5 hk_* 北向资金 (Northbound, 4 cols)

hkholdchg60d 等，SHAP rank 16。结构性 null：港股通名单外的 25% A 股永远是 NULL。同样 fillmissingwithzerotrack_mask 处理。

3.2.6 inst_* 机构持仓 (Institutional, 3 cols)

instappearcount60d / instnet_30d — 龙虎榜机构席位活跃度。

3.2.7 mg_* 融资融券 (Margin, 3 cols)

mgshortchg20d / mgbalance_pct 等。78% A 股有融资融券覆盖。

3.2.8 senti_* 情绪 (Sentiment, 3 cols)

涨停板池 + 同花顺热度榜派生。已知问题：sentithshotpct 99% null，因为同花顺热度榜只追踪 ~3000 只热门股；非热门股在 2024-08-29 之前完全没有数据。Phase 26 数据质量审计把 sentithshotpct 列入 includecolumnsv1_clean.txt 的永久排除清单。

3.2.9 sh_* 股东结构 (Shareholder, 2 cols)

shholdernumchg30d — 股东户数变化。86% null（季度披露，日频面板上稀疏）。

3.2.10 fund_* 基本面 (Fundamentals, 4 cols)

fundpettm / fundpb / fundroettm / fundrevenuegrowth。SHAP rank 11 (pettm) / rank 25 (roe_ttm)。

事故：688* 科创板的 fundpettm 在 2025-08 之前缺失约 600 只 × 每天的 hole，因为 Tushare dailybasic 接口对科创板支持不全。2026-05-08 批次用 bulk dailybasic 回填了所有 600+ × 历史日期。

3.2.11 ind_* 申万行业相对强度 (Industry, 2 cols)

indrelativestrength20d / indrelativestrength60d — 个股 vs 申万一级行业指数收益差。49-57% null（swindexmember 表只覆盖约 3000 只主板成分股）。

3.2.12 mkt_* 大盘 (Market, 2 cols)

mktindexpctchg5d / mktindexvolatility20d — 上证指数派生。Phase 16 关键发现：drop mkt 组反而 +0.428 lift！原因：mkt_ 在主板宇宙下高度共线（所有股票同一个上证指数派生量），模型用它做"今天大盘涨/跌"的偷懒预测，反而损害了选股能力。永久从主流配置移除。

3.2.13 tech / cmf / zt_* 技术指标 (Tech panel, 30 + 2 + 6 cols)

Phase 26 新增。tech 30 列 = MA5/10/20/60 比值 + KDJ 派生 + MACD 派生 + Bollinger 派生 + ATR 派生 + 振幅。cmf 2 列 = Chaikin Money Flow 60d/120d。zt_* 6 列 = 涨停板 30d/60d 频次 + 首板/连板/最长连板。

Phase 24/25 重大事故：24A 把 36 个技术因子直接接在 RL 训练 panel-load 时算（而不是上游 parquet），结果 T-1 hit 从 2.11% 跌到 0.40%（lift 0.45× < 随机 0.89×）。根因：KDJ/振幅近似自 close-only（panel 没有 OHLC），MA-cross / golden-cross 是二值事件标志，进 LayerNorm 后 z-score 把 binary 0/1 拉成极端 outlier，污染了梯度。Phase 26F 修复：把二值事件改为 指数衰减 τ=10d 编码 evt(t) = sum(1[event in last 10d] * exp(-(t-tau)/10))，T-1 hit 从 1.13× 反弹到 2.27×（best 2.41% hit at step 50k）。教训见 §7。

English summary. The 11 private A-share factor families are: mf (Money Flow Velocity, 14 base + 6 sign-preserving log variants — fixes the 1e8-yuan HUGETAIL scale issue); mfp (Main Force Position, 12 cols, independent of mf despite the similar prefix — Phase 16 found mfp was silently missing from FACTORCOLPREFIXES); cyq (chip distribution, 3 cols — Phase 26C2 v1.2 fix recovered T-1 lift from 1.47× to 2.61× by replacing the synthetic-backfill v1.0 with a bulk-API-recomputed v1.2); hm (Dragon-Tiger hot-money seats, 6 cols, structural null pre-2023-08-16); hk (Northbound, 4 cols, structural null for 25 % non-HK-Stock-Connect stocks); inst (institutional, 3); mg (margin trading, 3); senti (sentiment, 3, 99 % null for non-hot stocks); sh (shareholder, 2, 86 % null due to quarterly disclosure); fund (PE/PB/ROE/revenue growth, 4 — SHAP rank 11/25); ind (SW industry relative strength, 2); mkt (market index, 2 — dropped permanently in Phase 16 because removing it gave +0.428 adj Sharpe lift). Phase 26 added tech (30), cmf (2), zt (6) — note that binary event flags must be exp-decay encoded (τ=10d) not raw 0/1 to avoid the −33 % regression seen in Phase 24A.

3.3 因子前缀注册纪律 / Factor-Prefix Registration Discipline

中文. aurumqrl.dataloader.FACTORCOLPREFIXES 是 single source of truth。当前规范清单：

FACTORCOLPREFIXES = (
    "alpha", "mf", "mfp", "hm", "hk", "inst", "mg", "senti",
    "sh", "fund", "ind", "cyq", "gtja_",
    "tech", "cmf", "zt_",
)

漏一个前缀 = 静默丢一族特征 + Phase 16 复现。所有 PR 修改这个 tuple 必须同步更新：

• tests/testdataloader.py:testfactorcolprefixeslockdown —— 字典对比 + 顺序对比
• scripts/exportfactorpanel.py:FACTOR_PREFIXES —— PG 抽取脚本的镜像列表
• docs/FACTORS.md —— 表格 + 列名规范文档

English. aurumqrl.dataloader.FACTORCOLPREFIXES is the single source of truth (17 prefixes today). Missing one = silently lose a factor family = Phase 16 reproduction. Three sites must be kept in sync per PR: the tuple itself, the tests/testdataloader.py lockdown test, and scripts/exportfactorpanel.py.

3.4 SHAP 剪枝实验：345 → 226 / SHAP-Based Pruning

中文. 见 handoffs/2026-05-10-sl-extras/shap_audit/（paris 侧执行）：

• 方法：shap.TreeExplainer 跑在 Path 4 最佳 LightGBM 单模（nl31lr050mdl50seed44），10k 行 VALEFF 数据，按 mean(|SHAP|) 排名。
• Top 5 surprises：gtja159 一骑绝尘（mean|SHAP|=0.001270，gain 28.7%，162 个分裂点）、gtja158、gtja065、gtja140、gtja181。资金流第 7 名 mfbuysharemain，基本面第 11 名 fundpettm，北向第 16 名 hkholdchg_60d。
• 剪枝规则：mean(|SHAP|) < 1e-6 视为零贡献 → 119 个候选 → 保存到 dropcandidates.json。被剪掉的例子：alpha098、gtja054、gtja101、gtja190、alpha002、gtja001、mfnetaccel520、mfnet60d、gtja114、instnet30d。
• Path 6 验证（Bayesian opt 50 trials）：226 列训出来 H1 校准 mean_y = +0.028265 vs Path 4 全 345 列的 +0.028483，Δ = −0.0002（与噪声不可分辨）。bundle 从 40 MB 缩到 32 MB，训练时间 −15%。结论：超参数搜索已 saturate，剪枝是免费午餐。

English. SHAP-based feature pruning ran on the best Path 4 LightGBM single model (nl31lr050mdl50seed44) over 10k VALEFF rows. mean(|SHAP|) < 1e-6 ⇒ drop candidate ⇒ 119 columns saved to dropcandidates.json. Validating on Path 6 (226 cols + 50 Bayesian-opt trials): H1 calibrated meany +0.028265 vs full-345 Path 4 +0.028483 = −0.0002 (indistinguishable from noise). Bundle 40 MB → 32 MB, training −15 %. Lesson: hyperparameter search is saturated; SHAP pruning is a free lunch.

3.5 存储路径与流式处理 / Storage Layout & Streaming

中文. 因子 parquet 按家族 × 年份分片：

| 路径 | 内容 | |---|---| | data/duckdb/factoreval/alphapanel_year=YYYY.parquet | alpha101 全族（109 cols） | | data/duckdb/factoreval/gtjapanel_year=YYYY.parquet | gtja191 全族（193 cols，单年最大 1.37 GB） | | data/duckdb/factoreval/mfpanelyear=YYYY.parquet | mf 22 cols (14 + 6 log + 2 helper) | | data/duckdb/factoreval/cyqpanel/year=YYYY.parquet | canonical cyq 3 cols（v1.2 修正版） | | data/duckdb/factoreval/techpanel/year=YYYY.parquet | tech_ 30 cols | | data/duckdb/factoreval/techeventpanel/year=YYYY.parquet | techevt_ 8 cols（含 exp-decay 编码） | | data/duckdb/quotesenriched/year=YYYY.parquet | 11 个内部 enriched 家族 mfp/hm/hk/inst/mg/senti/sh/fund/ind/mkt/cyq legacy |

流式 concat 红线：14 GB ECS 上禁止跑 pl.concat(diagonalrelaxed) + sinkparquet 拼 10 年面板，会 OOM-killed 主机。正确做法是先 shard，然后 pl.scanparquet([shards], missingcolumns="insert") 流式扫，见 scripts/buildcombinedpanel_safe.py。

English. Factor parquets are sharded by family × year (see Chinese table). Streaming red line: on the 14 GB ECS, pl.concat(diagonalrelaxed) + sinkparquet over 10 years of panel data will OOM-kill the host. Correct pattern: shard first, then pl.scanparquet([shards], missingcolumns="insert") streaming scan. See scripts/buildcombinedpanel_safe.py.

---

§4 训练栈演进史 — Phase 0 → 14

§4 Training-Stack Evolution — Phase 0 to 14

中文. 本节是 GPU 训练栈本身的工程史 —— 从最初 11% GPU 利用率到 1M-step 隔夜训练的所有 stack diff、bug、消融。模型/数据/奖励的实验史在 §5（Phase 15-26）。两个 phase 编号体系独立：本节 Phase 0-14 是「框架建设」，§5 Phase 15-26 是「在已建好的框架上跑模型实验」。

完整的逐 phase 记录在 docs/TRAINING_HISTORY.md（1350 行），本节是其压缩版。

English. This section is the engineering history of the training stack itself — every stack diff, bug, and ablation from the initial 11 % GPU utilization to the 1M-step overnight training. The modeling / data / reward experiments live in §5 (Phases 15–26). The two numbering systems are independent: Phases 0–14 here are "framework construction"; §5 Phases 15–26 are "model experiments on top of the built framework".

The full per-phase record is in docs/TRAINING_HISTORY.md (1350 lines). This section is its compressed version.

Phase 0 — 合成数据流水线打通 / Synthetic Pipeline Bring-up

(~pre-2026-04-29)

Goal. Prove the parquet → env → SB3 PPO → ONNX → backtest → JSON link before touching real data.

Stack. StockPickingEnv (numpy panel, single-process), SB3 default MlpPolicy netarch=[64,64], PPO nenvs=1 batch=64 nsteps=2048 epochs=10, syntheticdemo.parquet (~200 SYN-coded fake stocks).

Bugs surfaced (4 in PR #1 / #2):

| # | Bug | Fix | |---|---|---| | 1 | gymnasium not always installed | lazy import + placeholder raises ImportError | | 2 | ONNX export device mismatch (CUDA policy + CPU dummy_obs) | move policy to CPU before export | | 3 | torch.onnx dynamo=True breaks SB3 Normal distribution | pass dynamo=False | | 4 | JSON serializer can't handle numpy.float32 | WandbMetricsCallback.appendjsonl got default=jsondefault |

Outcome. Pipeline end-to-end. ONNX exported. Backtest IC ≈ 0 (synthetic noise, expected).

Phase 1 — 第一次真实数据训练 / First Real-Data Run (2026-04-29 ~ 30)

Goal. Scale to a real factor panel and a real GPU. Run a 100k-step PPO on alpha101 to see how far naive setup goes.

Data. factorpanelalpha101short20232026.parquet — 105 alpha cols, 5743 stocks, 2023-01..2026-04. After mainboardnonst: 3043 × 800 × 105.

Config. PPO --total-timesteps 100000 --n-envs 8 --vec-normalize --learning-rate 3e-4 --target-kl 0.05 --max-grad-norm 0.3 netarch=[64,64] nfactors=16.

Bugs (8 new): NaN propagating through cross-section z-score (real PG data has NaN cells for suspended / pre-IPO; synthetic didn't); OOS obsdim mismatch (training universe = 3043, OOS = 3052 because some IPOs landed — env's observationspace is fixed at training time; fix = alignpaneltostocklist persisted in metadata.json); PPO approxkl=41,820 on first update (12.5M-param first layer + 48,688-dim obs; fix = targetkl=0.05 + maxgradnorm=0.3 → approxkl=0.028); meanfps=0 in summary (SB3 only emits time/fps on rollout-summary frames; fix = callback computes wall-time fps); metricssummary all-null (callback wrote raw SB3 keys; summarizemetrics() expects canonical schema; fix = raw→canonical mapping at write time); runs/ gitignore unanchored (web/app/runs/ silently dropped; fix = /runs/ anchored at root); alpha045 STHSF parity 44 % mismatch on Windows only (scipy rank-tie-break unstable across versions on 10-stock synthetic; fix = @pytest.mark.xfail(strict=False)); OSS admin AK disabled mid-flight (switch to wepa AK).

Outcome. 100k-step PPO ran clean. fps ~ 333. GPU util ~ 11 %. The 4070 was massively underutilized — wide first-layer of [64,64] was only 3 M params; GPU spent most of its time waiting on CPU rollouts.

Phase 2 — 联合 panel + 网络加宽 + featuregroupweights / Combined Panel + Network Widening (2026-04-30)

Data added. factorpanelcombinedshort2023_2026.parquet — 355 factor cols (105 alpha + 191 gtja + 14 mf + 12 mfp + 5 hk + 4 fund + 3 inst + 3 mg + 3 senti + 2 sh + 2 ind + 2 mkt + 6 hm + 3 cyq), 5643 stocks × 800 dates, 7.7 GB zstd. After main-board filter: 3014 × 600.

Code added. --policy-kwargs-json CLI accepts {"netarch":[2048,1024,512], "activationfn":"relu"}. --feature-group-weights-json accepts e.g. {"alpha":2.0, "mf":0.5}, applied after z-score in applyfeaturegroupweights so VecNormalize doesn't neutralize the weights.

Network widening. netarch=[2048,1024,512]. First-layer params for nfactors=64: 3014 × 64 × 2048 ≈ 395 M. GPU memory 3 GB → 12 GB peak; util peak 11 % → 57 %.

3-way alpha-prefix ablation:

| Run | --feature-group-weights-json | OOS IC | OOS top30 Sharpe | |---|---|---|---| | ablationalphaw05 | {"alpha*":0.5} | (~0) | (~random) | | ablationalphaw10 | {"alpha*":1.0} (no-op baseline) | (~0) | (~random) | | ablationalphaw20 | {"alpha*":2.0} | +0.0006 | −0.807 (random p50 −0.482) |

Decision. Framework works end-to-end. Numbers are noise at 15k steps. Validation passed; promote featuregroupweights as load-bearing CLI feature.

Phase 3 — 三轮 smoke R1/R2/R3 / Three Smoke Rounds (2026-05-01 morning)

Round 1 (R1). --policy-kwargs-json '{"netarch":[1024,512,256]}' + --feature-group-weights '{"alpha":2.0, "mf":1.5, "gtja*":1.0}' at 50k steps, nenvs=12, nsteps=2048, targetkl=0.05. First model with explainedvariance climbing. value_loss from 1.5e-2 → 4.3e-3 over 22 rollouts. OOS IC = +0.011, top30 Sharpe = +1.42 (random p50 −0.48, vs-p50 +1.90). GPU util 35 %, fps 312.

Round 2 (R2). Three changes at once: targetkl 0.05 → 0.10, nenvs 12 → 14, nsteps 2048 → 4096. First attempt OOMed (MemoryError allocating 8.83 GiB rollout buffer); reduced nsteps 4096 → 1024; ran again. OOS top30 Sharpe = +2.16, vs-p50 = +0.74 above R1. Convergence accelerated (explained_var 0.93 at 30k vs R1's 0.81). But: three changes at once = uninterpretable. Could be KL relaxation, env parallelism, or buffer length. Lesson recorded — Phase 3's central rule: one change per round.

Round 3 (R3). Just targetkl 0.10 + learningrate 3e-4 → 1e-4 anneal. OOS top30 Sharpe = +1.89 (R2 = +2.16, R1 = +1.42). Within seed variance of R2; suggests targetkl accounts for most of R2's lift, but nenvs / n_steps cannot be cleanly attributed.

Lesson. OOS Sharpe at 50k steps is noise. Don't pick winners from smokes; pick them from convergence-scale runs (≥ 1M ideally 5M). Burned three rounds arguing about R1/R2/R3 ranking before admitting differences were within seed variance.

Phase 4 — fps 扩展实验 + IPC 天花板 / fps Scaling and IPC Ceiling (2026-05-01 noon)

Goal. Find the n_envs ceiling.

Method. Smoke-grid nenvs ∈ {12, 14, 16, 18, 20} at fixed nsteps=1024.

| n_envs | fps | GPU util | Outcome | |---:|---:|---:|---| | 12 | 314 | 35 % | R1 baseline | | 14 | 366 | 41 % | linear scale | | 16 | 412 | 47 % | linear scale | | 18 | 455 | 53 % | linear scale starting to bend | | 20 | 458 | 56 % | bent — IPC bottleneck |

Realization. Above nenvs=18, adding env doesn't proportionally raise fps because the bottleneck is Python IPC between worker subprocs and the central learner, not GPU compute. And: nenvs=20 OOMed on rollout buffer alloc (14.7 GiB). Back to n_envs=12 for safety.

The IPC ceiling discovery changed the project direction. The classic SB3 setup (numpy panel + subproc envs + CPU rollouts → GPU train) is fundamentally CPU-rollout-bound. The 4070 was sitting at 56 % util at the bottleneck. To break through, we'd need to move rollouts onto the GPU itself. → Phase 5 / 6.

Phase 5 — 四个 realization 推出 GPU-框架重构 / Four Realizations Driving GPU-Framework Redesign (2026-05-01 afternoon)

Goal. Sit down, look at the data, decide whether to keep tuning or fundamentally redesign.

Four realizations:

• Brute-force capacity is a trap. 395 M-param flat MLP needed 12 GB VRAM, fps capped at 458, and didn't beat the per-stock symmetry prior. A symmetry-correct architecture (per-stock encoder, ~50 k params shared across all stocks) has dramatically more inductive bias for stock-picking AND is faster.
• The numpy panel is the wrong abstraction. Re-uploading the same panel to GPU memory every env reset is wasteful. The panel should live in GPU memory throughout training, indexed by env step.
• Observations should be indices, not values. Stock factor vectors don't change across env steps — only which date the env is at changes. Send (dateidx, stockcodes_idx) over the IPC boundary, do the GPU-side gather afterwards.
• VRAM and RAM are different. Confused them once: GPU showed 12 GB used, my Python proc was only 3.8 GB RSS. Always check nvidia-smi --query-gpu=memory.used per-process AND host Get-Process -RSS.

Decision. Design a GPU-vectorized framework: per-stock encoder + CUDA-resident panel + index-only observations + dual pooling head. Implementation in Phase 6/7.

Phase 6 / 7 — GPU-vectorized 框架 + 50k smoke / GPU Framework + 50k Smoke (2026-05-01 evening)

Design (Phase 6, designed afternoon; built in Phase 7).

• gpuenv.py: Gymnasium-compatible env that holds the entire panel as a CUDA tensor panel[ndates, nstocks, nfactors]. Step = advance one date; obs = panel[dateidx] slice + holdings mask. nenvs=16 in a single proc (vectorized across env axis on GPU, no IPC).
• PerStockEncoderPolicy (Deep Sets): apply the SAME MLP to each stock's factor row, then aggregate with mean+max dual pooling. Permutation-equivariant. Net_arch=[64, 32] per-stock, then a [64, 32] head — only ~50 k params total, shared across 3014 stocks.
• Action: scaled tanh on per-stock logits, top-K selection.

Smoke run (Phase 7). 50k steps, nenvs=16, nsteps=1024. fps 1490 (vs Phase 4 ceiling 458 — 3.25× lift). GPU util peak 78 %, mean 62 %. OOS IC +0.014, top30 Sharpe +1.78 — better than Phase 3 R3 +1.89 was within noise of, and convergence reached at 30k vs R3's 50k.

Result. The GPU framework eclipses the previous best at one-third the wall time. The redesign was worth it.

Phase 8 — GPURolloutBuffer (CUDA-resident) (2026-05-01 evening)

Bottleneck. Even with GPU env, rollout buffer was numpy on host. Every PPO update did host→device copies for each minibatch.

Fix. Wrote gpurolloutbuffer.py: holds obs / actions / values / log_probs / rewards / advantages / returns as CUDA tensors. PPO update reads directly from device memory — zero copies.

Outcome. fps 1490 → 1820. GPU util mean 62 % → 71 %. VRAM +1.2 GB (acceptable).

Phase 9 — IndexOnlyRolloutBuffer + n_steps=1024 / Index-Only Observations (2026-05-01 late evening)

Bottleneck identified. Even with cuda-resident buffer, each entry stored obs = (nstocks × nfactors) float32 = 3014 × 64 × 4 = 770 KB per step. At nenvs=16, nsteps=1024: 16 × 1024 × 770 KB = 12.6 GiB rollout buffer. We were paying for storing the entire factor cross-section in memory Nenvs × Nsteps times.

Insight. All observations are slices of the same panel. Store only the date index (4 bytes) and gather on-the-fly during minibatch.

Fix. IndexOnlyRolloutBuffer: stores dateidx[nenvs, nsteps] int32 (~64 KB) + holdingsmask[...] bool (~250 KB). Gather panel[date_idx[batch]] at minibatch read time. Effective batch size keeps the same numerical behavior; memory is 200× smaller.

Outcome. Rollout buffer 12.6 GiB → 0.06 GiB. Freed VRAM lets us raise nenvs=16 → 32 and nsteps=1024 → 2048 within the same 12 GB budget. fps 1820 → 2050. GPU util peak 88 %.

Phase 10 — Optimizer-Orphan Bug + LayerNorm + Dual Pooling (2026-05-01 night)

Bug. value_loss plateaued around 4e-3, never broke below. Suspected vanishing gradients in the value head. Inspected: value head's parameters were not in the optimizer. SB3 default uses one optimizer for both policy and value when they share an extractor; my custom policy split them and only registered policy params.

Fix. Explicit optim.AdamW([{params: policy.parameters()}, {params: valuenet.parameters(), lr: 1e-3}], lr=3e-4). Value head learning unlocked. explainedvariance climbed 0.78 → 0.99 within 50k steps.

Two more additions in this phase:

• LayerNorm after each per-stock MLP layer. Real-data cross-section z-scores still have outliers (after nantonum, a single inf cell at the cell level can still pull mean). LayerNorm gives stable gradients. ~Verified ablation: removing LayerNorm gave value_loss instability across seeds.
• Dual pooling head: aggregate per-stock representations by concat(mean, max) instead of pure mean. Mean captures average market state; max captures the most-extreme stock signal. Worth +0.06 explained_var on the 50k smoke.

Outcome. fps 2050 → 1980 (the LayerNorm + dual pool cost ~3 %, but explained_var=0.99 was worth it). bf16 was attempted but eliminated in Phase 11.

Phase 11 / 12 — bf16 / adaptive target_kl (eliminated) (2026-05-01 night)

Phase 11. Tried torch.autocast(dtype=bfloat16) for matmul+linear. Memory −20 %, fps +12 %. But: approx_kl became unstable, occasionally spiking to 0.3 (vs nominal 0.02). Inspected: tail of policy logits at bf16 dynamic range had quantization noise that compounded in KL computation. Rolled back to fp32.

Phase 12. Tried target_kl=0.10 adaptive (raise to 0.15 if violated 3x in a row). PPO update frequency dropped from every rollout to ~70 %. Total update count similar, value loss similar, IC similar — no measurable benefit. Removed for code simplicity.

Phase 13 — PPO SGD perf-probe / Profiling (2026-05-01 late night)

Goal. Profile a single PPO update with torch.profiler to find any remaining low-hanging fruit.

Findings.

• 53 % of SGD time was in advantage computation (computereturnsand_advantage).
• 22 % in policy log-prob recomputation.
• 11 % in value head forward.
• 14 % in optimizer.step().

Fix. Wrote computereturnsandadvantagevectorized() that uses prefix-sum on CUDA tensors instead of Python for-loop over time steps. 53 % → 7 %. SGD wall time per update −40 %.

fps 1980 → 2580. GPU util mean 78 %.

Phase 14 — TF32 + unique-date + 1M overnight / TF32 + Unique-Date + 1M Overnight (2026-05-02 → 2026-05-03 early)

Goal. Capacity build. Run 1M steps overnight on the post-Phase 13 stack to test stability and final convergence.

Two micro-optimizations going in:

• TF32: torch.backends.cuda.matmul.allowtf32 = True; cudnn.allowtf32 = True. Free ~12 % matmul speedup on RTX 4070 (Ampere) with no measurable accuracy loss.
• Unique-date dedup: same date often appears multiple times in a 16-env × 2048-step rollout because envs reset asynchronously. Detected and gathered only unique date_idx once per minibatch, then duplicated rows. Saves ~30 % gather time.

1M-step overnight run (alpha101+gtja191 296-col on short panel; nenvs=16, nsteps=2048, targetkl=0.05 fixed, lr=3e-4 → 1e-5 linear, entcoef 0.01 → 0):

• Wall time: 5h 47min
• fps mean: 460 (started 326 from cold start, reached 540 at ~200k)
• GPU util mean: 73 %
• Peak VRAM: 9.8 GiB / 12 GiB
• approx_kl trajectory: stable 0.025-0.045, no spikes
• explained_variance: 0.99 from ~150k onwards
• Final OOS top30 Sharpe: +5.83 (vs random p50 +0.62, vs-p50 +5.21)

Outcome. Stack hardened. Phase 14 is the "ready for real experiments" milestone. All Phase 15+ modeling experiments run on this exact stack.

Section recap. From Phase 0's 1k-step smoke (fps 50ish) to Phase 14's 1M-step overnight (fps 460), the framework went through 15 stack diffs, 24 documented bugs, and 5 mid-flight algorithmic redesigns. The biggest single jump was Phase 5/6/7 — moving rollouts onto the GPU 3.25×'d fps, and the symmetry-correct per-stock encoder simultaneously reduced parameter count by 8000× (395 M flat MLP → 50 k Deep Sets) while improving OOS Sharpe.

---

§5 模型实验史 — Phase 15 → 26

§5 Modeling Experiment History — Phase 15 to 26

中文. Phase 14 之后框架不再变了。Phase 15-26 是在固定 stack 上跑模型/数据/奖励实验。每个 phase 都有明确假设、单变量改动、量化决策。

English. After Phase 14 the framework stopped changing. Phases 15–26 are model / data / reward experiments on a frozen stack, each with a clear hypothesis, single-variable change, and quantitative decision.

Phase 15 — RL serving 集成 / RL Serving Integration (2026-05-02 ~ 03)

Goal. Take the best 1M-step model from Phase 14 and integrate it into the AurumQ platform for live serving.

Three SB3 PPO models registered:

| Agent ID | OOS Sharpe | Note | |---|---:|---| | phase15e150kgrand_champion | +6.277 | active production model | | phase15e100kalt_peak | +5.94 | alternative early-peak ckpt | | phase15e2225kcontinuation_peak | +5.83 | continuation of Phase 15e but with extended steps |

Bundle layout models/rl/<id>/:

• policy.zip — SB3 native (kept for PPO.load path)
• factor_schema.json — factor name list, ordering, hash
• metadata.json — trainstartdate, trainenddate, stockcodes, featuregroupweights, factorcount, policy_class
• goldeninference.json + goldenobs.npy + golden_scores.npy — sanity-check pair persisted with the bundle
• checksums.json — sha256 of all artifacts

5 admin-only endpoints under /api/v1/rl/agents/:

• POST /import-bundle — upload, validate schema, persist
• POST /{id}/validate — replay goldenobs through policy, compare to goldenscores
• POST /{id}/archive — soft delete
• POST /{id}/inference — async job (returns job_id)
• GET /api/v1/rl/inference-jobs/{job_id} — poll

Tech-debt addressed. SB3 PPO.load(device='cpu') + LRU(3) policy cache + single-flight lock to prevent concurrent reloads under high traffic.

Frontend. ModelHashBadge, MissingDataAlert, RlAgentsView, RlStockPicksView, useInferenceJob composable.

Phase 16 — 修复 eval bug 重新基线 / Eval Bug Fixes Force New Baseline (2026-05-03, 4h)

Goal. Re-validate Phase 15's "drop mkt_* group helps" finding under three independent bug fixes.

Three bugs:

• Reward double-shift fix. FactorPanelLoader was already encoding the fp-day forward return, but the env AND the importance-permutation pass were re-indexing t+fp ⇒ rewards were fp days too late.
• Sharpe over-annualization. Overlapping fp-day forward returns must be annualized by √(252/fp), not √252. For fp=10, that's an inflation factor of ~3.16×. Phase 15's legacy Sharpe +6.277 includes this inflation; the bug-corrected Sharpe is ~+1.99 on the same data.
• mfp prefix silently missing. 12 mfp cols had been dropped from FACTORCOL_PREFIXES. Training input was 343 cols, not 355.

Models retrained. 16a (drop mkt only), 16b (drop mkt+gtja_), 16c (extend 16b to 450k).

Key findings (under bug-fixed eval):

| Run | adj Sharpe | vs random p50 | IC | Note | |---|---:|---:|---:|---| | 15e legacy (uncorrected) | +6.27 | — | — | annualization artifact | | 16a (drop mkt_) | +1.593 | +0.428 | +0.0143 | new prod candidate | | 16b (drop mkt+gtja) | +1.32 | −0.27 | +0.0109 | gtja_ is load-bearing, contrary to Phase 15 belief | | 16c (16b @ 450k) | +1.36 | −0.23 | +0.0112 | extension didn't help |

Two robustly anti-helpful groups emerged in permutation importance: cyq (−0.142 ± 0.044) and inst (−0.115 ± 0.030). mfp turned out weakly positive (+0.047 ± 0.067), gtja_ load-bearing (+0.160 ± 0.126).

Decision. 16a → production. Phase 15 legacy peaks retired as annualization artifacts. Phase 17 scoped to (a) seed-sweep 16a (b) test "drop cyq+inst" hypothesis.

Phase 17 — 种子鲁棒性 + 条件重要性陷阱 / Seed Robustness + The Conditional-Importance Trap (2026-05-03, 7h)

Goal. Test whether Phase 16's "robust anti-helpful cyq+inst" signal transfers under retrain; measure seed dispersion.

Method. 17A: train drop_mkt+cyq+inst at seed=42, 300k. 17B/C/D: re-run 16a at seeds 1/2/3. 17E: extend 16a (seed=42) to 450k.

Key findings.

• 17A failed catastrophically. adj Sharpe = +0.861, vs-p50 = −0.304 (16a was +0.428). The cyq+inst drop hypothesis is FALSE. The "robust negative permutation importance" signal turned out to be conditional on the trained policy, not causal.
• Seed sweep. 3/4 seeds beat random; mean lift +0.249; range [−0.060, +0.428]. seed=42 sits at the upper edge of the noise band; seed=2 is a lone failure.
• 17E (450k) produced no new peak. Phase 16a's +1.593 at step 224k is confirmed as the seed=42 global maximum.

Critical lesson. Stop chasing factor drops based on permutation importance alone. Permutation importance reflects what the trained policy uses, not what is causally helpful for prediction. To test causality you must retrain after the drop.

Phase 18 — 多种子集成 / Multi-Seed Ensemble (2026-05-03, 6h)

Goal. Convert the seed-sensitivity finding into a deployable mitigation.

Method. Add seeds 4-7 (18A-D) at the unchanged 16a config. Build rank-mean / z-mean / z-median ensembles.

Key findings.

| Run | adj Sharpe | vs random p50 | |---|---:|---:| | seed=42 (16a) | +1.593 | +0.428 | | seed=1 (17B) | +0.97 | +0.115 | | seed=2 (17C) | +0.40 | −0.060 — failure | | seed=3 (17D) | +1.42 | +0.408 | | seed=4 (18A) | +1.917 | +0.752 — single-seed big win | | seed=5 (18B) | +1.84 | +0.596 | | seed=6 (18C) | +0.92 | +0.080 | | seed=7 (18D) | +0.27 | −0.140 — failure |

Across 8 seeds: mean vs-p50 +0.352, median +0.388, win rate 6/8. Top-K Jaccard between seeds was 0.003–0.010 — seeds chose almost completely disjoint baskets. This is exactly why ensembling lifts.

The 6-member rank-mean ensemble (excluding seeds 2 & 7): vs-p50 +0.711 (Δ vs seed=42 alone = +0.283), IC = +0.0278 (1.94× 16a), non-overlap Sharpe +1.938.

Decision. Ensemble passed strong-candidate gate. But: single-OOS-window optimum ≠ production Sharpe. Keep 16a live; advance ensemble as candidate pending fresh post-2026-04 holdout.

Phase 19 — 执行约束 + 多窗口验证 / Execution Constraints + Multi-Window Validation (2026-05-03, 6h)

Goal. Stress-test ens_rankmean6 against realistic costs, T+1 / limit-down filters, multi-window stability, and seed=4's contribution distribution.

Method. Quarter blocks, rolling 60-day windows (step 20), execution simulation at 30/60/100 bps round-trip with limit-down deferral.

Key findings.

• **3/3 q

---

README truncated. View on GitHub