Topics

#algorithmic-trading #backtesting-engine #cpp17 #deterministic #educational #hawkes-process #hft #hft-trading #high-frequency-trading #kernal-driver #lock-free #market-making #market-microstructure #market-microstructure-theory #order-book #quantitative-finance #research #simd #ultra-low-latency #zero-copy

Language breakdown

No language data available.

▸ Files click to expand

README

╔═══════════════════════════════════════════════════════════════════════════════╗
║                                                                               ║
║        ███████╗██╗   ██╗██████╗ ███╗   ███╗██╗ ██████╗██████╗  ██████╗        ║
║        ██╔════╝██║   ██║██╔══██╗████╗ ████║██║██╔════╝██╔══██╗██╔═══██╗       ║
║        ███████╗██║   ██║██████╔╝██╔████╔██║██║██║     ██████╔╝██║   ██║       ║
║        ╚════██║██║   ██║██╔══██╗██║╚██╔╝██║██║██║     ██╔══██╗██║   ██║       ║
║        ███████║╚██████╔╝██████╔╝██║ ╚═╝ ██║██║╚██████╗██║  ██║╚██████╔╝       ║
║        ╚══════╝ ╚═════╝ ╚═════╝ ╚═╝     ╚═╝╚═╝ ╚═════╝╚═╝  ╚═╝ ╚═════╝        ║
║                                                                               ║
║            Sub-Microsecond Execution Engine for Algorithmic Trading           ║
╚═══════════════════════════════════════════════════════════════════════════════╝

Ultra-Low Latency Trading System

Deterministic, nanosecond-precise execution engine for quantitative trading research

⚠️ IMPORTANT: Platform Requirements

This system requires x86-64 architecture (Intel/AMD) with AVX-512 support.

✅ Linux x86-64: Full support, < 1μs latency
❌ ARM64/Apple Silicon: Not supported (missing AVX-512, TSX, RDTSC)
❌ Windows: Not recommended (need PREEMPT_RT kernel)

📖 See BUILD_REQUIREMENTS.md for full details

Live Demo • Features • Quick Start • Benchmarks • Architecture • Docs

890ns median latency | Deterministic replay | Lock-free architecture | Research-grade framework

View Interactive Documentation →

[!TIP]

Branching Strategy:

* main: Fundamental execution infrastructure. Primary focus is Latency Minimization (median < 900ns) and system determinism.

* alpha-optimized: Quantitative modeling layer. Primary focus is Alpha & Signal Optimization (multi-kernel Hawkes, SIMD features) while keeping latency within sub-microsecond thresholds.

What Makes This Special?

Built for researchers and systems engineers pushing the boundaries of low-latency execution.

This isn't just another trading bot. It's a complete infrastructure for understanding, measuring, and optimizing execution latency at the hardware level.

The Problem

Traditional trading systems are black boxes with unpredictable latency, non-deterministic behavior, and poor visibility into where microseconds are lost.

The Solution

A transparent, deterministic execution engine that:

Achieves sub-microsecond decision latency (890ns median)
Guarantees bit-identical replay for audit and debugging
Provides nanosecond-level instrumentation at every stage
Uses zero-allocation hot paths and lock-free data structures
Simulates kernel-bypass networking (DPDK-style)
Implements institutional-grade logging and monitoring

Research & Education Only — Not production-ready. Hardware validation (DPDK/FPGA) is simulated. Learn more.

PROPRIETARY LICENSE — Commercial use prohibited. Written permission required. Contact: krishna@krishnabajpai.me

Performance Snapshot

| Component | Median | p99 | p99.9 | |------------------|--------------|-----------|--------------| | Market Data Ingestion | 87 ns | 124 ns | 201 ns | | Signal Extraction (SIMD) | 40 ns | 48 ns | 67 ns | | Hawkes Update (Power-Law) | 150 ns | 189 ns | 234 ns | | End-to-End Decision | 890 ns | 921 ns | 1047 ns | | Order Serialization | 34 ns | 41 ns | 58 ns |

Measurement Precision: ±5ns (TSC jitter) | ±17ns (PTP offset) Test Hardware: Intel Xeon Platinum 8280 @ 2.7GHz, isolated core, RT kernel

Key Features

Performance

Sub-microsecond decision latency
Zero-copy data paths
Lock-free SPSC/MPSC queues
Cache-aligned data structures
SIMD-optimized computations (AVX-512)

Determinism

Bit-identical replay guarantees
Event-driven scheduling
Fixed RNG seeds
Pre-allocated memory pools
TSC-level timestamp precision

Architecture

Kernel-bypass NIC simulation
Multivariate Hawkes process
Avellaneda-Stoikov market making
Adaptive risk management
C++/Rust FFI integration

Observability

Real-time metrics dashboard
Multi-layer audit logging
SHA-256 replay verification
Nanosecond-level tracing
Latency breakdown analysis

New: Jitter Profiler & Stall Detection

"If you can't measure tail latency, you can't fix it."

A built-in nanosecond-resolution profiler detects System Management Interrupts (SMIs) and scheduler preemption by measuring the cycle-gap between busy-wait iterations. This ensures deep visibility into the "Zero-Interruption" guarantee required for p99 latency stability.

Quick Start

Get running in 60 seconds:

# 1. Clone the repository
git clone https://github.com/krish567366/submicro-execution-engine.git
cd submicro-execution-engine
2. Build the system (automatic optimization flags)
./scripts/build_all.sh
3. Run deterministic backtest
scripts/run_backtest.py
4. View results
python3 scripts/verify_latency.py
open dashboard/index.html  # Interactive metrics dashboard

Expected Output (click to expand)

=== Low-Latency Trading System ===
Market data ingestion: 87ns median
Signal extraction: 40ns median  
Hawkes update: 150ns median
Decision latency: 890ns median
--- Cycle: 1000 --- Mid Price: $100.05 Position: 250 Active Quotes: Bid=100.04 Ask=100.06 Spread=2.00 bps Hawkes: Buy=12.456 Sell=11.234 Imbalance=0.052 Regime: NORMAL (multiplier=1.0) Last Cycle Latency: 847 ns (0.847 µs) Determinism verified: SHA-256 match

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Market Data Feed (Simulated)                         │
│                    Kernel-Bypass NIC • Zero-Copy DMA Transfer                │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │ 87ns median
                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    Lock-Free Ring Buffer (SPSC)                              │
│              Power-of-2 Size • Cache-Line Aligned • No Allocations           │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │ O(1) operations
                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                   Order Book Reconstruction                                  │
│            Price-Level Aggregation • L2 Depth Tracking                       │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │
                ┌───────────────┴───────────────┐
                ▼                               ▼
┌─────────────────────────────┐   ┌─────────────────────────────────────────┐
│   Hawkes Process Engine     │   │  Microstructure Features                │
│   • Self/Cross Excitation   │   │  • Deep OFI (10 levels)                │
│   • Power-Law Kernel        │   │  • Order Book Imbalance                │
│   • Buy/Sell Intensity      │   │  • Flow Toxicity (Kyle λ)              │
└──────────────┬──────────────┘   └──────────────┬──────────────────────────┘
               │  150ns median                   │ 40ns (SIMD)
               └───────────────┬─────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                  FPGA DNN Inference (Simulated)                              │
│              12 Features → 8 Hidden → 3 Outputs • 400ns Fixed                │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│              Avellaneda-Stoikov Market Making Strategy                       │
│        HJB Equation • Inventory Skew • Latency-Aware Pricing                 │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │ 890ns E2E
                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                    Risk Control (Pre-Trade + Kill-Switch)                    │
│          Position Limits • Regime Detection • Atomic Checks                  │
└───────────────────────────────┬─────────────────────────────────────────────┘
                                │ 34ns serialization
                                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                       Order Submission                                       │
│                  Pre-Serialized Orders • Zero Copy                           │
└─────────────────────────────────────────────────────────────────────────────┘

See ARCHITECTURE.md for detailed component documentation

Determinism & Reproducibility

One of the system's core guarantees is bit-identical replay capability:

Fixed RNG seeds — Deterministic random number generation Event-driven scheduling — No wall-clock dependencies Pre-allocated memory — No allocator non-determinism Timestamp-ordered events — Consistent processing order

Verification

# Run backtest
scripts/run_backtest.py
Verify deterministic replay
cd logs
sha256sum -c MANIFEST.sha256
strategy_trace.log: OK
order_flow.log: OK
latency_metrics.log: OK

Complete Documentation

| Document | Description | |----------|-------------| | ARCHITECTURE.md | Order path, cache layout, thread model | | BENCHMARK_GUIDE.md | Latency measurement methodology | | LATENCY_BUDGET.md | Component-level breakdown | | INSTITUTIONALLOGGING_COMPARISON.md | Audit-grade logging | | PRODUCTION_READINESS.md | Deployment considerations | | ENGAGEMENTS.md | Commercial & Research support | | logs/README.md | Multi-layer timestamp verification |

Contributing

We welcome contributions! Here's how to get started:

Report a Bug

Open an issue with:

System configuration (CPU, OS, compiler)
Reproducible example
Expected vs actual behavior
Relevant logs

Propose a Feature

Check existing issues/PRs
Open an issue describing the feature
Discuss implementation approach
Submit a PR with tests

Submit a Pull Request

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Make your changes with tests
Ensure ctest and cargo test pass
Commit with clear messages
Push and open a PR

Development Guidelines

Code style: Follow existing patterns (run clang-format)
Tests: Add tests for new features
Benchmarks: Measure latency impact
Documentation: Update relevant markdown files

🌟 Star History

Star History

Commercial & Research Engagements

This repository serves as a framework for research and evaluation. For institutional requirements requiring production-grade performance or custom research collaborations, we offer several engagement models:

Institutional Implementation: Custom DPDK/RDMA kernel-bypass and FPGA HLS/RTL acceleration.
Performance Engineering: Architecture reviews, latency bottleneck audits, and system tuning.
Research Collaboration: Market microstructure modeling and low-latency ML inference.
Strategy Optimization: Migration of Python/Julia models to AVX-512 optimized C++.

See ENGAGEMENTS.md for more details.

Contact: krishna@krishnabajpai.me

Notice: Pricing is not disclosed publicly. All engagements are scoped and quoted based on project complexity and hardware requirements.

Kernel-bypass NICs

True kernel bypass depends on:

Specific NIC hardware (NVIDIA/Mellanox, Intel, Solarflare/Xilinx, FPGA NICs)
Driver stacks (DPDK, RDMA verbs, Onload, custom DMA paths)
NIC firmware and queue configuration

There is no realistic way to ship a single, generic kernel-bypass implementation that works across all vendors and drivers. The only viable approach is to define a clean abstraction and allow users to plug in vendor-specific implementations.

FPGA Inference

The same constraint applies to FPGA-based inference:

FPGA implementations depend on the exact FPGA model, vendor toolchain (Xilinx Vitis/Vivado, Intel Quartus), and PCIe/DMA configuration.
The inference pipeline (HLS, RTL, data layout, batching, clocking) is tightly coupled to the target hardware.
Bitstreams are not portable across vendors or across FPGA families.

In this repository:

FPGA inference is represented as an architectural interface / placeholder.
Software emulation is used where hardware is not available.
For specific hardware targets, custom HLS/RTL integration can be implemented.

📖 Academic References

Click to expand bibliography

Hawkes Processes

Hawkes, A. G. (1971). "Specular Point Processes" Biometrika
Bacry, E., et al. (2015). "Hawkes Processes in Finance" Market Microstructure and Liquidity

Market Making

Avellaneda, M., & Stoikov, S. (2008). "High-frequency trading in a limit order book" Quantitative Finance
Guéant, O., et al. (2013). "Dealing with the inventory risk" Mathematics and Financial Economics

Market Microstructure

Cartea, Á., et al. (2015). "Algorithmic and High-Frequency Trading" Cambridge University Press
Lehalle, C.-A., & Laruelle, S. (2018). "Market Microstructure in Practice" World Scientific
Easley, D., et al. (2012). "Flow Toxicity and Liquidity in a High-Frequency World" Review of Financial Studies

System Design

Nygren, E. (2015). "Linux Kernel Development for Real-Time Systems" O'Reilly
Gregg, B. (2013). "Systems Performance: Enterprise and the Cloud" Prentice Hall

Important Disclaimers

RESEARCH & EDUCATION ONLY

This system is NOT:

Production-ready trading software
Connected to any exchanges
Financial advice or recommendation
Guaranteed to be profitable

This system IS:

A research framework
An educational tool
A latency benchmarking platform
A deterministic execution skeleton

Real production HFT requires:

Hardware FPGA acceleration (Xilinx, Altera)
True kernel-bypass (DPDK, Solarflare OpenOnload)
Exchange connectivity (FIX, proprietary protocols)
Compliance systems (kill-switches, position limits)
Risk management infrastructure
Extensive testing and regulatory approval
Vendor-specific hardware integration (DPDK/FPGA)

Legal: No warranty. Use at your own risk. See LICENSE for details.

Contact & Community

Questions? Ideas? Collaboration?

❤️ Support This Project

If this research and codebase helps with your work, please consider sponsoring:

Your support helps fund:

Hardware: Dedicated access to DPDK-enabled NICs and FPGA accelerator cards for validation
Funding: Production-grade bitstreams and vendor-specific implementations
Research: Advanced algorithms and optimization techniques
Documentation: Comprehensive guides and institutional-grade whitepapers
Open Source: Keeping the core research framework freely available

Related Projects

DPDK — Data Plane Development Kit
Solarflare OpenOnload — Kernel-bypass networking
Folly — Facebook's lock-free structures
QuantLib — Quantitative finance library

Built for Speed. Designed for Reliability. Optimized for Discovery.

If you find this useful, please star the repository

_{Made with care by quantitative systems engineers}

Trading • Low-Latency • Research • Open Source

License

PROPRIETARY LICENSE - See LICENSE file for details

IMPORTANT: This software is for academic research and educational purposes only. Commercial use is strictly prohibited. Written permission required for any use beyond personal learning and non-commercial experimentation.

To request permission: Contact krishna@krishnabajpai.me

🔗 More in this category