Duck Takes Flight

A high-performance data streaming system using DuckDB and Apache Arrow Flight.

Ok, now do it in Rust

Components

• DuckDB Flight Server (duckdbflightserver.py): A server that exposes DuckDB through Arrow Flight protocol
• Data Loader (load_data.py): Continuously generates and loads random data into DuckDB
• Query Client (query_data.py): Executes continuous queries against the loaded data

Setup

• Install dependencies:

pip install duckdb pyarrow

• Start the server:

python duckdbflightserver.py

• Start the data loader:

python load_data.py

• Start the query client:

python query_data.py

Data Schema

The system creates a table called concurrent_test with the following schema:

• batch_id: BIGINT
• timestamp: VARCHAR
• value: DOUBLE
• category: VARCHAR

Features

• Persistent storage using DuckDB
• High-performance data transfer using Arrow Flight
• Continuous data loading and querying
• Memory-efficient batch processing
• Aligned Arrow buffers for optimal performance