English | 中文

Introduction

• ape-dts is a data migration tool enabling any-to-any data transfers.
• It also provides data subscription and data processing.
• It is lightweight, efficient and standalone, requiring no third-party components or extra storage.
• Designed for cloud-native stateless component scenarios.
• In Rust.

Key features

• Supports data migration between various databases, both homogeneous and heterogeneous.
• Supports snapshot and cdc tasks with resume from breakpoint.
• Supports checking and revising data.
• Supports filtering and routing at the database, table, and column levels.
• Implements different parallel algorithms for different sources, targets, and task types to improve performance.
• Allows loading user-defined Lua scripts to modify the data.

Supported task types

| | mysql -> mysql | pg -> pg | mongo -> mongo | redis -> redis | mysql -> kafka | pg -> kafka | mysql -> starrocks | mysql -> clickhouse | mysql -> tidb | pg -> starrocks | pg -> clickhouse | mysql -> doris | pg -> doris | | :----------------------- | :------------- | :------- | :------------- | :------------- | :------------- | :---------- | :----------------- | :------------------ | :------------ | :-------------- | :--------------- | :------------- | :---------- | | Snapshot | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | CDC | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | Data check/revise/review | ✔ | ✔ | ✔ | | | | | | ✔ | | | | | | Structure migration | ✔ | ✔ | | | | | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |

Advanced

Crate features

The dt-main crate provides several optional components which can be enabled via Cargo [features]:

• metrics: Enable Prometheus format task metrics HTTP service interface.

After enabling this feature, you can customize the metrics service with the following configuration:

[metrics]
  # http service host
  http_host=127.0.0.1
  # http service port
  http_port=9090
  # http service worker count
  workers=2
  # prometheus metrics const labels
  labels=yourlabel1:yourvalue1,yourlabel2:yourvalue2

• TBD

Quick starts

CLI

dtscli is a lightweight local CLI for creating and managing ApeCloud DTS tasks. It can generate task configs, start dt-main, list tasks, stream logs, and stop, restart, or delete local task records.

For installation and detailed usage, see dt-cli/README.md.

Tutorial

• prerequisites
• mysql -> mysql
• pg -> pg
• mongo -> mongo
• redis -> redis
• mysql -> starrocks
• mysql -> doris
• mysql -> clickhouse
• mysql -> tidb
• mysql -> kafka -> consumer
• pg -> starrocks
• pg -> doris
• pg -> clickhouse
• pg -> kafka -> consumer
• snapshot + cdc without data loss
• modify data by lua

Run tests

Refer to test docs for details.

More docs

• Configurations

- config details

• Structure tasks

- migration - check - check by Liquibase

• Snapshot tasks

- data migration - data check - data revise - data review - resume at breakpoint - multiple tables in parallel

• CDC tasks

- data sync - heartbeat to source database - two-way data sync - generate sqls from CDC - resume at breakpoint

• Custom consumers

- mysql/pg -> kafka -> consumer

• Data processing

- modify data by lua

• Monitor

- monitor info - position info

• Task templates

- mysql -> mysql - pg -> pg - mongo -> mongo - redis -> redis - mysql/pg -> kafka - mysql -> starrocks - mysql -> doris - mysql -> clickhouse - pg -> starrocks - pg -> doris - pg -> clickhouse

Benchmark

• MySQL -> MySQL, Snapshot

| Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) | | :------- | :--------- | :------------------- | :----------------------------- | :----------------------------- | | ape_dts | 1c2g | 71428 | 8.2% / 5.2% | 211% / 5.1% | | ape_dts | 2c4g | 99403 | 14.0% / 5.2% | 359% / 5.1% | | ape_dts | 4c8g | 126582 | 13.8% / 5.2% | 552% / 5.1% | | debezium | 4c8g | 4051 | 21.5% / 5.2% | 51.2% / 5.1% |

• MySQL -> MySQL, CDC

| Method | Node Specs | RPS(rows per second) | Source MySQL Load (CPU/Memory) | Target MySQL Load (CPU/Memory) | | :------- | :--------- | :------------------- | :----------------------------- | :----------------------------- | | ape_dts | 1c2g | 15002 | 18.8% / 5.2% | 467% / 6.5% | | ape_dts | 2c4g | 24692 | 18.1% / 5.2% | 687% / 6.5% | | ape_dts | 4c8g | 26287 | 18.2% / 5.2% | 685% / 6.5% | | debezium | 4c8g | 2951 | 20.4% / 5.2% | 98% / 6.5% |

• Image size

| ape_dts:2.0.25-alpha.1 | debezium/connect:2.7 | | :--------------------- | :------------------- | | 71.4 MB | 1.38 GB |

• more benchmark details

Contributing

Structure

Modules

• dt-main: program entry
• dt-precheck: pre-check, to minimize interruptions during subsequent data operations by identifying issues early for fast failure
• dt-connector: extractors + sinkers for databases
• dt-pipeline: pipeline to connect extractors and sinkers
• dt-parallelizer: parallel algorithms
• dt-task: create extractors + sinkers + pipelines + parallelizers according to configurations
• dt-common: common utils, basic data structures, metadata management
• dt-tests: integration tests

• related sub module: mysql binlog connector in rust

Build

• Minimum supported Rust version (MSRV)

The current minimum supported Rust version (MSRV) is 1.85.0.

• cargo build
• build images

Checklist

• run cargo clippy --all-targets --all-features --workspace fix all clippy issues.

Community

If you have any questions, you can reach out to us through:

• ApeDTS GitHub Discussions
• ApeDTS Wechat Account with note ape-dts: