#Deduplication
Showing 32 of 32 repositories tagged #deduplication, ranked by stars
Fast, secure, efficient backup program
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Deduplicating archiver with compression and authenticated encryption.
Prometheus Alertmanager
Find duplicate files
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
rustic - fast, encrypted, and deduplicated backups powered by Rust
Simple, configuration-driven backup software for servers and workstations
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
Config driven, easy backup cli for restic.
Scalable data pre processing and curation toolkit for LLMs
Replace complex Borg Backup terminal commands with a beautiful web UI. Create, schedule, and restore backups with just a few clicks.
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
Open source project for data preparation for GenAI applications
Fast Multimodal Semantic Deduplication & Filtering
Data deduplication engine, supporting optional compression and public key encryption.
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
A high-performance tool for video upscaling, interpolation, depth estimation, and more. Available as a CLI and Adobe Extension.
Productivity improvements for Rust ecosystem: warnings are skipped until errors are fixed, LSP-independent Neovim integration, etc.
CLI utility to find near duplicate images and remove all but the best copy.
Context intelligence layer for LLM agents: persistent memory with write-time dedup, sensitivity tagging, conflict detection, and hierarchical decay. ~12ms. No LLM calls. MIT.
Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.
[UNMAINTAINED] A transactional and deduplicating virtual file system
An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.
A maximum-strength name parser for record linkage.
Parse bank statements — CAMT/ISO 20022, PAIN.001, CSV (EN/DE/FR/ES), OFX/QFX, MT940, PDFs (digital + scanned) — into unified Transactions. Deterministic parsers, LLM/vision fallback, Golden Rule + multi-currency + continuity verification, page provenance, dedupe, review, accuracy evals, CLI + REST API. 100% test coverage.
A Rust based deduplication tool
xnew is a fast, low-memory CLI that appends only unique lines to files. Built in Go for large datasets, it streams input efficiently and scales cleanly from thousands to hundreds of millions of lines.
Safe Rust CLI to auto-organize your Downloads and other messy folders.
:bar_chart: python package for performing deduplication using flexible text matching and cleaning in pandas dataframe
Optimal distributed data deduplication and supervised learning pipeline using Apache Spark