#Data-quality
Showing 60 of 86 repositories tagged #data-quality, ranked by stars
Learn how to develop, deploy and iterate on production-grade ML applications.
π Papers & tech blogs by companies sharing their work on data science & machine learning in production.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Always know what to expect from your data.
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
Refine high-quality datasets and visual AI models
Evidently is ββan open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
The Open Source Feature Store for AI/ML
lakeFS - Data version control for your data lake | Git for data
Learn how to design, develop, deploy and iterate on production-grade ML applications.
Compare tables within or across databases
An open-source data logging library for machine learning models and data pipelines. π Provides visibility into data quality & model performance over time. π‘οΈ Supports privacy-preserving data collection, ensuring safety & robustness. π
Data Contracts engine for the modern data stack. https://www.soda.io
The Virtual Feature Store. Turn your existing data infrastructure into a feature store.
Feathr β A scalable, unified data and AI engineering platform for enterprise
Scalable data pre processing and curation toolkit for LLMs
re_data - fix data issues before your users & CEO would discover them π
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Automatically find issues in image datasets and practice data-centric computer vision.
A curated, but incomplete, list of data-centric AI resources.
π Awesome Data Catalogs and Observability Platforms.
Home of the Open Data Contract Standard (ODCS).
Production-Grade ML System for Automated Unit of Measure Error Detection | 88-92% Accuracy | 94% Autonomy | KNIME Workflow
Local-first ETL/ELT studio: a drag-and-drop visual pipeline designer that compiles to SQL and runs on DuckDB. Tiny desktop app, no servers, git-friendly workspaces.
Compilation of high-profile real-world examples of failed machine learning projects
Know your data betterοΌDatavines is Next-gen Data Observability Platform, support metadata manage and data quality.
Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
Engine for AI/ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
Code review for data in dbt
The toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling.
π Mega Scale Multimodal DataPipeline for SOTA Foundation Models
Open-Source Software, Tutorials, and Research on Data-Centric AI π€
FeatHub - A stream-batch unified feature store for real-time machine learning
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
A SQL transformation engine that type-checks your whole pipeline and catches breaking changes before they run β branches, replay, column-level lineage, compile-time contracts, per-model cost. Adapters: Databricks, Snowflake, BigQuery, DuckDB. Single static Rust binary. Apache 2.0.
Possibly the fastest DataFrame-agnostic quality check library in town.
AtroCore is an enterprise-ready, highly configurable, and scalable open-source Data Management and System Integration Platform. It can be used for Master Data Management (MDM), Product Information Management (PIM), Business Process Management (BPM), and much more.
Great Expectations Airflow operator
Installer for DataKitchen's Open Source Data Observability Products. Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems end to end.
PyTorch dataset debugger for computer vision β pause training, mine live loss signals to surface mislabels, class imbalance & outliers, then curate your image, video & LiDAR data without restarting
Papers about training data quality management for ML models.
Zero-config entity resolution & record linkage. The zero-tuning Fellegi-Sunter path beats hand-tuned Splink head-to-head and scales from a CSV to a verified 100M-row dedupe in 9.2 min. Fuzzy/exact/probabilistic + PPRL + LLM + identity graph. Python + edge-safe TypeScript (WASM), SQL-native in Postgres & DuckDB, MCP/REST + dbt/Airflow.
Home of the Open Data Product Standard (ODPS).
A curated list of awesome resources such as books, tutorials, courses, open-source libraries, exercises, and other materials that support Pythonistas in the making, and Pythonistas migrating into Data Science! π
C++ accelerated data quality toolkit for Python: CSV parsing, cleaning, schema validation, profiling, and pandas integration.
An open-source alternative to Avo. Trackboard keeps your analytics deterministic and safe, while integrating AI to design, implement, test, and triage events seamlessly.
A curated list of awesome open source tools and commercial products for monitoring data quality, monitoring model performance, and profiling data π
Official Monte Carlo toolkit for AI coding agents. Skills and plugins that bring data and agent observability β monitoring, triaging, troubleshooting, health checks β into Claude Code, Cursor, and more.
A GitHub Action that makes it easy to use Great Expectations to validate your data pipelines in your CI workflows.
Swiple enables you to easily observe, understand, validate and improve the quality of your data
A collection of scripts written to complete DQLab Data Analyst Career Track π
A lightweight, declarative PySpark framework for data quality validation β check columns, rows, and entire datasets directly in your Spark pipelines
DataOps Data Quality TestGen is part of DataKitchen's Open Source Data Observability. DataOps TestGen delivers simple, fast data quality test generation and execution by data profiling, Β new dataset hygiene review, AI generation of data quality validation tests, ongoing testing of data refreshes, & continuous anomaly monitoring
Jumbune, an open source BigData APM & Data Quality Management Platform for Data Clouds. Enterprise feature offering is available at http://jumbune.com. More details of open source offering are at,
Soda Spark is a PySpark library that helps you with testing your data in Spark Dataframes
LibrerΓa para la evaluaciΓ³n de calidad de datos, e interacciΓ³n con el portal de datos.gov.co
Data Quality Gate based on AWS
Lightweight DataFrame validation decorators for Pandas, Polars, Modin, and PyArrow. No custom types required.
Udacity Data Engineering Nanodegree Program
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.