vowl

vowl (vee-owl 🦉) is a validation engine for Open Data Contract Standard (ODCS) data contracts. Define your validation rules once in a declarative YAML contract and get rich, actionable reports on your data's quality.

🏆 Official ODCS Vendor: vowl is actively maintained and listed on the official ODCS vendors list as a natively compatible tool.

Table of Contents

Part 1 · Getting Started

• Features
• Installation
• Validate in 3 lines

• Data Contracts
• Auto-Generated Checks
• Library Metrics (type: library)
• Format Checks
• Validation Results
• Architecture

• Common sources — Local DataFrame · PySpark · Ibis (20+ backends) · Concurrent Checks
• Filtering & cross-source — Filter Conditions · Multi-Source · Compatibility Mode
• Advanced & extending — Servers in Contract · Custom Adapters · Remote Contracts (Git/S3)

• Roadmap
• Contributing
• License

Part 1 · Getting Started

Features

• Extensible Check Engine: Ships with a SQL check engine out of the box, with the architecture designed to support custom check types beyond SQL.
• Auto-Generated Rules: Checks are automatically derived from contract metadata (logicalType, logicalTypeOptions, required, unique, primaryKey) and library metrics (nullValues, missingValues, invalidValues, duplicateValues, rowCount).
• Any DataFrame, Any Backend: Load any Narwhals-compatible DataFrame type (pandas, Polars, PySpark, etc.) or connect to 20+ backends via Ibis. SQL dialect translation is handled by SQLGlot.
• Server-Side Execution: SQL checks run server-side through Ibis without materialising tables on the client.
• Multi-Source Validation: Validate across tables in different source systems with cross-database joins.
• Declarative ODCS Contracts: Define validation rules in YAML following the Open Data Contract Standard.
• Flexible Filtering: Filter conditions with wildcard pattern matching, ideal for incremental validation of new data.
• Rich Reporting: Detailed summaries, row-level failure analysis, saveable reports, and a chainable ValidationResult API.
• No Silent Gaps: Unimplemented or unrecognised checks surface as ERROR, not quietly skipped, so nothing slips through the cracks.

Installation

pip install vowl

Or install from source:

pip install git+https://github.com/govtech-data-practice/vowl.git

Optional extras are available: vowl[spark], vowl[all]. vowl[spark] supports both classic Spark and Spark Connect (pyspark[connect], requires pyspark >= 3.4). On runtimes pinned to pyspark 3.0–3.3, use vowl[spark-classic] for classic Spark only (no Spark Connect). For local development, testing, and release workflow, see CONTRIBUTING.md.

Validate in 3 lines

import pandas as pd  # or any Narwhals-compatible DataFrame
from vowl import validate_data
df = pd.readcsv("tests/hdbresale/HDBResaleWithErrors.csv") result = validatedata("tests/hdbresale/hdbresalesimple.yaml", df=df) result.displayfullreport()

=== Data Quality Validation Results ===
   Contract Version:      v3.1.0
   Contract ID:           c11443ee-542f-4442-b28d-2d224342be37
   Schemas:               hdbresaleprices
OVERALL DATA QUALITY    Overall:      Checks Pass Rate:       17 / 20 (85.0%)
hdbresaleprices:      Overall:        Checks Pass Rate:       17 / 20 (85.0%)        ERRORED Checks:         0      Single Table:        Checks Pass Rate:       17 / 20 (85.0%)        ERRORED Checks:         0        Unique Passed Rows:     201,863 / 201,879 (99.9%)      Multi Table:        Checks Pass Rate:       0 / 0 (N/A)        ERRORED Checks:         0        Non-unique Failed Rows: 0
CHECK RESULTS +-----------------------------------------+---------------------------------------+-------------------+--------+---------------+---------------+--------+----------------+ | checkid                                | Target                                | tablesin_query   | status | operator      | expected      | actual | execution time | +-----------------------------------------+---------------------------------------+-------------------+--------+---------------+---------------+--------+----------------+ | Month                                   | hdbresaleprices.month               | hdbresaleprices | FAILED | mustBe        | 0             | 2      | 17.84 ms       | | Year                                    | hdbresaleprices.leasecommencedate | hdbresaleprices | FAILED | mustBe        | 0             | 2      | 26.09 ms       | | floorareamustbelessthan200        | hdbresaleprices.floorareasqm      | hdbresaleprices | FAILED | mustBe        | 0             | 12     | 13.58 ms       | +-----------------------------------------+---------------------------------------+-------------------+--------+---------------+---------------+--------+----------------+ | AddressBlockHouseNumber                 | hdbresaleprices.block               | hdbresaleprices | PASSED | mustBe        | 0             | 0      | 17.26 ms       | | blockcolumnexistscheck               | hdbresaleprices.block               | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.56 ms        | | flatmodelcolumnexistscheck          | hdbresaleprices.flatmodel          | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 31.60 ms       | | flattypecolumnexistscheck           | hdbresaleprices.flattype           | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.45 ms        | | flattypeinvalidValues                 | hdbresaleprices.flattype           | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 17.79 ms       | | floorareasqmcolumnexistscheck      | hdbresaleprices.floorareasqm      | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.90 ms        | | hdbresalepricesrowCount              | hdbresaleprices                     | hdbresale_prices | PASSED | mustBeBetween | [0, 30000000] | 201879 | 6.32 ms        | | leasecommencedatecolumnexistscheck | hdbresaleprices.leasecommencedate | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.24 ms        | | monthcolumnexistscheck               | hdbresaleprices.month               | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.43 ms        | | monthlogicaltypecheck                | hdbresaleprices.month               | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 8.16 ms        | | remainingleasecolumnexistscheck     | hdbresaleprices.remaininglease     | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 3.81 ms        | | resalepricecolumnexistscheck        | hdbresaleprices.resaleprice        | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.33 ms        | | resalepricemustnotexceed2m         | hdbresaleprices.resaleprice        | hdbresaleprices | PASSED | mustBe        | 0             | 0      | 18.07 ms       | | storeyrangecolumnexistscheck        | hdbresaleprices.storeyrange        | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 3.94 ms        | | streetnamecolumnexistscheck         | hdbresaleprices.streetname         | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.83 ms        | | towncolumnexistscheck                | hdbresaleprices.town                | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 4.53 ms        | | townnullValues                         | hdbresaleprices.town                | hdbresale_prices | PASSED | mustBe        | 0             | 0      | 10.13 ms       | +-----------------------------------------+---------------------------------------+-------------------+--------+---------------+---------------+--------+----------------+ Total Execution:       210.88 ms
=== Failed Checks and Rows (up to 5 row(s) per failed check) ===
hdbresaleprices     Single checks
[Month]         Operator:   mustBe         Expected:   0         Actual:     2         Target:   hdbresaleprices.month         Details:  Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM         Rule:     SELECT COUNT(*) FROM "hdbresaleprices" WHERE NOT REGEXPMATCHES(TRYCAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')         Rows shown: 2 of 2 +----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+ | month    | town   | flattype | block | streetname  | storeyrange | floorareasqm | flatmodel    | leasecommencedate | remaininglease    | resaleprice | +----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+ | 2017-jan | BEDOK  | 5 ROOM    | 21    | CHAI CHEE RD | 07 TO 09     | 130.0          | Adjoined flat | 1972                | 54 years 06 months | 530000.0     | | 2017-jan | BISHAN | 3 ROOM    | 105   | BISHAN ST 12 | 04 TO 06     | 4.0            | Simplified    | 1985                | 67 years 11 months | 395000.0     | +----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+
[floorareamustbelessthan200]         Operator:   mustBe         Expected:   0         Actual:     12         Target:   hdbresaleprices.floorareasqm         Details:  Validates that floor area must be less than 200         Rule:     SELECT COUNT(*) FROM "hdbresaleprices" WHERE TRYCAST(floorarea_sqm AS BIGINT) >= 200         Rows shown: 5 of 12 +---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+ | month   | town            | flattype | block | streetname         | storeyrange | floorareasqm | flatmodel         | leasecommencedate | remaininglease    | resaleprice | +---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+ | 2017-06 | KALLANG/WHAMPOA | 3 ROOM    | 38    | JLN BAHAGIA         | 01 TO 03     | 215.0          | Terrace            | 1972                | 54 years 01 month  | 830000.0     | | 2017-09 | CHOA CHU KANG   | EXECUTIVE | 641   | CHOA CHU KANG ST 64 | 16 TO 18     | 215.0          | Premium Maisonette | 1998                | 79 years 04 months | 888000.0     | | 2017-12 | KALLANG/WHAMPOA | 3 ROOM    | 65    | JLN MA'MOR          | 01 TO 03     | 249.0          | Terrace            | 1972                | 53 years 07 months | 1053888.0    | | 2018-01 | CHOA CHU KANG   | EXECUTIVE | 639   | CHOA CHU KANG ST 64 | 10 TO 12     | 215.0          | Premium Maisonette | 1998                | 79 years           | 900000.0     | | 2018-09 | KALLANG/WHAMPOA | 3 ROOM    | 41    | JLN BAHAGIA         | 01 TO 03     | 237.0          | Terrace            | 1972                | 52 years 10 months | 1185000.0    | +---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+
[Year]         Operator:   mustBe         Expected:   0         Actual:     2         Target:   hdbresaleprices.leasecommencedate         Details:  Based on ISO 8601, assumed to be in UTC +8 | YYYY         Rule:     SELECT COUNT(*) FROM "hdbresaleprices" WHERE NOT REGEXPMATCHES(TRYCAST(leasecommencedate AS TEXT), '^[0-9]{4}$')         Rows shown: 2 of 2 +---------+------------+-----------+-------+------------------+--------------+----------------+----------------+---------------------+--------------------+--------------+ | month   | town       | flattype | block | streetname      | storeyrange | floorareasqm | flatmodel     | leasecommencedate | remaininglease    | resaleprice | +---------+------------+-----------+-------+------------------+--------------+----------------+----------------+---------------------+--------------------+--------------+ | 2017-01 | ANG MO KIO | 3 ROOM    | 219   | ANG MO KIO AVE 1 | 07 TO 09     | 67.0           | New Generation | 1977.0              | 59 years 06 months | 297000.0     | | 2017-01 | ANG MO KIO | 3 ROOM    | 211   | ANG MO KIO AVE 3 | 01 TO 03     | 67.0           | New Generation | abc                 | 59 years 03 months | 325000.0     | +---------+------------+-----------+-------+------------------+--------------+----------------+----------------+---------------------+--------------------+--------------+

Next, Part 2 · Core Concepts explains what a contract looks like and what you get back. If you'd rather see runnable code, jump to Part 3 · Usage Patterns for PySpark, Ibis connections, multi-source validation, and more.

Part 2 · Core Concepts

This section walks through the four ideas you need: how you declare rules (Data Contracts), what vowl generates for you (Auto-Generated Checks), the declarative metrics you can add (Library Metrics and Format Checks), and what you get back (Validation Results). It closes with the Architecture.

Data Contracts

Instead of writing validation logic in Python, you declare it in a YAML file following the Open Data Contract Standard (ODCS). This separates your rules from your code, making them easier to manage, version, and share.

Example hdbresalesimple.yaml (trimmed for readability):

kind: DataContract
apiVersion: v3.1.0
version: 1.0.0
id: c11443ee-542f-4442-b28d-2d224342be37
status: draft
name: HDB Resale Flat Prices
schema:
  - name: hdbresaleprices # This becomes the table name in your SQL queries
    properties:
      # --- SQL Check: regex-based format validation ---
      - name: month
        logicalType: string
        quality:
          - type: sql
            name: Month
            description: Based on ISO 8601, assumed to be in UTC +8 | YYYY-MM
            mustBe: 0
            query: |-
              SELECT COUNT(*)
              FROM "hdbresaleprices"
              WHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';
            dimension: conformity
# --- Library Metric: null-value check ---       - name: town         quality:           - type: library             metric: nullValues             mustBe: 0             dimension: completeness
# --- Library Metric: valid-value list ---       - name: flat_type         quality:           - type: library             metric: invalidValues             mustBe: 0             dimension: conformity             arguments:               validValues:                 - 1 ROOM                 - 2 ROOM                 - 3 ROOM                 - 4 ROOM                 - 5 ROOM                 - EXECUTIVE                 - MULTI-GENERATION
# --- SQL Check: business rule ---       - name: floorareasqm         quality:           - name: floorareamustbelessthan200             description: Validates that floor area must be less than 200             type: sql             dimension: consistency             query: SELECT COUNT(*) FROM "hdbresaleprices" WHERE floorareasqm >= 200             mustBe: 0
# --- SQL Check: resale price cap ---       - name: resale_price         quality:           - name: resalepricemustnotexceed_2m             description: Resale price must not be more than 2 million SGD             type: sql             dimension: conformity             query: >-               SELECT COUNT(*) FROM "hdbresaleprices" WHERE resale_price > 2000000             mustBe: 0
# --- Table-Level Library Metric ---     quality:       - type: library         metric: rowCount         mustBeBetween:           - 0           - 30000000

Auto-Generated Checks

You don't have to write every check by hand. When a contract is loaded, vowl automatically derives checks from your column metadata — so simply declaring logicalType, required: true, unique: true, and similar gives you validation for free. These auto-generated checks run before any explicit quality checks you've authored.

The check types currently generated:

| Generated from | What vowl validates | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | name | Column declared in the contract exists in the source table | | logicalType | Values can be cast to the declared SQL type for integer, number, boolean, date, timestamp, and time | | logicalTypeOptions.minLength | String length is at least the configured minimum | | logicalTypeOptions.maxLength | String length does not exceed the configured maximum | | logicalTypeOptions.pattern | String values match the configured regex pattern | | logicalTypeOptions.minimum | Value is greater than or equal to the configured minimum | | logicalTypeOptions.maximum | Value is less than or equal to the configured maximum | | logicalTypeOptions.exclusiveMinimum | Value is strictly greater than the configured minimum | | logicalTypeOptions.exclusiveMaximum | Value is strictly less than the configured maximum | | logicalTypeOptions.multipleOf | Value is a multiple of the configured number | | logicalTypeOptions.format | Value satisfies the declared format (see Format Checks) | | required: true | Column contains no NULL values | | unique: true | Non-null values are unique | | primaryKey: true | Values are both unique and non-null |

For example, a property like this:

- name: block
    logicalType: string
    logicalTypeOptions:
        maxLength: 10
    required: true

produces three generated checks: a column-exists check, a maxLength option check, and a required (no-NULL) check. Because string does not currently generate a SQL cast-based type check, the logicalType entry above contributes metadata for option checks rather than a standalone type-validation query. If you use integer, number, boolean, date, timestamp, or time, vowl also generates a logicalType SQL check automatically. You only need to define extra quality entries when you want custom business rules beyond the contract metadata.

Library Metrics (type: library)

Instead of writing SQL by hand, you can declare common data quality metrics using type: library in your quality blocks. vowl auto-generates the appropriate SQL at runtime.

Column-level metrics (under a property's quality):

| metric | What it checks | Arguments | | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------ | | nullValues | Count of NULL values in the column | - | | missingValues | Count of values matching a configurable missing-values list | arguments.missingValues: list of sentinel values (use null for SQL NULL) | | invalidValues | Count of values that fail valid-value or pattern criteria | arguments.validValues: allowed values list and/or arguments.pattern: regex | | duplicateValues | Count of duplicate non-NULL values in the column | - |

Table-level metrics (under a schema's quality):

| metric | What it checks | Arguments | | ----------------- | ------------------------------------------------ | ----------------------------------------------------- | | rowCount | Total number of rows in the table | - | | duplicateValues | Count of duplicate rows across specified columns | arguments.properties: list of column names to check |

All library metrics support unit: "percent" to return the result as a percentage of total rows instead of an absolute count. They also accept any of the standard check operators (mustBe, mustBeGreaterThan, etc.).

Example:

properties:
  - name: town
    quality:
      - type: library
        metric: nullValues
        mustBe: 0
        dimension: completeness
- name: flat_type     quality:       - type: library         metric: invalidValues         mustBe: 0         dimension: conformity         arguments:           validValues:             - 3 ROOM             - 4 ROOM             - 5 ROOM             - EXECUTIVE
quality:   - type: library     metric: rowCount     mustBeGreaterThan: 0     dimension: completeness
- type: library     metric: duplicateValues     mustBe: 0     dimension: uniqueness     arguments:       properties:         - month         - block         - street_name

Format Checks

The logicalTypeOptions.format key validates that column values conform to a declared format. The check generated depends on the column's logicalType. In short:

• Integer formats (i8…u64) — range-check a fixed-width integer type.
• String formats (uuid, email, ipv4, ipv6, hostname, uri) — match a built-in regex.
• Date/timestamp/time formats — a JDK DateTimeFormatter pattern (e.g. yyyy-MM-dd) is converted to a regex and matched against string-cast values.
• Number formats (f32, f64) — recognised but metadata-only (no check).

- name: age
  logicalType: integer
  logicalTypeOptions:
    format: u8 # 0 – 255
name: request_id
  logicalType: string
  logicalTypeOptions:
    format: uuid
name: created_at
  logicalType: timestamp
  logicalTypeOptions:
    format: "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"

Validation Results

The validate_data function returns a powerful ValidationResult object that provides multiple ways to interact with your validation results.

Core Methods

| Method/Property | What It Does | Returns | | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | print_summary() | Prints high-level statistics (pass/fail counts, success rate, performance) | self (chainable) | | showfailedrows(maxrows=5) | Displays sample of failed rows in console. Use maxrows=-1 for all rows. | self (chainable) | | displayfullreport(max_rows=5) | Prints summary + shows failed rows (convenience method) | self (chainable) | | save(outputdir=".", prefix="vowlresults", outputmode=None, checkinfo=None) | Saves enhanced CSV and summary JSON to disk. outputmode can be "failedrows", "annotated", or "both"; checkinfo shapes the annotated checkinfo column ("names"/"summary"/"full") | self (chainable) | | getoutputdfs(checks=None) | Returns per-check failed rows as {check_id: DataFrame} | Dict[str, DataFrame] | | getconsolidatedoutput_dfs(checks=None) | Deduplicates failed rows across checks, grouped by table | Dict[str, DataFrame] | | getannotatedoutput(checks=None, checkinfo=None) | Returns full in-scope tables with a checkinfo column (JSON array of objects) marking failed rows | Dict[str, Dict[str, DataFrame]] | | .passed (property) | Boolean indicating if all checks passed | True/False |

Annotated Output

getannotatedoutput() returns the full in-scope table with a checkinfo column that marks which rows failed which checks. Passing rows have null in the checkinfo column. This is useful when you need to see failures in the context of the full dataset rather than just the isolated failed rows.

It returns a nested dict with two reserved keys:

• "annotated" — a {schema: table} dict where each table is your full in-scope data plus a checkinfo column. Every original row is present; checkinfo is null for rows that passed everything and holds a JSON array of objects describing the failing check(s) otherwise.
• "residues" — failed rows for checks that cannot be merged onto a single table (cross-table, aggregation, and column-subset checks). Single-table contracts produce none. Residues are per-check (one entry per non-mergeable check, keyed "<schema>::<checkname>") and carry the same checkinfo column as the annotated tables (a single-element JSON array, shaped by the same preset) plus tablesinquery — so everything getannotatedoutput() returns is read the same way.

• "names" — [{"check_name": ...}, ...] (just the failing check name(s)).
• "summary" — [{"check_name", "dimension", "tags", "target"}, ...].
• "full" — [<full checkdefinition> + "checkname" + "target", ...].

import json
result = validate_data("contract.yaml", df=df) output = result.getannotatedoutput(check_info="summary")
annotated = output["annotated"]["hdbresaleprices"].to_pandas() 
columns: <original columns> + check_info
 
output["residues"] — cross-table or non-mergeable failures (empty for single-table contracts)

check_info is a JSON string; parse it to read each failing check's dimension/tags/target.
failing = annotated[annotated["check_info"].notna()].iloc[0]
print(json.loads(failing["check_info"]))
[{"check_name": "Year", "dimension": "conformity",
  "tags": ["SG-DRM v5.0"], "target": "hdbresaleprices.leasecommencedate"}]

Annotated Table

# Sort so flagged rows surface first; passing rows (check_info = null) sink to the bottom.
flaggedfirst = annotated.sortvalues("checkinfo", naposition="last").reset_index(drop=True)
flaggedfirst[["month", "town", "block", "floorarea_sqm",                "leasecommencedate", "check_info"]]

clean = annotated[annotated["checkinfo"].isna()].drop(columns=["checkinfo"])
201,861 clean rows of 201,879, ready for downstream use — no join back to the source needed

When a check spans more than one table (cross-table, aggregation, or column-subset checks), its failed rows can't be folded onto a single annotated table, so they surface under "residues" instead. Residues are per-check — one entry per non-mergeable check, keyed "<schema>::<checkname>", each carrying its own failed rows plus the same checkinfo column the annotated tables use (a single-element JSON array) and tablesinquery:

Residues

output = result.getannotatedoutput()
print("Residue keys:", list(output["residues"].keys())) for key, residue in output["residues"].items():     df = residue.to_pandas()     print(f"\nResidue '{key}': {len(df)} failed row(s)")     print(df[["employeeid", "payrollid", "month", "checkinfo", "tablesin_query"]])

For the full eligibility rules and worked examples of each non-mergeable category, see Known Issues: Annotated Output. The usage patterns notebook walks through these examples end-to-end.

The save() method also supports annotated output via output_mode:

# Save annotated tables (full tables with check_info marking failures)
result.save(output_mode="annotated")
Shape the check_info column: "names" (default), "summary", or "full"
result.save(outputmode="annotated", checkinfo="summary")
Save both failed-rows CSVs and annotated tables
result.save(output_mode="both")

You can also set the output mode globally via ValidationConfig:

from vowl import validate_data
from vowl.config import ValidationConfig
config = ValidationConfig(output_mode="annotated") result = validate_data("contract.yaml", df=df, config=config) result.save()  # uses the configured output_mode

Architecture

vowl has a modular architecture built around Ibis as the universal query layer.

┌─────────────────────────────────────────────────────────────────────────────┐
│                              validate_data()                                │
│                           (Main Entry Point)                                │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           DataSourceMapper                                  │
│              (Auto-detects input type → creates adapter)                    │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
          ┌──────────────────────────┼──────────────────────────┐
          ▼                          ▼                          ▼
┌──────────────────┐      ┌──────────────────┐      ┌──────────────────┐
│   IbisAdapter    │      │ MultiSourceAdapter│      │  Custom Adapter  │
│                  │      │                  │      │                  │
│ • pandas/Polars  │      │ • Cross-database │      │ • Extend         │
│ • PySpark        │      │   validation     │      │   BaseAdapter    │
│ • PostgreSQL     │      │ • Data federation│      │                  │
│ • Snowflake      │      │                  │      │                  │
│ • BigQuery       │      │                  │      │                  │
│ • 20+ backends   │      │                  │      │                  │
└──────────────────┘      └──────────────────┘      └──────────────────┘
          │                          │                          │
          └──────────────────────────┼──────────────────────────┘
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Executors                                      │
│                                                                             │
│  ┌─────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐     │
│  │  IbisSQLExecutor│  │MultiSourceSQLExecutor│  │  Custom Executor   │     │
│  │                 │  │                     │  │                     │     │
│  │ Runs SQL checks │  │ Mode 1: delegate to │  │ Extend BaseExecutor │     │
│  │ via Ibis        │  │ backend (same conn) │  │ or SQLExecutor      │     │
│  │ (server-side)   │  │ Mode 2: materialise │  │                     │     │
│  │                 │  │ to DuckDB via Arrow │  │                     │     │
│  └─────────────────┘  └─────────────────────┘  └─────────────────────┘     │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           ValidationResult                                  │
│                                                                             │
│  • Per-check failed rows with checkid & tablesin_query columns            │
│  • Detailed check results and metrics                                       │
│  • Export to CSV/JSON                                                       │
└─────────────────────────────────────────────────────────────────────────────┘

Key Components

| Component | Description | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DataSourceMapper | Auto-detects a single input source (DataFrame, Spark object, Ibis backend, or connection string) and creates the appropriate adapter | | IbisAdapter | Universal adapter supporting 20+ backends via Ibis (pandas, Polars, PySpark, PostgreSQL, Snowflake, BigQuery, etc.) | | MultiSourceAdapter | Routes checks across multiple data sources, separating single-table checks (delegated to per-schema adapters) from multi-table checks (sent to MultiSourceSQLExecutor) | | IbisSQLExecutor | Executes SQL-based quality checks through the Ibis query layer (server-side) | | MultiSourceSQLExecutor | Executes cross-source SQL with two modes: direct delegation when all tables share the same compatible backend, or DuckDB materialisation when backends differ. Tables are exported as Arrow and loaded into a local DuckDB for cross-database joins | | Contract | Parses ODCS YAML contracts into executable validation rules | | ValidationResult | Rich result object with enhanced DataFrames, metrics, and export capabilities |

Part 3 · Usage Patterns

Interactive demo: Try the usage patterns notebook for a hands-on walkthrough of the examples below.

The patterns are grouped from most common to most advanced:

• Common sources — point vowl at the data you already have: a local DataFrame, PySpark, or any of 20+ Ibis backends.
• Filtering & cross-source — validate a subset of rows, or join across multiple source systems — optionally through DuckDB ATTACH.
• Advanced & extending — drive connections from servers in the contract, build custom adapters/executors, or load contracts from Git and S3.

Common sources

Local DataFrame (Pandas/Polars)

import pandas as pd
from vowl import validate_data
df = pd.read_csv("data.csv") result = validate_data("contract.yaml", df=df) result.displayfullreport()

PySpark

from pyspark.sql import SparkSession
from vowl import validate_data
Create SparkSession (user-managed)
spark = SparkSession.builder.appName("vowl").getOrCreate()
try:     sparkdf = spark.read.table("mytable")     result = validatedata("contract.yaml", df=sparkdf)     result.displayfullreport() finally:     # User is responsible for stopping the SparkSession     spark.stop()

Note: The library does not manage the SparkSession lifecycle. You must create and stop it yourself. This is by design - SparkSession is a heavy, application-owned resource with specific configuration requirements.

Ibis Connections (20+ Backends)

# Ibis supports: Amazon Athena, BigQuery, ClickHouse, Dask, Databricks, DataFusion,
Druid, DuckDB, Exasol, Flink, Impala, MSSQL, MySQL, Oracle, pandas, Polars,
PostgreSQL, PySpark, RisingWave, SingleStoreDB, Snowflake, SQLite, Trino, ...
Find out more at https://github.com/ibis-project/ibis
import ibis from vowl import validate_data from vowl.adapters import IbisAdapter
con = ibis.postgres.connect(...)  # Redshift can be supported via Postgres connections too
result = validate_data("contract.yaml", adapter=IbisAdapter(con)) result.displayfullreport()

For MySQL, select the database when you create the connection, for example via ibis.mysql.connect(..., database="my_db") or a connection URI that already includes the database name. vowl does not issue USE database during validation; it runs read-only SELECT queries against the active database on the existing connection. If you need to avoid relying on the connection's default database, use qualified table names such as mydb.mytable in your contract queries.

Concurrent Checks (PooledAdapter)

When a contract has many independent checks and the backend can serve several queries at once, run them concurrently by wrapping a connection factory in a PooledAdapter. It keeps a thread-safe pool of connections (one per worker) and dispatches checks across them; the verdicts are identical to a sequential run.

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter, PooledAdapter, MultiSourceAdapter
factory: returns a fresh adapter (new connection) on each call. Called once
per pooled connection, so the table must be available on every connection.
def make_adapter():
    con = ibis.duckdb.connect("my_db.duckdb")
    return IbisAdapter(con)
pooled = PooledAdapter(factory=makeadapter, maxconcurrency=4)
PooledAdapter is a connection pool, so wire it in via MultiSourceAdapter
(keyed by schema name) and pass it through adapters=.
multi = MultiSourceAdapter({"my_table": pooled})
result = validate_data("contract.yaml", adapters=multi)
result.displayfullreport()

Filtering & cross-source

Explicit Adapter with Filter Conditions

from vowl import validate_data
from vowl.adapters import IbisAdapter
from datetime import datetime, timedelta
import ibis
date_limit = (datetime.today() - timedelta(days=7)).strftime("%Y-%m-%d") con = ibis.postgres.connect(...)
Using dict for filter conditions with wildcard patterns
Wildcards use glob-style matching: * (any chars), ? (single char), [seq] (char in seq)
adapter = IbisAdapter(
    con,
    filter_conditions={
        # Exact match
        "TableA": {
            "field": "date_dt",
            "operator": ">=",
            "value": date_limit
        },
        # Wildcard: matches employees, emphistory, empdetails, etc.
        "emp*": {
            "field": "date_dt",
            "operator": ">=",
            "value": date_limit
        },
        # Wildcard: matches ordersarchive, customersarchive, etc.
        "*_archive": {
            "field": "is_deleted",
            "operator": "=",
            "value": False
        },
        # Apply to ALL tables
        "*": {
            "field": "tenant_id",
            "operator": "=",
            "value": 123
        },
    }
)
Note: If multiple patterns match a table, conditions are combined with AND
Multiple filter conditions on same table (combined with AND)
adapter = IbisAdapter(
    con,
    filter_conditions={
        "TableA": [
            {"field": "datedt", "operator": ">=", "value": datelimit},
            {"field": "status", "operator": "=", "value": "active"},
        ]
    }
)
result = validate_data("contract.yaml", adapter=adapter) result.displayfullreport()

Multi-Source Validation

There are two ways to validate across tables in different databases.

Option A: DuckDB ATTACH (recommended: streams data, no materialisation)

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter
con = ibis.duckdb.connect()
Attach multiple remote databases
con.rawsql("ATTACH 'postgresql://user:pass@host:5432/salesdb' AS pgsales (TYPE postgres, READ_ONLY)")  # trufflehog:ignore
con.rawsql("ATTACH 'sqlite:///path/to/users.db' AS sqliteusers (TYPE sqlite, READ_ONLY)")
Switch back to local DuckDB so views live in memory
con.raw_sql("USE memory")
Create views as prefix-free shortcuts to the attached tables
con.rawsql("CREATE VIEW transactions AS SELECT * FROM pgsales.transactions")
con.rawsql("CREATE VIEW users AS SELECT * FROM sqliteusers.users")
Now vowl (and your contract queries) can reference tables without alias prefixes
result = validate_data("contract.yaml", adapter=IbisAdapter(con))
result.displayfullreport()

Note: DuckDB evaluates views dynamically at query time, so this does not materialise or copy data. It streams live from your attached databases; you just get cleaner, prefix-free table names in your contracts. DuckDB ATTACH supports PostgreSQL, MySQL, and SQLite.

Option B: Multi-Source Adapters (materialises data locally)

from vowl import validate_data
from vowl.adapters import IbisAdapter
import ibis
con_a = ibis.postgres.connect(...) con_b = ibis.sqlite.connect(...)
adapters = {     "tablea": IbisAdapter(cona),     "tableb": IbisAdapter(conb) }
result = validate_data("contract.yaml", adapters=adapters) result.displayfullreport()

Why this exists: A fallback for backends that DuckDB ATTACH does not support (e.g. Snowflake, BigQuery, Databricks, Oracle, MSSQL). The MultiSourceAdapter materialises entire tables on the client via Arrow into a local DuckDB instance, so prefer ATTACH whenever possible. DuckDB ATTACH only supports PostgreSQL, MySQL, and SQLite. It cannot be used as a general-purpose multi-source strategy because of namespace, credential, and filter limitations. It also preserves a known dark pattern: SQL checks can reference tables not declared in the contract's schema block, and those queries succeed with MultiSourceAdapter (everything is materialised locally) but fail with DuckDB ATTACH (only explicitly attached tables are visible).

Compatibility Mode (DuckDB ATTACH)

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter
ATTACH lets DuckDB query your remote database directly.
Data is streamed on demand, not materialised locally.
All SQL is evaluated by DuckDB, so dialect differences are eliminated.
con = ibis.duckdb.connect()
con.rawsql("ATTACH 'postgresql://user:pass@host:5432/mydb' AS pg (TYPE postgres, READONLY)")  # trufflehog:ignore
con.raw_sql("USE pg")  # Allows querying tables without the pg. alias
result = validate_data("contract.yaml", adapter=IbisAdapter(con)) result.displayfullreport()

When to use this: Your remote backend doesn't support a SQL feature that a check needs, or you want a single local engine for reproducible results regardless of the source database. DuckDB ATTACH supports PostgreSQL, MySQL, and SQLite.

Advanced & extending

Using Servers Defined in Data Contract

```python from vowl import validate_data from vowl.contracts import Contract from vowl.adapters import IbisAdapter import ibis

Load the contract and get server configuration

Or: contract.get_server("uat") # falls back to matching by environment

Or: contract.get_server() # returns the first server

Create connection based on server config

Create adapter and validate