Data Contract Execution Engine

Overview

Data Contract Execution Engine (DCEE) is a lightweight Python framework for validating data against contracts and enforcing SLA rules. Built on pandas and boto3, it provides simple, fast data validation without heavy dependencies.

It reads YAML contract definitions that specify:

• Schema requirements (column names, types, nullability)
• SLA rules (min/max rows, data completeness)
• Source and target S3 paths for data files

The engine validates data on ingestion to ensure quality and consistency before writing to target destinations. Deploy as a Lambda function for serverless validation pipelines.

---

Key Features

• Simple Python implementation without heavy dependencies
• YAML-based contract definitions for schemas and SLA rules
• Data quality validation including schema checks and SLA enforcement
• AWS Lambda deployment for serverless processing
• S3 integration for reading and writing data
• Pandas-based data manipulation

---

Limitations

• CSV files only - Currently supports CSV format

- No support for TXT files with custom delimiters (tab, pipe, space, etc.) - No support for other formats (JSON, Parquet, Excel, etc.) - See CONTRIBUTING.md for how to add multi-format support

• Single-machine processing - Best for datasets under 1GB

- Loads entire dataset into memory - For larger files, consider using Spark for distributed processing

---

Installation

• Clone the repository:

git clone https://github.com/<your-username>/data-contract-execution-engine.git
cd data-contract-execution-engine

• Install dependencies:

pip install -r requirements.txt

---

Quick Start

See QUICKSTART.md for detailed instructions including:

• Local testing with sample data
• Testing with local file paths
• Deploying to AWS Lambda
• Running tests with coverage

Basic example:

1. Create a Contract

Define a YAML contract file (contracts/sample_contract.yaml):

name: "Customer Data Contract"
version: "1.0"
sources3path: "s3://my-bucket/input/customers.csv" targets3path: "s3://my-bucket/output/customers_validated.csv"
schema:   columns:     customer_id:       type: "integer"       nullable: false     name:       type: "string"       nullable: false     email:       type: "string"       nullable: true
sla:   min_rows: 1   max_rows: 1000000   completeness_threshold: 0.95

2. Run Locally

from engine.contractparser import loadcontract
from engine.pipeline_generator import PipelineGenerator
import pandas as pd
Load contract
contract = loadcontract("contracts/samplecontract.yaml")
Read data
df = pd.readcsv("sampledata.csv")
Validate
pipeline = PipelineGenerator(contract)
results = pipeline.generate(df)
print(f"Validation passed: {results['success']}")

3. Deploy to AWS Lambda

See LAMBDA_DEPLOYMENT.md for complete step-by-step deployment instructions.

---

Project Structure

├── engine/
│   ├── init.py
│   ├── contract_parser.py      # Load and parse YAML contracts
│   ├── validation_engine.py     # Schema and data quality validation
│   ├── sla_enforcer.py          # SLA rule enforcement
│   └── pipeline_generator.py    # Orchestrate validation pipeline
├── runtime/
│   └── lambda_handler.py        # AWS Lambda entry point
├── contracts/
│   └── sample_contract.yaml     # Example contract
├── tests/
│   └── ...                      # Unit tests
├── requirements.txt
└── README.md

---

Dependencies

• pandas - Data manipulation
• pyyaml - Contract parsing
• boto3 - AWS S3 access
• pytest - Testing framework

---

Contributing

Contributions are welcome! See CONTRIBUTING.md for:

• Development setup and coding standards
• How to implement new features
• Testing requirements
• Planned enhancements (multi-format file support, custom transformations, etc.)

Quick steps:

• Fork the repository
• Create a feature branch (git checkout -b feature/my-feature)
• Commit changes (git commit -am 'Add my feature')
• Push to branch (git push origin feature/my-feature)
• Create a Pull Request

---

License

Licensed under the Apache License, Version 2.0. See LICENSE file for details.