pip install arnio

⚡ Quickstart

If you're new to Arnio, the example below demonstrates a simple first-run workflow for loading, cleaning, and preparing CSV data before converting it back into a pandas DataFrame. The workflow starts by loading a CSV dataset into an Arnio frame for preprocessing and cleaning.

import arnio as ar
Load CSV directly through C++ — no Python parsing overhead
frame = ar.readcsv("messysales_data.csv")
Strict mode (default) fails on inconsistent row widths
frame = ar.readcsv("messysales_data.csv", mode="strict")
Permissive mode fills missing trailing values with nulls
frame = ar.readcsv("messysales_data.csv", mode="permissive")

Each pipeline step applies a specific transformation such as trimming whitespace, normalizing text formatting, handling missing values, and removing duplicate rows.

# Declare what clean data looks like — arnio handles the rest
clean = ar.pipeline(frame, [
    ("strip_whitespace",),
    ("normalizecase", {"casetype": "lower"}),
    ("fill_nulls", {"value": 0.0, "subset": ["revenue"]}),
    ("drop_nulls",),
    ("drop_duplicates",),
])

After preprocessing is complete, the cleaned result can be converted back into a standard pandas DataFrame for further analysis or integration with existing workflows.

# Out comes a standard pandas DataFrame — use it like you always have
df = ar.to_pandas(clean)
Use copy=True when you need defensive pandas-owned buffers
safedf = ar.topandas(clean, copy=True)

Dry Run Validation

Use dry_run=True to validate pipeline configuration and step execution without returning transformed output.

ar.pipeline(
    frame,
    [
        ("drop_nulls",),
    ],
    dry_run=True,
)

Need step timings for debugging? Opt in without changing the default pipeline return type:

clean, metadata = ar.pipeline(
    frame,
    [("stripwhitespace",), ("dropduplicates",)],
    return_metadata=True,
)
print(metadata["step_timings"]) print(metadata["applied_steps"]) print(metadata["row_counts"]) print(metadata["execution_summary"])

Quick Example

import arnio
frame = arnio.read_csv("sample.csv")
Preview first 5 rows
frame.preview(5)
Generate and view scannable summary statistics
print(frame.describe())

Pipeline validation behavior

Pipeline step specifications are validated before execution begins.

Malformed step tuples, invalid kwargs structures, or unknown step names fail early before any pipeline steps execute.

ar.pipeline(
    frame,
    [
        ("strip_whitespace",),
        ("bad_step", "oops", "extra"),
    ],
)

This prevents partial pipeline execution when later pipeline steps are invalid.

from_dict support

This adds support for creating an ArFrame from a Python dictionary.

You can build an ArFrame directly from a dictionary of equal-length columns, which is useful for small inline datasets that you want to pass into a pipeline.

import arnio as ar
data = {"name": ["Alice", "Bob"], "age": [25, 30]}
frame = ar.from_dict(data) 
or
 frame = ar.ArFrame.from_dict(data)

Already working with a pandas DataFrame? Arnio can also be integrated directly into an existing pandas workflow without changing your current data-processing approach:

import pandas as pd
import arnio as ar
df = pd.readcsv("messysales_data.csv")
clean_df = df.arnio.clean([     ("strip_whitespace",),     ("normalizecase", {"casetype": "lower"}),     ("drop_duplicates",), ])
report = clean_df.arnio.profile()
trimmed_df = (     df.arnio.stripwhitespace(subset=["customername"])       .arnio.clipnumeric(upper=100, subset=["loyaltyscore"])       .arnio.dropnulls(subset=["customername"]) )

Cross-field validation rules

Pass a rules list to Schema for checks that span multiple columns. Each rule receives the full pandas DataFrame and must return a list[ValidationIssue] — an empty list means the rule passed.

import arnio as ar
def endafterstart(df):     return [         ar.ValidationIssue(             column="end_date",             rule="cross_field",             message="enddate must be >= startdate",             row_index=int(i) + 1,         )         for i, row in df.iterrows()         if row["enddate"] < row["startdate"]     ]
schema = ar.Schema(     {"startdate": ar.String(), "enddate": ar.String()},     rules=[endafterstart], )
result = schema.validate(ar.read_csv("events.csv")) print(result.passed)

Row index convention: ValidationIssue.row_index values are 1-based and

count data rows only. The header row is excluded. row_index=1 is the first data

row in the file.

Schema diff reports

Use diff_schema() to compare expected and observed data contracts across datasets, releases, or generated schemas.

import arnio as ar
expected = ar.Schema({     "id": ar.Int64(nullable=False, unique=True),     "email": ar.Email(nullable=False), })
observed = ar.Schema({     "id": ar.Int64(nullable=False),     "created_at": ar.DateTime(format="%Y-%m-%d"), })
diff = ar.diff_schema(expected, observed) print(diff.summary()) print(diff.to_markdown())

CI data contracts (GitHub Actions)

If you want to block schema drift or invalid rows in pull requests, see DATACONTRACTCI.md for an inert copy-paste GitHub Actions workflow example.

Example contract files are included under examples/contracts/.

Select specific columns

Use select_columns() to create a new ArFrame with only the required columns before converting to pandas.

selected = ar.select_columns(frame, ["name", "revenue"])
print(selected.columns) 
['name', 'revenue']

• Preserves the requested column order.
• Returns a new ArFrame.
• Raises ValueError if any requested column does not exist.
• Raises TypeError if columns is not a sequence of strings.

Handling missing values

Arnio supports configuring which strings are treated as null during CSV parsing using the nullvalues parameter in readcsv and scan_csv. By default, Arnio preserves its existing behavior and treats only empty cells as null. Custom matching is case-insensitive and applies to cell values only (not headers).

# Default behavior: empty cells are null
frame = ar.read_csv("data.csv")
Provide a custom list of sentinels (overrides the empty-cell default)
frame = ar.readcsv("data.csv", nullvalues=["", "MISSING", "UNKNOWN"])
Disable null sentinel handling completely
frame = ar.readcsv("data.csv", nullvalues=[])

Handling decimal separators

Use decimal_separator when numeric CSV data uses a separator other than the default dot. This is explicit by design: Arnio does not auto-detect decimal formats because a comma can also be the CSV delimiter.

# Semicolon-delimited CSV with unquoted European decimals
frame = ar.readcsv("prices.csv", delimiter=";", decimalseparator=",")
Comma-delimited CSV still needs quoted comma-decimal values
frame = ar.readcsv("prices.csv", decimalseparator=",")

The default remains decimal_separator=".", so existing dot-decimal files keep their current behavior. If you also use thousands_separator, it must differ from decimal_separator.

Handling invalid UTF-8 bytes

Use encoding_errors to control how invalid UTF-8 bytes are handled during CSV parsing.

# Raise an error on invalid UTF-8 bytes (default)
frame = ar.read_csv(
    "data.csv",
    encoding_errors="strict",
)
Replace invalid bytes with the Unicode replacement character (�)
frame = ar.read_csv(
    "data.csv",
    encoding_errors="replace",
)
Ignore invalid bytes completely
frame = ar.read_csv(
    "data.csv",
    encoding_errors="ignore",
)

• "strict" (default)
• "replace"
• "ignore"

Every step above executes in C++. Your Python code is a configuration — not the execution engine.

Explore more in the examples/ folder — ready-to-run recipes for sales, customers, survey, logs, and finance datasets.

Security note: CSV formula injection

Arnio preserves cell values by default when writing CSV files. It does not rewrite strings that begin with spreadsheet formula prefixes such as =, +, -, or @ unless you opt in to formula escaping.

If you export Arnio-cleaned data back to CSV and expect users to open that file in Excel, Google Sheets, LibreOffice, or another spreadsheet application, treat untrusted text fields as potentially executable spreadsheet formulas. Use escape_formulas=True when exporting user-controlled text to prefix formula-like string cells with a single quote while leaving numeric columns unchanged:

ar.writecsv(cleanframe, "safeexport.csv", escapeformulas=True)

This is especially important for customer names, notes, comments, imported form fields, and any other free-text values that may come from outside your trust boundary. Arnio keeps the export policy explicit, so existing CSV output remains unchanged unless this option is enabled.

Error Handling

readcsv and scancsv

| Input | Raises | Message | |:---|:---|:---| | File not found | CsvReadError | Cannot open file: <path> | | Zero-byte file | CsvReadError | CSV file is empty: '<path>' | | Blank header line | CsvReadError | CSV header contains an empty column name | | Binary / NUL bytes | CsvReadError | CSV input contains NUL bytes and appears to be binary or corrupted |

Schema Validation

ar.validate() returns a ValidationResult; it does not raise for validation failures. Check result.passed and result.issues for dtype or required_column rule violations.

validate() currently operates on a single in-memory ArFrame. Chunked validation via readcsvchunked() iterators is not yet supported directly. Validate each chunk individually or materialize the data before validation when working with streamed/chunked inputs.

Pipeline Step Errors

Unknown step names raise UnknownStepError before execution begins.

# Pass sample_size to control how many rows are evaluated for type inference
schema = ar.scancsv("100GBfile.csv", sample_size=500)
{'id': 'int64', 'name': 'string', 'is_active': 'bool', 'revenue': 'float64'}

# events.jsonl
{"user": "alice", "score": 9.5, "active": true}
{"user": "bob",   "score": 8.1, "active": false}
frame = ar.read_jsonl("events.jsonl")
Limit rows
frame = ar.read_jsonl("large.jsonl", nrows=1000)
Stream large JSONL/NDJSON files in bounded chunks
for chunk in ar.readjsonlchunked("large.jsonl", chunksize=50_000):
    process(chunk)
Non-UTF-8 encoding
frame = ar.read_jsonl("data.ndjson", encoding="latin-1")
Plug straight into the cleaning pipeline
clean = ar.pipeline(frame, [("stripwhitespace",), ("dropnulls",)])

pip install arnio[parquet]

# Import a Parquet file into an ArFrame (no pandas intermediate)
frame = ar.read_parquet("data.parquet")
Read only selected columns
frame = ar.read_parquet("data.parquet", usecols=["name", "age"])
Export an ArFrame to Parquet
ar.write_parquet(frame, "output.parquet")
Choose compression codec: "snappy" (default), "gzip", "zstd", "brotli", "none"
ar.write_parquet(frame, "output.parquet", compression="zstd")
Control row group size for large files
ar.writeparquet(frame, "output.parquet", rowgroupsize=50000)
.pq extension also accepted for both read and write
frame = ar.read_parquet("data.pq")
ar.write_parquet(frame, "output.pq")

frame = ar.readcsv("hugefile.csv")
print(frame.preview())      # first 5 rows (default) print(frame.preview(n=10))  # first 10 rows

from decimal import Decimal
import pandas as pd
import arnio as ar
df = pd.DataFrame({     "price": [Decimal("19.99"), Decimal("29.95")] })
frame = ar.from_pandas(df)  # Decimal values safely preserved as exact strings result = ar.to_pandas(frame) 
result["price"] will be string objects ["19.99", "29.95"]

def removeoutliers(df, column="revenue", threshold=100000):
    return df[df[column] <= threshold]
ar.registerstep("removeoutliers", remove_outliers) ar.registerstep("team:dropnulls", remove_outliers)  # namespaced custom step
Use builtin: for an explicit built-in step, and your own prefixes
like team: or plugin_name: to avoid name collisions.
Introspect built-in and custom step names without reaching into internals.
print(ar.list_steps())
Opt in to a context object only when you need execution metadata.
def capture_context(df, context=None):
    print(context.stepname, context.stepindex, context.total_steps)
    return df
Now use it in any pipeline alongside native C++ steps
clean = ar.pipeline(frame, [
    ("builtin:strip_whitespace",),
    ("remove_outliers", {"column": "revenue", "threshold": 50000}),
    ("drop_duplicates",),
])

signatures = ar.getbuiltinstep_signatures()
print(list(signatures["drop_nulls"].parameters))  # ["subset"]
print(list(signatures["filter_rows"].parameters))  # ["column", "op", "value"]

ar.reset_steps()
print(ar.list_steps()) 
Only built-in steps remain

def custom_logging(df):
    print("Running step v1")
    return df
ar.registerstep("logdata", custom_logging)
This will succeed and safely overwrite the original logic
def customloggingv2(df):
    print("Running step v2")
    return df
ar.registerstep("logdata", customloggingv2, overwrite=True)

frame = ar.read_csv("data.csv")

frame.head()     # first 5 rows (default) frame.head(10)   # first 10 rows frame.tail(3)    # last 3 rows
n larger than row count returns all rows safely
frame.head(1000)
n=0 returns an empty ArFrame
frame.head(0)

Pipeline verbose diagnostics

Enable lightweight pipeline diagnostics with verbose=True:

result = ar.pipeline(
    frame,
    [
        ("strip_whitespace",),
        ("drop_nulls",),
    ],
    verbose=True,
)

This logs step execution order, execution path, elapsed time, and row-count changes through the arnio logger.

🔗 Integrations

Arnio is designed to make the rest of the Python data stack more productive, not to replace it.

| Workflow | How Arnio helps | |:---|:---| | pandas | Clean, validate, and profile messy DataFrames through df.arnio. | | NumPy | Prepare typed numeric data before array/modeling workflows. | | scikit-learn | Use Arnio cleaning as a preprocessing layer before model training. | | DuckDB / Arrow | Validate and prepare data before analytics and columnar exchange. Export ArFrame to pyarrow.Table via `ar.to_arrow(frame). | | notebooks | Inspect quality issues and cleaning suggestions before analysis. |

Use ar.registerduckdb(frame, conn, "tablename") to register an ArFrame directly as a DuckDB relation without writing pandas conversion glue yourself. DuckDB is an optional dependency — install it with pip install "arnio[duckdb]" when needed. You can also install DuckDB directly with pip install duckdb.

For development and CI, pip install -e ".[dev]" now includes DuckDB so the integration test module in tests/testintegrationsduckdb.py runs in the default test job.

ArnioCleaner enforces a strict transformer contract by default:

• Row-count-changing steps (dropnulls, dropduplicates,

• Column schema-changing steps (renamecolumns, dropcolumns,

Arnio integrates directly with pandas through a registered DataFrame accessor. After importing Arnio, every pandas DataFrame gains an .arnio accessor that provides cleaning, profiling, validation, and pipeline utilities without requiring you to leave your existing pandas workflow.

The accessor is registered through pandas' DataFrame extension API, allowing Arnio methods to be accessed directly as df.arnio.

Cleaning data with .clean()

Use .clean() to apply common data-cleaning operations and return a cleaned pandas DataFrame.

• strip_whitespace – remove leading and trailing whitespace from string values
• drop_nulls – remove rows containing null values
• drop_duplicates – remove duplicate rows
• steps – optional Arnio pipeline steps. When provided, .clean() executes the supplied pipeline and returns the resulting pandas DataFrame.

Profiling data with .profile()

Use .profile() to generate a data quality summary.

.profile() returns a DataQualityReport containing dataset quality information and profiling insights that can help identify issues before downstream analysis.

Accessor vs ar.pipeline()

| Approach | Best for | |-----------|-----------| | df.arnio.clean() | Existing pandas workflows | | df.arnio.profile() | Quick quality assessment | | df.arnio.pipeline(...) | Custom pipeline execution from a DataFrame | | ar.pipeline(...) | Direct Arnio/ArFrame workflows |

| Decision | What it means | |:---|:---| | Columnar storage | Data lives in typed std::vectors — vector, vector, vector — not rows of variants. Cache-friendly and SIMD-ready. | | Boolean null masks | Nulls are tracked in a separate vector, keeping data vectors dense. No sentinel values, no NaN tricks. | | Two-pass CSV read | Pass 1 infers types across all rows. Pass 2 parses values directly into the correct typed column. No string→object→cast overhead. | | Zero-copy bridge | to_pandas() exposes C++ memory directly via NumPy's buffer protocol where supported. Numeric columns preserve the fast zero-copy path by default, while copy=True requests defensive pandas-owned buffers. | | Step registry | Built-in and native steps use the C++ core via STEPREGISTRY; Python-backed built-ins dispatch through PYTHONSTEP_REGISTRY; custom user-defined steps follow the same Python registry path. Adding a new cleaning primitive is a single function + one registry entry. |

> Reference environment: Ubuntu, Python 3.12, synthetic messy CSV inputs.<br> > Reproduce: make benchmark — generates deterministic tall and wide datasets and runs both engines.

benchmarks/generatedata.py uses deterministic NumPy seeds, so every run creates the same benchmarks/benchmark1m.csv tall input and benchmarks/benchmarkwide.csv wide input. The benchmark then executes three pandas runs and three arnio runs for each case, printing average wall-clock time from time.perfcounter() and peak Python allocation from tracemalloc. For cleaner comparisons, close other memory-heavy processes and run the script from the repository root after installing the same Python, pandas, NumPy, compiler, and arnio commit you want to compare.

Arnio is near memory parity in the reference benchmark while replacing ad-hoc Python string loops with a compiled, declarative pipeline. Validate memory and speed on your own workload. The execution time gap is a known, active optimization target — the current dropduplicates and stripwhitespace implementations use unoptimized row-key serialization.

• Native C++ parsing eliminates Python memory spikes
• Columnar storage matches pandas' internal efficiency
• Declarative API eliminates .apply() spaghetti
• Zero-copy bridge for numeric conversions

• drop_duplicates — replace string serialization with hash-based comparisons
• strip_whitespace — in-place mutation instead of copy-on-write
• Parallel column processing via std::thread
• Help close the gap →

To measure the peak memory and execution time of the auto_clean pipeline using realistic dataset sizes:

• ar.read_csv performance
• ar.auto_clean(mode="safe") performance (low-risk cleanup like whitespace trimming)
• ar.auto_clean(mode="strict") performance (includes type casting and deduplication)

• --repeat N runs each operation multiple times and reports average (and min/max range).
• --seed N changes the deterministic dataset seed.
• --reuse-file reuses an existing dataset file instead of regenerating it.
• --keep-file keeps the generated CSV (otherwise it is removed at the end).

Most operations below run natively in C++. Currently, filterrows, replacevalues and standardizemissingtokens run via the Python (pandas) backend and may be optimized in C++ later.

| Primitive | What it does | Example | |:---|:---|:---| | dropnulls | Remove rows with null/empty values | ar.dropnulls(frame, subset=["age"]) | | dropcolumns | Remove selected columns while preserving the remaining order | frame = ar.dropcolumns(frame, ["debug_col"]) | | dropemptycolumns | Remove columns whose values are all null/empty | frame = ar.dropemptycolumns(frame) | | keeprowswithnulls | Keep only rows that contain at least one null | ar.keeprowswithnulls(frame, subset=["age"]) | | validatecolumnsexist | Fail early when required columns are missing | ar.validatecolumnsexist(frame, ["age"]) | | filterrows | Filter rows using comparison operators | ar.filterrows(frame, column="age", op=">", value=18) | | fillnulls | Replace nulls with a scalar | ar.fillnulls(frame, 0, subset=["revenue"]) | | dropduplicates | Deduplicate rows (first/last/none) | ar.dropduplicates(frame, keep="first") | | dropconstantcolumns | Remove columns with only one unique value | ar.dropconstantcolumns(frame) | | clipnumeric | Clip numeric values to lower and/or upper bounds | ar.clipnumeric(frame, lower=0, upper=100) | | coalescecolumns | Select the first non-null value from a list of columns | ar.coalescecolumns(frame, subset=["phone", "mobile"], output_column="contact") | | combinecolumns | Combine multiple columns into a single output column | ar.combinecolumns(frame, subset=["first", "last"], separator=" ", output_column="name") | | stripwhitespace | Trim leading/trailing spaces from strings | ar.stripwhitespace(frame) | | standardizemissingtokens | Replace common missing-value strings with NaN | ar.standardizemissingtokens(frame) | | normalizecase | Force lower/upper/title case | ar.normalizecase(frame, case_type="title") | | renamecolumns | Rename columns via mapping | ar.renamecolumns(frame, {"old": "new"}) | | casttypes | Cast column types with errors="raise", "coerce", or "ignore" | ar.casttypes(frame, {"age": "int64"}, errors="raise") | | roundnumericcolumns | Round numeric columns (non-numeric columns in subset ignored safely) | ar.roundnumericcolumns(frame, decimals=2) | | replacevalues | Replace values using a mapping (column or whole-frame). Handles None/NaN. | ar.replacevalues(frame, {"active": "A", "inactive": "I"}, column="status") | | clean | Convenience shorthand supporting config dicts | ar.clean(frame, stripwhitespace={"subset": ["name"]}, dropnulls=True) | | safedividecolumns | Divide one column by another, handling zero/null denominators | ar.safedividecolumns(frame, numerator="revenue", denominator="cost", output_column="ratio") | | dropcolumnsmatching | Drop columns whose names match a regex pattern | ar.dropcolumnsmatching(frame, pattern="^temp_") | | trimcolumnnames | Strip leading/trailing whitespace from column names | ar.trimcolumnnames(frame) | | selectcolumns | Return a new frame containing only selected columns | ar.selectcolumns(frame, ["id", "name"]) | | slugifycolumnnames | Normalise column names to snakecase | ar.slugifycolumn_names(frame) |

ArFrame.select_dtypes — type-based column selection

Returns a new ArFrame containing only the columns whose dtype matches the filter. Raises ValueError if no columns match.

Valid dtype strings: "int64", "float64", "string", "bool", "null"

• At least one of include or exclude must be given — raises ValueError otherwise.
• include and exclude must not overlap — raises ValueError if they share a dtype.
• Unknown dtype strings raise ValueError with a list of valid options.
• Raises ValueError when no columns match (never returns an empty frame silently).
• Column order in the result always matches the original frame.

winsorize_outliers() clips extreme numeric values using lower and upper quantiles. Non-numeric columns are ignored unless explicitly selected in subset.

It can also be used inside ar.pipeline() as ("winsorize_outliers", {"lower": 0.05, "upper": 0.95}).

Use replace_values to substitute values using a mapping. It works as a pipeline step (Python backend) and can operate on a single column or the whole frame when column is omitted. It also understands null semantics: using None (or np.nan) as a mapping key targets existing nulls, and mapping a value to None creates real nulls.

Whole-frame example (no column):

Use filter_rows to keep only rows matching a condition.

• >
• <
• >=
• <=
• ==
• !=

Use keeprowswith_nulls to audit incomplete data — keep only rows that have at least one null.

This normalizes values such as "yes", "no", "true", "false", "y", "n", "1", and "0" into boolean values while preserving unsupported values unchanged.

> When the denominator is zero or null, the result is replaced with fill_value (default 0.0) instead of raising an error or producing NaN/Inf.

| Pandas Dtype | Support Status | Notes / Fix Hints | |---|---|---| | int64 / Int64 | ✅ Supported | Fully supported with native C++ columnar storage. Nulls mapped to pd.NA. | | float64 / Float64 | ✅ Supported | Fully supported with zero-copy conversion. Nulls mapped to np.nan or pd.NA. | | bool / boolean | ✅ Supported | Native booleans supported with C++ backing. Nulls mapped to pd.NA. | | string / string[python] | ✅ Supported | Native string extension type. Recommended for text. Nulls mapped to pd.NA. | | object (strings / scalars) | ✅ Supported | Handled as text or coerced to common type if mixed. | | object (nested / lists / dicts) | ❌ Unsupported | Nested structures not allowed in flat columnar storage. Raises TypeError. | | category | ❌ Unsupported | Raises TypeError with fix hint. Convert to string: df["col"].astype(str) | | datetime64[ns] / timezone-aware | ❌ Unsupported | Raises TypeError with fix hint. Use df["col"].astype(str) or string timestamps. | | timedelta64[ns] | ❌ Unsupported | Raises TypeError with fix hint. Use df["col"].dt.total_seconds(). | | complex64 / complex128 | ❌ Unsupported | Raises TypeError with fix hint. Split into real/imag columns or convert to strings. |

• Zero-copy Optimization: Numeric columns (int64, float64) are optimized for fast zero-copy conversion between C++ and pandas where supported.
• Defensive Buffers: Pass copy=True to to_pandas() when downstream pandas code needs defensive pandas-owned column buffers.
• Boolean Buffers: Boolean conversion is copied because std::vector cannot be exposed as a zero-copy NumPy buffer.
• Null Handling: Columns with null masks are automatically converted to pandas nullable Extension dtypes (Int64, BooleanDtype, StringDtype).
• Index Drop: pandas DataFrame indexes are currently not preserved during frompandas() conversion; converted frames receive a default RangeIndex when converted back via topandas().
• Validation: Attempting to convert any unsupported type will raise a clear, user-friendly TypeError detailing the column name and how to fix/preprocess it.

In this example, country becomes required only when user_type == "international".

PhoneNumber() validates common international and formatted phone number strings.

• +1-555-123-4567
• +91 9876543210
• 5551234567

URL() validates that values are well-formed URLs. By default, both http and https schemes are accepted.

Use allowed_schemes to restrict which URL schemes are valid:

Any URL with a scheme not in allowed_schemes will fail validation.

See examples/schema_validation.py for a complete runnable tutorial covering Schema, field types, invalid-row reporting, and ValidationResult output.

ValidationResult.to_markdown() is useful in CI logs, GitHub comments, or data quality reports because it renders a compact validation summary plus a GitHub-friendly issue table.

| order_id | customer | city | |:--|:--|:--| | 1001 | Ishan | Paris | | 1002 | Prasoon | London | | 1002 | Prasoon | London | | 1003 | Pranay | New York | | 1004 | Dhruv | Tokyo |

Expected cleaned output with mode="strict":

mode="safe" only trims whitespace. Use mode="strict" when you also want deterministic built-in cleanup such as exact duplicate removal. If strict mode proposes type casts, run dryrun=True first and pass the exact proposed mapping as confirmedcasts with allowlossycasts=True.

Arnio provides detailed profiling for datasets via ar.profile(). To generate the report shown in these examples, the following code was used:

| Field or export | Aggregate-only? | May expose raw / sample data? | | --- | --- | --- | | rowcount, columncount, duplicaterows, duplicateratio, qualityscore, scorecomponents | Yes | No | | nullcount, nullratio, uniquecount, uniqueratio, whitespace / empty-string counts | Yes | No | | Numeric min / max / mean / std / q25–q95 | Yes | Statistics only; small tables can still be identifying | | Numeric iqr, outlierlowerbound, outlierupperbound, outliercount, outlierratio | Yes | Aggregate Tukey-fence summary (thresholds and counts, not which rows are outliers) | | semantictype, suggesteddtype, warnings | Metadata / hints | Can imply PII type (for example email-like), not redaction | | ColumnProfile.samplevalues (in-memory) | No | Yes — first N non-null values (samplesize on ar.profile()) | | ColumnProfile.top_values | Includes counts / ratios | Yes — frequent actual values (exact or approximate; see below) | | report.todict() | Mixed | Yes — includes samplevalues and top_values unless you redact samples | | report.todict(redactsamplevalues=True) | Mixed | samplevalues → "[REDACTED]" (same list length); top_values[*].value → "[REDACTED]" while counts and ratios remain | | report.to_markdown(), report.summary() | Yes | No raw cell values in output | | report.tohtml() / notebook display of report | Partial | Shows topvalues chips; does not list samplevalues. Use redacttopvalues=True or excludecolumns for safer sharing. | | report.topandas() | Partial | Includes topvalues, not sample_values | | ProfileComparison.todict() | Nested profiles | Yes — embeds leftprofile / rightprofile via default todict() |

• JSON logs and artifacts: report.todict(redactsample_values=True) before writing or uploading.
• Collect fewer samples: ar.profile(frame, samplesize=0) skips samplevalues (defaults still apply to top_values counts on string columns).
• Text summaries for CI or comments: prefer report.to_markdown() or report.summary() when you do not need per-value examples.
• Notebooks and HTML exports: use report.tohtml(redacttopvalues=True) to replace every top-value chip label with [REDACTED] while preserving counts and ratios. To drop entire sensitive columns from the table, add excludecolumns=["ssn", "email"]. Avoid saving unredacted report.to_html() output for sensitive data.
• GitHub bug reports and examples: use synthetic data (user@example.com, ID-001), a minimal CSV, and redacted to_dict() output — not production dumps.
• Pandas export: ar.to_pandas(frame) returns full table data; redaction applies to quality reports, not the underlying frame.
• Profile comparison: ProfileComparison.todict() nests full profiles; build shared artifacts with profile.todict(redactsamplevalues=True) if needed.

When approxtopvalues=True, high-cardinality string columns estimate top_values from a deterministic sample. Each column may set topvaluesisapproximate, topvaluessamplecount, and topvaluessample_ratio. Counts and ratios are sample-based, but displayed values are still real strings from your data — treat them like top_values for privacy.

DataQualityReport includes a notebook-friendly HTML dashboard. In a notebook, simply evaluate report in a cell to see a rich, static summary (quality score, duplicates, nulls, warnings, top values, and cleaning suggestions).

For numeric columns with at least four non-null values, Arnio reports iqr (q75 − q25), Tukey fences outlierlowerbound (q25 − 1.5×IQR) and outlierupperbound (q75 + 1.5×IQR), plus outliercount and outlierratio. A value is counted as an outlier only if it is strictly less than the lower bound or strictly greater than the upper bound. With fewer than four non-null values, quantiles may still appear but IQR/outlier fields are null in JSON.

Illustrative age column (not from the user_id / email / score sample below):

Use ar.checkqualitygates() when profile drift should become a pass/fail decision for CI, data releases, or monitoring.

> Scoring Contract: The qualityscore starts at 100.0 and subtracts capped penalties for duplicates, nulls, and suggested dtype mismatches. The scorecomponents` field exposes these penalties as negative values. (Note: Semantic-validity pe