Topics

#data-quality #data-science #imputation #mice #missing-data #pandas #pypi #python #sklearn #visualization

Language breakdown

Python 98.5%

HTML 1.5%

▸ Files click to expand

README

missingly

This README describes the v1.0.0+ public API. For historical experiments see the legacy-experiments branch.

Missing data analysis for pandas — batteries included.

[ Python ]()

🌐 English | Deutsch | فارسی

missingly is a Python package for **diagnosing, visualising, and imputing missing data** in pandas DataFrames. It provides:

A fluent df.miss.* accessor that mirrors the ergonomics of the R naniar package.
sklearn-compatible transformers (MissinglyImputer) for use inside Pipeline.
One-shot HTML reports (create_report).
Statistical tests for MCAR / MAR / MNAR mechanisms.
Time-series-aware gap analysis and imputation.

Multiple Imputation (advanced)

For statistically valid inference after imputation, generate m datasets with imputemice(..., nimputations=m) and pool the model results using Rubin's Rules via the utilities in missingly.mi:

import numpy as np
import pandas as pd
from missingly import impute_mice
from missingly.mi import poolscalarestimates
from sklearn.linear_model import LinearRegression
1. Generate m imputed datasets
dfs = imputemice(df, nimputations=5)
2. Fit model on each imputed dataset
beta1ests, beta1vars = [], []
for d in dfs:
    reg = LinearRegression().fit(d[["x"]], d["y"])
    beta1ests.append(float(reg.coef[0]))
    resid = d["y"] - reg.predict(d[["x"]])
    ss_x = float(((d["x"] - d["x"].mean()) ** 2).sum())
    beta1vars.append(float(np.var(resid, ddof=2)) / ssx)
3. Pool with Rubin's Rules
result = poolscalarestimates(beta1ests, beta1vars)
print(f"Pooled beta1 = {result['q_bar']:.3f}  (total var = {result['t']:.4f})")

For multivariate models use poollinearregression_results(coefs, covs) which accepts arrays of shape (m, p) and (m, p, p) and returns a pooled coefficient vector plus a pooled covariance matrix.

Public API (v1)

The symbols below are the stable, supported API surface for v1. Breaking changes to these will be announced via a major-version bump.

`df.miss.*` accessor

import missingly  # registers df.miss automatically
import pandas as pd
import numpy as np
df = pd.DataFrame({"a": [1, np.nan, 3], "b": [np.nan, np.nan, 6]})
df.miss.n_miss()            # 3  — total missing count df.miss.pct_miss()          # 50.0  — % missing across whole DataFrame df.miss.missvarsummary()  # per-column summary table df.miss.vis_miss()          # missingness matrix visualisation df.miss.impute(strategy="mean")  # returns imputed DataFrame

Summary & Diagnosis

import missingly as mi
mi.n_miss(df)             # int — total missing count mi.pct_miss(df)           # float — overall % missing mi.missvarsummary(df)   # pd.DataFrame — per-column breakdown mi.misscasesummary(df)  # pd.DataFrame — per-row breakdown mi.mcar_test(df)          # Little's MCAR test result mi.marmnartest(df)      # MAR vs MNAR indicator mi.diagnose_missing(df)   # mechanism + recommendation dict

Visualisation

The visualisation layer lives in missingly.visualisation and is re-exported through missingly.visualise for backwards compatibility.

Module layout

| Module | Contents |

|---|---|

| missingly.visualisation.static | All matplotlib-based functions |

| missingly.visualisation.interactive | All Plotly backends (called when interactive=True) |

| missingly.visualisation.base | Shared helpers: rtlsafe, safelabels, nullity, pctlabels |

| missingly.visualise | Thin re-export facade — use this in application code |

Basic

import missingly as mi
import pandas as pd, numpy as np
df = pd.DataFrame({     "age":    [25, np.nan, 47, 33, np.nan],     "income": [50000, 62000, np.nan, np.nan, 71000],     "city":   ["A", "B", np.nan, "A", "C"], })
mi.vis_miss(df)          # annotated tile matrix with per-column % labels mi.matrix(df)            # raw presence/absence heatmap mi.bar(df)               # bar chart: count of missing per column mi.miss_case(df)         # bar chart: count of missing per row mi.missvarpct(df)      # horizontal bars: % missing per variable, sorted

Patterns

mi.miss_patterns(df)     # horizontal bars: top-N most frequent missingness patterns
mi.miss_cooccurrence(df) # symmetric heatmap: how often two columns miss together
mi.upset(df)             # UpSet plot of intersecting missingness sets
                         # returns dict of Axes: {"intersections", "matrix", "totals"}

Correlation / Clustering

# Nullity-correlation heatmap (Pearson on binary missingness indicators).
Delegates to dataqualitytoolkit.visualization.correlation_heatmap when
that package is installed; falls back to a pure-seaborn renderer otherwise.
mi.heatmap(df)
mi.heatmap(df, mask_insignificant=True)  # grey out non-significant cells
Hierarchical clustering of rows by missingness pattern.
Returns a single matplotlib.axes.Axes (not a dict).
ax = mi.miss_cluster(df)
Dendrogram of variables clustered by nullity correlation.
mi.dendrogram(df)

Interactive

Pass interactive=True to any function below to get a Plotly figure that can be panned, zoomed, and exported to HTML. When Plotly is not installed the function silently falls back to the static backend.

mi.vis_miss(df, interactive=True)
mi.heatmap(df, interactive=True)
mi.matrix(df, interactive=True)
mi.bar(df, interactive=True)
mi.missvarpct(df, interactive=True)
mi.miss_cooccurrence(df, interactive=True)
mi.miss_case(df, interactive=True)
mi.upset(df, interactive=True)
mi.miss_patterns(df, interactive=True)

Imputation

mi.impute_mean(df)           # mean imputation
mi.impute_median(df)         # median imputation
mi.impute_mode(df)           # mode imputation
mi.impute_knn(df)            # k-NN imputation (Euclidean distance, numeric-safe)
mi.impute_mice(df)           # MICE (IterativeImputer + BayesianRidge)
mi.impute_rf(df)             # Random Forest imputation
mi.impute_gb(df)             # Gradient Boosting imputation
Multiple Imputation — generate m datasets for Rubin pooling
dfs = mi.imputemice(df, nimputations=5)

KNN with Gower distance (mixed numeric + categorical)

By default impute_knn ordinal-encodes categorical columns and uses Euclidean distance — fast and suitable for mostly-numeric datasets.

For datasets dominated by categorical columns, pass metric="mixed" to use Gower distance instead. Gower treats numeric and categorical columns correctly: numeric columns are normalised by their range, nominal columns are compared by exact match.

df_mixed = pd.DataFrame({
    "age":    [25, np.nan, 35, 40],
    "city":   ["London", "Paris", None, "Berlin"],
    "grade":  ["A", "B", "A", None],
})
Euclidean KNN (default) — fast, ordinal-encodes categoricals
result = mi.imputeknn(dfmixed, n_neighbors=3)
Gower KNN — statistically sound for heavy-categorical data
result = mi.imputeknn(dfmixed, n_neighbors=3, metric="mixed")

Performance note: Gower distance is O(n²) in both memory and

runtime. Avoid metric="mixed" for datasets with more than ~10 000 rows.

The same metric parameter is available on MissinglyImputer:

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([     ("impute", mi.MissinglyImputer(strategy="knn", metric="mixed", n_neighbors=5)),     ("model",  LogisticRegression()), ]) pipe.fit(Xtrain, ytrain)

Time-series missingness

For time-indexed data, missingly provides gap-aware summary statistics, visualisation helpers, and interpolation-based imputation.

import missingly as mi
import pandas as pd
import numpy as np
Build a temperature series with some gaps
index = pd.date_range("2024-01-01", periods=14, freq="D")
temp  = [5.1, 4.8, np.nan, np.nan, 6.2, 6.5, np.nan, 7.0,
         7.3, np.nan, np.nan, np.nan, 8.1, 8.4]
ts = pd.DataFrame({"temp": temp}, index=index)
1. Summarise gaps
summary = mi.misstssummary(ts, col="temp")
print(summary)
n_miss          5
n_gaps          3
meangaplen    1.67
maxgaplen     3
longestgapstart  2024-01-10
longestgapend    2024-01-12
2. Visualise missingness over the time axis
ax = mi.vistsmiss(ts)
3. Impute with linear interpolation
tsfilled = mi.imputets(ts, strategy="linear")
print(ts_filled.isnull().sum())  # temp    0

Available strategies for impute_ts: ffill, bfill, linear, time, spline. Use limit=n to cap how many consecutive NaNs are filled.

# Fill at most 2 consecutive NaNs, leave longer gaps as-is
tspartial = mi.imputets(ts, strategy="linear", limit=2)

Gap inspection with gap_table:

from missingly.timeseries import gap_table
gt = gap_table(ts) print(gt) 
   column  gapstart   gapend  gap_length
 0    temp 2024-01-03 2024-01-04           2
 1    temp 2024-01-07 2024-01-07           1
 2    temp 2024-01-10 2024-01-12           3

sklearn Pipeline integration

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([     ("impute", mi.MissinglyImputer(strategy="knn")),     ("model",  LogisticRegression()), ]) pipe.fit(Xtrain, ytrain)

Installation

# Core package
pip install missingly
With interactive Plotly charts
pip install missingly[interactive]
With Persian / Arabic (RTL) support for static matplotlib plots
Required when column names or labels contain Persian/Arabic characters
pip install missingly[rtl]
Everything (interactive + RTL)
pip install missingly[all]