sklearn-diagnose

An intelligent diagnosis layer for scikit-learn: evidence-based model failure detection with LLM-powered summaries.

Philosophy

This library uses LLM-powered analysis for model diagnosis. All hypotheses are probabilistic and evidence-based.

sklearn-diagnose acts as an "MRI scanner" for your machine learning models — it diagnoses problems but never modifies your models. The library follows an evidence-first, LLM-powered approach:

• Signal Extractors: Compute deterministic statistics from your model and data
• LLM Hypothesis Generation: Detect failure modes with confidence scores and severity
• LLM Recommendation Generation: Generate actionable recommendations based on detected issues
• LLM Summary Generation: Create human-readable summaries

Key Features

• Model Failure Diagnosis: Detect overfitting, underfitting, high variance, label noise, feature redundancy, class imbalance, and data leakage symptoms
• Interactive Chatbot: Launch a web-based chatbot to have conversations about your diagnosis results
• Cross-Validation Interpretation: CV interpretation is a core signal extractor within sklearn-diagnose, used to detect instability, overfitting, and potential data leakage
• Evidence-Based Hypotheses: All diagnoses include confidence scores and supporting evidence
• Actionable Recommendations: Get specific suggestions to fix identified issues
• Read-Only Behavior: Never modifies your estimator, parameters, or data
• Universal Compatibility: Works with any fitted scikit-learn estimator or Pipeline

Installation

pip install sklearn-diagnose

This installs sklearn-diagnose with all required dependencies including:

• LangChain (v1.2.0+) for AI agent capabilities
• langchain-openai for OpenAI model support
• langchain-anthropic for Anthropic model support
• python-dotenv for environment variable management

Interactive Chatbot Included

The interactive chatbot feature is included by default! When you install sklearn-diagnose, you get:

• FastAPI for the web server
• Uvicorn for running the server
• python-multipart for form handling
• Bundled React frontend - no Node.js or npm required!

Quick Start

from sklearn.linear_model import LogisticRegression
from sklearn.modelselection import traintest_split
from sklearndiagnose import setupllm, diagnose
Set up LLM (REQUIRED - must specify provider, model, and api_key)
Using OpenAI:
setupllm(provider="openai", model="gpt-4o", apikey="your-openai-key")
setupllm(provider="openai", model="gpt-4o-mini", apikey="your-openai-key")
Or using Anthropic:
setupllm(provider="anthropic", model="claude-3-5-sonnet-latest", apikey="your-anthropic-key")
Or using OpenRouter (access to many models):
setupllm(provider="openrouter", model="deepseek/deepseek-r1-0528", apikey="your-openrouter-key")
Your existing sklearn workflow
Xtrain, Xval, ytrain, yval = traintestsplit(X, y, test_size=0.2)
model = LogisticRegression()
model.fit(Xtrain, ytrain)
Diagnose your model
report = diagnose(
    estimator=model,
    datasets={
        "train": (Xtrain, ytrain),
        "val": (Xval, yval)
    },
    task="classification"
)
View results
print(report.summary())          # LLM-generated summary
print(report.hypotheses)         # Detected issues with confidence
print(report.recommendations)    # LLM-ranked actionable suggestions

With a Pipeline

import os
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.compose import ColumnTransformer
from sklearndiagnose import setupllm, diagnose
Set up LLM (required - do this once at startup)
os.environ["OPENAIAPIKEY"] = "your-key"
setupllm(provider="openai", model="gpt-4o")  # apikey optional when env var set
Build your pipeline
preprocessor = ColumnTransformer([
    ("num", StandardScaler(), numerical_cols),
])
pipeline = Pipeline([     ("preprocess", preprocessor),     ("model", LogisticRegression()) ]) pipeline.fit(Xtrain, ytrain)
Diagnose works with any estimator
report = diagnose(
    estimator=pipeline,
    datasets={
        "train": (Xtrain, ytrain),
        "val": (Xval, yval)
    },
    task="classification"
)

With Cross-Validation Results

from sklearn.modelselection import crossvalidate
Run cross-validation
cvresults = crossvalidate(
    model, Xtrain, ytrain,
    cv=5,
    returntrainscore=True,
    scoring='accuracy'
)
Diagnose with CV evidence (no holdout set needed)
report = diagnose(
    estimator=model,
    datasets={
        "train": (Xtrain, ytrain)
    },
    task="classification",
    cvresults=cvresults
)

Interactive Chatbot

Launch an interactive web-based chatbot to explore your diagnosis results through natural conversation with an LLM.

Features

• Interactive Q&A: Ask questions about your diagnosis results in natural language
• Full Context: The chatbot has complete access to all detected issues, recommendations, and model signals
• Code Examples: Get implementation help with ready-to-use code snippets
• Conversation History: Maintains context throughout your session
• Markdown Rendering: Formatted responses with syntax highlighting
• Responsive UI: Modern React interface with Tailwind CSS

The chatbot dependencies (FastAPI, Uvicorn, python-multipart) are included by default. The React frontend is bundled - no Node.js or npm required!

Usage

Just ONE terminal, ONE Python script:

from sklearndiagnose import setupllm, diagnose, launch_chatbot
1. Configure LLM
setupllm(provider="openai", model="gpt-4o", apikey="sk-...")
2. Diagnose your model
report = diagnose(
    estimator=model,
    datasets={"train": (Xtrain, ytrain), "val": (Xval, yval)},
    task="classification"
)
3. Launch chatbot (opens browser automatically)
launch_chatbot(report)

That's it! The browser opens automatically to http://localhost:8000 and you can start chatting.

Works on both Windows and Mac/Linux - no platform-specific setup needed!

Complete Example

Run the provided example script:

# On Windows
python tests/example_diagnose.py
On Mac/Linux
python3 tests/example_diagnose.py

This will:

• Generate synthetic test data with deliberate issues
• Train a model
• Run diagnosis
• Launch the chatbot automatically

Example Questions

Once the chatbot is running, try asking:

• "What are the main issues with my model?"
• "How do I fix the class imbalance?"
• "Show me code to implement your first recommendation"
• "Why is feature redundancy a problem?"
• "What causes overfitting in my case?"
• "How do I tune the decision threshold?"

Chatbot Architecture

Browser (http://localhost:8000)
         ↓
FastAPI Server (serves both frontend & API)
  ├── /assets/*     → Static files (JS, CSS)
  ├── /api/*        → REST API endpoints
  └── /*            → React frontend (SPA)
         ↓
ChatAgent (maintains conversation history)
         ↓
LLM Client (OpenAI/Anthropic/OpenRouter)

Troubleshooting

Chat responses not working:

• Verify you called setup_llm() before diagnose()
• Check your API key is valid in .env file or environment variables

Port already in use:

• Default port is 8000
• Change if needed: launch_chatbot(report, port=9000)

Browser doesn't open automatically:

• Manually navigate to http://localhost:8000

"Frontend not built" error:

• This shouldn't happen with pip install
• If developing from source, run: cd frontend && npm run build

Customization

Configure the chatbot server:

launch_chatbot(
    report,
    host="127.0.0.1",       # Server host
    port=8000,               # Server port
    autoopenbrowser=True   # Auto-open browser
)

Detected Failure Modes

| Failure Mode | What It Detects | Key Signals | |--------------|-----------------|-------------| | Overfitting | Model memorizes training data | High train score, low val score, large gap | | Underfitting | Model too simple for data | Low train and val scores | | High Variance | Unstable across data splits | High CV fold variance, inconsistent predictions | | Label Noise | Incorrect/noisy target labels | Ceilinged train score, scattered residuals | | Feature Redundancy | Correlated/duplicate features | Detailed correlated pair list with correlation values | | Class Imbalance | Skewed class distribution | Class distribution, per-class recall/precision, recall disparity | | Data Leakage | Information from future/val in train | CV-to-holdout gap, suspicious feature-target correlations |

Output Format

report = diagnose(...)
Human-readable summary (includes both diagnosis and recommendations)
report.summary()
"## Diagnosis
 Based on the analysis, here are the key findings:
 - Overfitting (95% confidence, high severity)
   - Train-val gap of 25.3% indicates overfitting
 - Feature Redundancy (90% confidence, high severity)
   - 4 highly correlated feature pairs detected (max correlation: 99.9%)
   - Correlated feature pairs:
   -   - Feature 0 ↔ Feature 10: 99.9% correlation
   -   - Feature 1 ↔ Feature 11: 99.8% correlation
 
 ## Recommendations
 1. Increase regularization strength
    Stronger regularization penalizes model complexity..."
Structured hypotheses with confidence scores
report.hypotheses
[
  Hypothesis(name=FailureMode.OVERFITTING, confidence=0.85, 
             evidence=['Train-val gap of 23.0% is severe'], severity='high'),
  Hypothesis(name=FailureMode.FEATURE_REDUNDANCY, confidence=0.90,
             evidence=['4 highly correlated pairs detected',
                       'Correlated feature pairs:',
                       '  - Feature 0 ↔ Feature 10: 99.9% correlation',
                       '  - Feature 1 ↔ Feature 11: 99.8% correlation'],
             severity='high')
]
Access hypothesis details
h = report.hypotheses[0]
h.name.value        # 'overfitting' (string)
h.confidence        # 0.85
h.evidence          # ['Train-val gap of 23.0% is severe']
h.severity          # 'high'
Actionable recommendations (Recommendation objects)
report.recommendations
[
  Recommendation(action='Increase regularization strength', 
                 rati,
                 related_hypothesis=FailureMode.OVERFITTING),
  Recommendation(action='Reduce model complexity', 
                 rati,
                 related_hypothesis=FailureMode.OVERFITTING)
]
Access recommendation details
r = report.recommendations[0]
r.action              # 'Increase regularization strength'
r.rationale           # 'Stronger regularization penalizes...'
r.related_hypothesis  # FailureMode.OVERFITTING
Raw signals (Signals object with attribute access)
report.signals.train_score      # 0.94
report.signals.val_score        # 0.71
report.signals.cv_mean          # 0.73 (if CV provided)
report.signals.cv_std           # 0.12 (if CV provided)
report.signals.to_dict()        # Convert to dict for serialization

Design Principles

Evidence-Based Diagnosis

Every hypothesis is backed by quantitative evidence. The LLM analyzes deterministic signals and generates hypotheses with confidence scores

Confidence Scoring & Guardrails

• All hypotheses include explicit confidence scores (0.0 - 1.0)
• "Insufficient evidence" responses when signals are ambiguous
• Uncertainty is communicated clearly, never hidden
• No model changes are suggested automatically

Read-Only Guarantee

sklearn-diagnose never:

• Calls .fit() on your estimator
• Modifies estimator parameters
• Mutates your training data
• Refits or retrains models

Validation Set vs Cross-Validation

sklearn-diagnose follows strict rules:

• y_val is OPTIONAL — You can diagnose with only training data + CV results
• CV evidence overrides holdout logic — When both present, CV provides richer signals
• Never mix the two — Holdout and CV answer different questions

API Reference

diagnose()

Main entry point for model diagnosis.

def diagnose(
    estimator,              # Any fitted sklearn estimator or Pipeline
    datasets: dict,         # {"train": (X, y), "val": (X, y)} - val is optional
    task: str,              # "classification" or "regression"
    cvresults: dict = None # Output from crossvalidate() - optional
) -> DiagnosisReport:

Parameters:

• estimator: A fitted scikit-learn estimator or Pipeline. Must already be fitted.
• datasets: Dictionary with "train" key required, "val" key optional. Each value is a tuple of (X, y).
• task: Either "classification" or "regression"
• cvresults: Optional dictionary from sklearn.modelselection.cross_validate()

Returns:

DiagnosisReport object with:

• .hypotheses: List of detected issues with confidence scores
• .recommendations: List of actionable fix suggestions (LLM-ranked)
• .signals: Raw computed statistics
• .summary(): Human-readable summary (LLM-generated)

Configuration

LLM Backend (Required)

sklearn-diagnose uses LangChain under the hood for LLM integration. Each diagnosis involves three AI agents:

• Hypothesis Agent: Analyzes signals and detects failure modes
• Recommendation Agent: Generates actionable fix suggestions
• Summary Agent: Creates human-readable summaries

from sklearndiagnose import setupllm

Using OpenAI
setupllm(provider="openai", model="gpt-4o", apikey="sk-...")
Using Anthropic
setupllm(provider="anthropic", model="claude-3-5-sonnet-latest", apikey="sk-ant-...")
Using OpenRouter (access to many models)
setupllm(provider="openrouter", model="deepseek/deepseek-r1-0528", apikey="sk-or-...")

Using Environment Variables

You can set API keys via environment variables in two ways:

Option 1: Set programmatically in Python

import os
from sklearndiagnose import setupllm
Set environment variable in your code
os.environ["OPENAIAPIKEY"] = "sk-..."
setupllm(provider="openai", model="gpt-4o")  # apikey is automatically loaded
Or for Anthropic
os.environ["ANTHROPICAPIKEY"] = "sk-ant-..."
setup_llm(provider="anthropic", model="claude-3-5-sonnet-latest")
Or for OpenRouter
os.environ["OPENROUTERAPIKEY"] = "sk-or-..."
setup_llm(provider="openrouter", model="deepseek/deepseek-r1-0528")

Option 2: Use a .env file (recommended for production)

Create a .env file in your project root:

# .env file
OPENAIAPIKEY=sk-...
ANTHROPICAPIKEY=sk-ant-...
OPENROUTERAPIKEY=sk-or-...

The library uses python-dotenv internally to automatically load the .env file (no need to import or call load_dotenv() yourself):

from sklearndiagnose import setupllm
API keys are automatically loaded from .env file
setup_llm(provider="openai", model="gpt-4o")
setup_llm(provider="anthropic", model="claude-3-5-sonnet-latest")
setup_llm(provider="openrouter", model="deepseek/deepseek-r1-0528")

Architecture

sklearn-diagnose/                 # Project root
├── sklearn_diagnose/             # Main package
│   ├── init.py               # Package exports (setupllm, diagnose, launchchatbot, types)
│   ├── api/
│   │   ├── init.py
│   │   └── diagnose.py           # Main diagnose() function
│   ├── core/
│   │   ├── init.py
│   │   ├── schemas.py            # Data structures (Evidence, Signals, Hypothesis, etc.)
│   │   ├── evidence.py           # Input validation, read-only guarantees
│   │   ├── signals.py            # Signal extraction (deterministic metrics)
│   │   ├── hypotheses.py         # Rule-based hypotheses (fallback/reference)
│   │   └── recommendations.py    # Example recommendation templates for LLM guidance
│   ├── llm/
│   │   ├── init.py           # Exports setup_llm and LLM utilities
│   │   └── client.py             # LangChain-based AI agents (hypothesis, recommendation, summary)
│   ├── server/                   # Chatbot backend (NEW)
│   │   ├── init.py
│   │   ├── app.py                # FastAPI application with CORS and routes
│   │   └── chat_agent.py         # ChatAgent for conversation management
│   └── chatbot.py                # Chatbot launcher function
├── frontend/                     # React chatbot UI (NEW)
│   ├── src/
│   │   ├── components/           # React components (Header, ChatInterface, etc.)
│   │   ├── hooks/                # Custom hooks (useChat)
│   │   ├── services/             # API client
│   │   ├── App.jsx               # Main React app
│   │   ├── main.jsx              # React entry point
│   │   └── index.css             # Global styles with Tailwind
│   ├── package.json              # Node dependencies
│   ├── vite.config.js            # Vite configuration with API proxy
│   ├── tailwind.config.js        # Tailwind CSS config
│   └── index.html                # HTML entry point
├── tests/
│   ├── init.py
│   ├── conftest.py               # Pytest fixtures and MockLLMClient for testing
│   ├── unittestdiagnose.py     # Comprehensive test suite (includes chatbot tests)
│   └── example_diagnose.py       # Example script demonstrating full workflow with chatbot
├── .github/
│   └── workflows/
│       └── tests.yml             # GitHub Actions CI (runs tests on push/PR)
├── .env.example                  # Template for API keys (copy to .env)
├── .gitignore
├── AGENTS.md                     # AI agents architecture documentation
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
├── MANIFEST.in
├── README.md
└── pyproject.toml

Processing Flow

User Input (model, data, task)
         │
         ▼
┌─────────────────────────────┐
│  1. Signal Extraction       │  Deterministic metrics
│     (signals.py)            │  (trainscore, valscore, cv_std, etc.)
└─────────────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  2. Hypothesis Agent        │  Failure modes with confidence & severity
│     (LangChain create_agent)│  (overfitting: 95%, high severity)
└─────────────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  3. Recommendation Agent    │  Actionable recommendations
│     (LangChain create_agent)│  (guided by example templates)
└─────────────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│  4. Summary Agent           │  Human-readable summary
│     (LangChain create_agent)│
└─────────────────────────────┘
         │
         ▼
    DiagnosisReport

Contributing

Contributions are welcome! Please read our Contributing Guidelines before submitting pull requests.

License

MIT License - see LICENSE for details.

Citation

If you use sklearn-diagnose in your research, please cite:

@software{sklearn_diagnose,
  title = {sklearn-diagnose: Evidence-based model failure diagnosis for scikit-learn},
  year = {2025},
  url = {https://github.com/leockl/sklearn-diagnose}
}

---

Please give my GitHub repo a ⭐ if this was helpful. Thank you! 🙏