arianmokhtariha/data2prompt — 20 Stars

Topics

#ai #ai-tools #anthropic #claude #cli #code2prompt #command-line #command-line-tool #data-analysis #data-science #gemini #genai #gpt #lllm-tools #prompt #prompt-engineering #prompt-generator #prompt-toolkit

Language breakdown

Python 100.0%

▸ Files click to expand

README

High-performance codebase-to-prompt orchestration for Data Science workflows and data-heavy projects.

data2prompt is a CLI tool designed to bridge the gap between local data-heavy projects and Large Language Model (LLM) context windows. Unlike generic code-packagers, it provides an intelligent,optimized output for LLM attention mechanism, token-aware representation of a project's structure and content.

📝 Important Note

Data2prompt is purpose-built for data-heavy projects (.csv, .sql, .xlsx, .ipynb), not large pure-code repositories. It intelligently samples and truncates data files to prevent context window explosion while preserving semantic structure.

🎯 Why Data2Prompt?

Generic code-to-prompt tools choke on data files—they either skip them entirely or dump raw CSVs that waste 90% of your context window. Data2Prompt solves this with intelligent sampling, schema extraction, and LLM-optimized formatting specifically designed for data science workflows.

Data2Prompt Comparison

✨ Core Features

Smart Jupyter Parsing: Intelligently extracts code, markdown, and text outputs from .ipynb files while stripping heavy Base64 images and raw HTML to preserve context.
Multi-Format Sampling: Advanced sampling strategies for CSV, SQL, and Excel files to preserve schema and data context which reduces the data size significantly while extracting the needed context for llm.

Stats-Aware Metadata: Each table is annotated with a metadata block — column dtypes, missing counts/percentages, and a describe() summary — all computed on the full* dataset (not the sample), so the LLM sees true data quality. Toggle with --no-stats-summary. Secret-Safe .env Handling: .env files are surfaced as variable names* with redacted values (KEY=<redacted>) so the LLM understands the project's configuration without ever leaking secrets. Disable with --no-env-keys.

Direct Clipboard Output: --clipboard copies the generated prompt straight to your system clipboard (no file), using native OS tools with a graceful file fallback.
Aggressive truncations: To preserve context, long lines are truncated to neutralize line injections and avoid exploding the context windows, if a tabular data was still to large after sampling it will get truncated to a certain amount, also if a raw text file of unhandled type was too large it will get truncated to a certain amount.
Defensive Processing: Automatic binary detection (Null-byte checks), Checks if a file is binary by looking for a Null byte in the first 1024 bytes.
Optimized LLM attention: The default output format is markdown with well structured schema and another option is xml output with xml style tags to enhance LLM anchoring for complex analysis and large context windows
Token-Aware Output: Real-time token estimation using tiktoken (o200k_base) to ensure prompts fit target LLMs (Claude 3.5, GPT-4o, Gemini 1.5) and advanced offline token counting via regex.
Professional TUI: A high-fidelity terminal interface built with Rich, featuring a Matrix-style startup animation, a live progress bar, and a final scan report — identical on every platform.
Dynamic Markdown Wrapping: Uses intelligent backtick depth to ensure robust nesting of code blocks in the final output.
Gitignore aware: Respects the .gitignore rules by default and you can turn this feature off with cli argument(--no-gitignore) if needed.

🏗️ Architecture

The codebase is organized into small, single-responsibility modules — parsing, output generation, scanning, and UI are kept separate so each can be tested and extended on its own:

Registry & Strategy Patterns: A ParserRegistry handles extensible file

parsing and an OutputGenerator strategy supports multiple output formats (Markdown, XML).

Centralized Configuration: Core logic, magic numbers, and default ignore

lists live in one place: src/data2prompt/constants.py.

Strict Type Hinting: Fully typed function signatures (PEP 484) across all

modules.

UI Encapsulation: Terminal feedback is handled by a dedicated UIHandler,

keeping presentation separate from logic.

For a deep dive into the module layout and data flow, see the Architecture Documentation.

🚀 Quick Start

Installation

Ensure you have Python 3.10+ installed.

Recommended — using pipx (installs as a global CLI tool):

Don't have pipx? Install it first:

pip install pipx pipx ensurepath

Then install data2prompt:

pipx install data2prompt

Alternative — using pip (requires an active virtual environment):

pip install data2prompt

Update to the latest version:

# with pipx pipx upgrade data2prompt
with pip
pip install --upgrade data2prompt

Install from the source

# Clone the repository
git clone https://github.com/arianmokhtariha/data2prompt.git
cd data2prompt
Install normally
pip install .
Or Install in editable mode
pip install -e .

Optional: Parquet, Feather, and Arrow support

Support for .parquet, .feather, and .arrow files requires pyarrow, which is not bundled by default. Choose the command that matches how you installed data2prompt:

If pyarrow is not installed, these files still appear in the output with a short inline note explaining why they were skipped.

Usage

Run data2prompt in your project root to generate a structured prompt:

# Basic usage (defaults to markdown output)
data2prompt
Custom output with xml format and specific sampling
data2prompt --output myanalysis --format xml --csv-sample-size 50 --ignore-folders venv .pytestcache