Topics

#agent-use-ready #ai-agents #claude-code #claude-skills #doc-converter #docling #document-ai #document-processing #harness-engineering #knowledge-pipeline #llm #ocr #pdf-converter #pdf-parser #pdf-to-md #rag #research-papers #workflow

Language breakdown

Python 88.9%

Shell 11.1%

▸ Files click to expand

README

DocMeld

Lightweight Doc to agent-ready knowledge pipeline

Quick Start • Architecture • Python API • CLI • Configuration • Contributing

DocMeld converts PDF, Word, and PowerPoint documents into structured, agent-consumable formats through a three-stage pipeline — without requiring expensive OCR, VLM, or multimodal models. Built for the age of AI agents, it bridges the gap between static documents and the structured knowledge that LLMs need.

Most tools stop at format conversion. DocMeld goes further: Document → Structured Elements → Page Knowledge → AI-Enriched Metadata, producing outputs ready for RAG pipelines, agent systems, and downstream AI workflows.

Supported formats: .pdf, .docx, .doc (via LibreOffice), .pptx, .ppt (via LibreOffice).

Why DocMeld?

| | DocMeld | MinerU | Docling | Marker | MarkItDown | |---|---|---|---|---|---| | No ML models required | ✅ | ❌ | ❌ | ❌ | ✅ | | Runs fully offline (core) | ✅ | ❌ | ✅ | ✅ | ✅ | | Agent-ready outputs | ✅ | ❌ | ❌ | ❌ | ❌ | | AI metadata enrichment | ✅ | ❌ | ❌ | ❌ | ❌ | | Lightweight install | ✅ | ❌ | ❌ | ❌ | ✅ | | MIT license | ✅ | ❌ (AGPL) | ✅ | ❌ (GPL) | ✅ | | Swappable backends | ✅ | ❌ | N/A | ❌ | ❌ |

Quick Start

Installation

pip install docmeld

Optional backends for richer formats:

pip install docmeld[docling]   # Docling backend (DOCX + advanced PDF)
pip install docmeld[pptx]      # PowerPoint (.pptx) via python-pptx
pip install docmeld[office]    # Everything: Docling + python-pptx

Legacy .doc / .ppt additionally require LibreOffice (soffice) on your PATH.

Process your first document

from docmeld import DocMeldParser
parser = DocMeldParser("research_paper.pdf")   # or .docx / .pptx result = parser.process_all() print(f"Processed {result.successful}/{result.totalfiles} files in {result.processingtime_seconds}s")

Or from the command line:

docmeld process research_paper.pdf
docmeld process quarterly_deck.pptx --backend auto

That's it. Your document is now structured JSON, page-by-page JSONL, and (optionally) AI-enriched metadata.

Pipeline Architecture

DocMeld uses a three-stage medallion architecture. Each stage is independently runnable and idempotent — re-running skips already-processed files.

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│   BRONZE    │      │   SILVER    │      │    GOLD     │
│             │      │             │      │             │
│  Doc → JSON │─────▶│ JSON → JSONL│─────▶│ JSONL → AI  │
│  elements   │      │  pages      │      │  metadata   │
│             │      │             │      │             │
│  PyMuPDF /  │      │  Title      │      │  DeepSeek   │
│  Docling /  │      │  hierarchy  │      │  enrichment │
│  python-pptx│      │             │      │             │
└─────────────┘      └─────────────┘      └─────────────┘
   offline              offline            requires API key

Bronze: Document → Structured JSON

Extracts document elements (titles, text, tables, images, charts, formulas, SmartArt, speaker notes, comments, and more) into a unified JSON format with element IDs and parent-child hierarchy. The page unit is the physical page for PDF/Word and the slide for PowerPoint.

[
  {
    "type": "title",
    "level": 0,
    "content": "Executive Summary",
    "page_no": 1,
    "elementid": "e0001",
    "parent_id": ""
  },
  {
    "type": "text",
    "content": "The company reported strong Q2 results...",
    "page_no": 1,
    "elementid": "e0002",
    "parentid": "e0001"
  },
  {
    "type": "table",
    "content": "| Metric | Q1 | Q2 |\n|---|---|---|\n| Revenue | 10M | 15M |",
    "summary": "Items: Revenue",
    "page_no": 2,
    "elementid": "e0003",
    "parentid": "e0001",
    "table_data": {
      "headers": ["Metric", "Q1", "Q2"],
      "rows": [["Revenue", "10M", "15M"]],
      "num_rows": 1,
      "num_cols": 3
    }
  }
]

Supported element types:

| Type | Fields | Description | |---|---|---| | title | level, content | Headings with hierarchy (0–5) | | text | content | Paragraph content (hyperlinks preserved inline as text) | | table | content, summary, table_data | Markdown tables with structured data | | image | imagename, image, bbox, imageid | Base64-encoded images with metadata | | chart | chart_type, content, image | Chart data as a markdown table + image fallback | | formula | content, formula_type | Equations (LaTeX / OMML) | | smartart | smartart_type, content, image | SmartArt diagram text (PPTX) | | notes | content | Speaker notes (PPTX) | | group | content, childcount | Grouped shapes; children link via parentid (PPTX) | | comment | content, author | Reviewer comments (PPTX) | | footer | content, page_scope | Slide/page footers | | header / footnote / endnote | content, … | Document margins & notes (DOCX) |

All elements include pageno, elementid, and parent_id for cross-referencing. Elements on hidden slides carry hidden: true.

Silver: JSON → Page-by-Page JSONL

Transforms flat element lists into self-contained page documents with title hierarchy tracking, markdown rendering, and global table numbering.

{
  "metadata": {
    "uuid": "a1b2c3d4-...",
    "source": "research_paper.pdf",
    "page_no": "page1",
    "session_title": "# Executive Summary\n"
  },
  "page_content": "# Executive Summary\n\nThe company reported strong Q2 results...\n\n[[Table1]]\n| Metric | Q1 | Q2 |\n|---|---|---|\n| Revenue | 10M | 15M |\n[/Table1]"
}

Each page carries its full title context, so pages are independently meaningful — ideal for chunked retrieval in RAG systems.

Gold: JSONL → AI-Enriched Metadata

Adds semantic descriptions and keywords to each page using DeepSeek-chat, with exponential backoff retry and per-page error resilience.

{
  "metadata": {
    "uuid": "a1b2c3d4-...",
    "source": "research_paper.pdf",
    "page_no": "page1",
    "session_title": "# Executive Summary\n",
    "description": "Company reports strong Q2 results with 50% revenue growth",
    "keywords": ["revenue", "quarterly results", "growth", "financial performance"]
  },
  "page_content": "..."
}

The gold stage is optional — bronze and silver run fully offline with zero API calls.

Knowledge Generation (v0.3.1+)

Beyond per-page enrichment, DocMeld can generate structured, agent-ready artifacts from document content:

| Feature | CLI | Python API | Output | |---------|-----|-----------|--------| | Categorize | docmeld categorize papers/ | parser.process_categorize() | Topic clusters + categories.json | | PRD Generator | docmeld prd paper.pdf | parser.process_prd() | Product Requirements Document | | Workflow | docmeld workflow paper.pdf | parser.process_workflow() | Step-by-step implementation workflow | | Skills | docmeld skills book.pdf | parser.process_skills() | Claude Code skill files |

All four support swappable LLM backends via the LLMProvider Protocol (see Python API).

Output Structure

After processing research_paper.pdf:

research_paper.pdf                              # Original (untouched)
researchpapera3f5c2/                          # Output folder (name + MD5 suffix)
├── researchpapera3f5c2.json                  # Bronze: structured elements
├── researchpapera3f5c2.jsonl                 # Silver: page-by-page documents
└── researchpapera3f5c2_gold.jsonl            # Gold: AI-enriched (optional)

Output folder names are sanitized and include an MD5 hash suffix for uniqueness, ensuring safe cross-platform filenames even for PDFs with unicode or special characters.

Python API

Full Pipeline

from docmeld import DocMeldParser
Single file — all three stages
parser = DocMeldParser("paper.pdf")
result = parser.process_all()
Batch — process every PDF in a folder
parser = DocMeldParser("/path/to/papers/")
result = parser.process_all()
print(f"{result.successful}/{result.totalfiles} files, {result.processingtime_seconds}s")

Individual Stages

from docmeld import DocMeldParser
parser = DocMeldParser("paper.pdf")
Bronze only
bronze = parser.process_bronze()
print(f"{bronze.elementcount} elements across {bronze.pagecount} pages")
print(f"Output: {bronze.output_path}")
Silver (requires bronze output)
silver = parser.processsilver(bronze.outputpath)
print(f"{silver.pagecount} pages → {silver.outputpath}")
Gold (requires silver output + API key)
gold = parser.processgold(silver.outputpath)
print(f"{gold.pagesenriched} enriched, {gold.pagesfailed} failed")

Swappable Backends

DocMeld supports multiple parsing backends through a pluggable architecture. With --backend auto (default), the format is detected from the file extension and routed automatically:

# PDF (default): PyMuPDF — lightweight, fast
parser = DocMeldParser("paper.pdf", backend="pymupdf")
DOCX: Docling (IBM's ML-powered OOXML parser)
parser = DocMeldParser("report.docx", backend="docling")
PPTX: python-pptx — native slide/shape extraction
parser = DocMeldParser("deck.pptx", backend="pptx")
Legacy .doc / .ppt: LibreOffice bridge → PDF → PyMuPDF
parser = DocMeldParser("old.ppt", backend="soffice")
Auto-detect by extension (recommended)
parser = DocMeldParser("anything.pptx", backend="auto")

| Backend | Formats | Requires | |---|---|---| | pymupdf | .pdf | core install | | docling | .docx, .pdf | docmeld[docling] | | pptx | .pptx | docmeld[pptx] | | soffice | .doc, .ppt | LibreOffice | | auto | all of the above | per-format |

Working with Elements

import json
Load bronze output
with open("papera3f5c2/papera3f5c2.json") as f:
    elements = json.load(f)
Filter by type
titles = [e for e in elements if e["type"] == "title"]
tables = [e for e in elements if e["type"] == "table"]
Navigate hierarchy via parent_id
for elem in elements:
    if elem["parentid"] == "e001":
        print(f"  Child of first title: {elem['content'][:50]}")
Access structured table data
for table in tables:
    headers = table["table_data"]["headers"]
    rows = table["table_data"]["rows"]
    print(f"Table: {len(rows)} rows × {len(headers)} cols")

Knowledge Generation

# Categorize papers into topic clusters
parser = DocMeldParser("/path/to/papers/")
result = parser.process_categorize(reorganize=False)
Generate a PRD from a research paper
prd = parser.process_prd()
Extract a workflow
wf = parser.process_workflow()
Extract Claude Code skills from a book
skills = parser.process_skills()

Swappable LLM Provider

from docmeld.gold.provider import LLMProvider
class MyProvider:     def extract_metadata(self, content): ...     def generate(self, prompt): ...     def categorize(self, prompt): ...
parser = DocMeldParser("paper.pdf", provider=MyProvider()) prd = parser.process_prd()  # uses your provider, not DeepSeek

When no provider is given, a DeepSeekClient is constructed from environment variables — behavior is unchanged.

Result Models

All pipeline stages return typed Pydantic models:

BronzeResult(outputpath, outputdir, elementcount, pagecount, skipped)
SilverResult(outputpath, pagecount, skipped)
GoldResult(outputpath, pagesenriched, pages_failed, skipped)
ProcessingResult(totalfiles, successful, failed, failures, processingtime_seconds, ...)
CategorizeResult(indexpath, totalpapers, totalcategories, papersfailed, reorganized)
PrdResult(outputpath, sections, sourcepdf, skipped)
WorkflowResult(outputpath, sections, sourcepdf, skipped)
SkillsResult(outputdir, skillcount, source_pdf, skipped)

CLI Reference

# Full pipeline (bronze → silver → gold)
docmeld process paper.pdf
docmeld process /path/to/papers/
Individual stages
docmeld bronze paper.pdf                    # Doc → JSON
docmeld silver papera3f5c2/papera3f5c2.json    # JSON → JSONL
docmeld gold papera3f5c2/papera3f5c2.jsonl     # JSONL → enriched JSONL
Choose parsing backend
docmeld bronze report.docx --backend docling
docmeld bronze deck.pptx   --backend pptx
docmeld process paper.pdf  --backend auto         # default — detects format
Knowledge Generation (requires DEEPSEEKAPIKEY)
docmeld categorize /path/to/papers/               # Topic clustering → categories.json
docmeld categorize /path/to/papers/ --reorganize  # Move files into category folders
docmeld prd paper.pdf                             # Generate Product Requirements Document
docmeld workflow paper.pdf                        # Extract step-by-step workflow
docmeld skills book.pdf                           # Extract Claude Code skills

Configuration

Gold Stage (AI Enrichment)

Create a .env.local file in your working directory:

DEEPSEEKAPIKEY=yourkeyhere
Optional: custom API endpoint
DEEPSEEKAPIENDPOINT=https://api.deepseek.com

The gold stage is entirely optional. Bronze and silver stages run offline with no API keys, no network calls, and no model downloads.

Logging

DocMeld writes timestamped log files (docmeldYYYYMMDDHHMMSS.log) to the working directory. Console output shows INFO-level messages; log files capture full DEBUG output.

Unified Element Schema

DocMeld enforces a strict element schema via Pydantic models. This contract guarantees downstream consumers always get a predictable structure.

from docmeld.bronze.element_types import (
    TitleElement,    # type, level, content, pageno, elementid, parent_id
    TextElement,     # type, content, pageno, elementid, parent_id
    TableElement,    # type, content, summary, pageno, elementid, parentid, tabledata
    ImageElement,    # type, imagename, content, image, imageid, bbox, ...
    ChartElement,    # type, charttype, content, image, imagename, ...
    FormulaElement,  # type, content, formula_type, ...
    SmartArtElement, # type, smartart_type, content, image, ...
    NotesElement,    # type, content, ...            (speaker notes)
    GroupElement,    # type, content, child_count, ...
    CommentElement,  # type, content, author, ...
    # + HeaderElement, FooterElement, FootnoteElement, EndnoteElement
)

Element types are validated at creation time. All 14 types share type, pageno, elementid, parent_id, and an optional hidden flag. New types may be added in minor versions, but existing types will never change shape in minor/patch releases.

Roadmap

[x] Bronze → Silver → Gold pipeline
[x] CLI interface with subcommands
[x] Swappable backends (PyMuPDF + Docling)
[x] Element hierarchy (elementid / parentid)
[x] Structured table data extraction
[x] Idempotent processing
[x] Batch folder processing
[x] DOCX support (Docling backend)
[x] PPTX / PPT support (python-pptx + LibreOffice bridge)
[x] Rich element types (chart, formula, SmartArt, notes, comments, groups)
[x] Research paper batch categorization + topic clustering
[x] Paper-to-PRD generation
[x] Paper-to-workflow extraction
[x] Book-to-Claude-Skills generation
[x] Swappable LLM provider
[ ] OCR for scanned PDFs (pip install docmeld[ocr])
[ ] Agent prompt generation
[ ] LangChain / LlamaIndex integration

Development

Setup

git clone https://github.com/agentii-ai/docmeld.git
cd docmeld
python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

Quality Gates

pytest tests/ -v --cov=docmeld       # 315 tests, 78% coverage
ruff check docmeld/                   # Linting
black --check docmeld/                # Formatting
mypy docmeld/                         # Strict type checking

Project Structure

docmeld/
├── docmeld/
│   ├── init.py              # Public API (DocMeldParser, version)
│   ├── parser.py                # Pipeline orchestrator
│   ├── cli.py                   # CLI entry point (argparse)
│   ├── bronze/
│   │   ├── backends/
│   │   │   ├── pymupdf_backend.py   # PyMuPDF + pymupdf4llm
│   │   │   ├── docling_backend.py   # Docling (optional, DOCX/PDF)
│   │   │   ├── pptx_backend.py      # python-pptx (PPTX slide extraction)
│   │   │   └── soffice_backend.py   # LibreOffice bridge (.doc/.ppt)
│   │   ├── element_extractor.py     # Extraction + post-processing
│   │   ├── element_types.py         # Pydantic element models
│   │   ├── filename_sanitizer.py    # Safe filenames + MD5 hashing
│   │   └── processor.py            # Bronze orchestrator
│   ├── silver/
│   │   ├── page_aggregator.py       # Group elements by page
│   │   ├── page_models.py          # Result models (Pydantic)
│   │   ├── markdown_renderer.py     # Elements → markdown
│   │   ├── title_tracker.py         # Title hierarchy state
│   │   └── processor.py            # Silver orchestrator
│   ├── gold/
│   │   ├── deepseek_client.py       # API client + retry logic
│   │   ├── provider.py              # LLMProvider Protocol (swappable)
│   │   ├── metadata_extractor.py    # Content → description + keywords
│   │   └── processor.py            # Gold orchestrator
│   ├── categorize/                  # Topic clustering (categorize)
│   ├── prd/                         # PRD generation (prd)
│   ├── workflow/                    # Workflow extraction (workflow)
│   ├── skills/                      # Skills extraction (skills)
│   └── utils/
│       ├── env_loader.py            # .env.local loading
│       ├── logging.py               # Timestamped log setup
│       ├── progress.py              # Progress indicators
│       ├── silver_io.py             # Shared JSONL loading
│       ├── content.py               # Content aggregation
│       └── text.py                  # Text helpers
├── scripts/                     # Example scripts (not shipped in package)
├── tests/                       # Unit, integration, contract tests
├── pyproject.toml
├── CONTRIBUTING.md
├── CHANGELOG.md
└── LICENSE                      # MIT

Contributing

We welcome contributions. See CONTRIBUTING.md for the full guide. The short version:

Fork and clone
Write tests first (TDD is non-negotiable)
Run all quality gates before pushing
Open a PR with a clear description

License

MIT License — see LICENSE for details.

Citation

@software{docmeld2026,
  title     = {DocMeld: Lightweight PDF, Word & PowerPoint to Agent-Ready Knowledge Pipeline},
  year      = {2026},
  license   = {MIT},
  url       = {https://github.com/agentii-ai/docmeld}
}

🔗 More in this category