What is Claw Compactor?

Claw Compactor is an open-source LLM token compression engine built around a 14-stage Fusion Pipeline. Each stage is a specialized compressor — from AST-aware code analysis to JSON statistical sampling to simhash-based deduplication — chained through an immutable data flow architecture where each stage's output feeds the next.

Demo

$ claw-compactor benchmark ./my-workspace
Claw Compactor v7.0 — Fusion Pipeline Benchmark   ─────────────────────────────────────────────────
Scanning workspace... 47 files, 234,891 tokens
Stage Results:   ┌──────────────────┬──────────┬───────────┬──────────┐   │ Stage            │ Applied  │ Reduction │ Time     │   ├──────────────────┼──────────┼───────────┼──────────┤   │ Cortex           │ 47/47    │ —         │ 12ms     │   │ Photon           │ 3/47     │ 2.1%      │ 4ms      │   │ RLE              │ 41/47    │ 8.3%      │ 6ms      │   │ SemanticDedup    │ 47/47    │ 12.7%     │ 18ms     │   │ Ionizer          │ 8/47     │ 71.2%     │ 9ms      │   │ Neurosyntax      │ 23/47    │ 18.4%     │ 31ms     │   │ TokenOpt         │ 47/47    │ 4.1%      │ 3ms      │   │ Abbrev           │ 12/47    │ 6.8%      │ 5ms      │   └──────────────────┴──────────┴───────────┴──────────┘
Summary:     Before:  234,891 tokens ($2.35 at GPT-4 rates)     After:   108,250 tokens ($1.08)     Saved:   126,641 tokens (53.9%) — $1.27/run     Time:    88ms total
Estimated monthly savings at 100 runs/day: $3,810

How It Compares

| Feature | Claw Compactor | LLMLingua-2 | SelectiveContext | gzip + base64 | |:--------|:-:|:-:|:-:|:-:| | Compression rate | 15–82% | 30–70% | 10–40% | 60–80% | | ROUGE-L @ 0.3 | 0.653 | 0.346 | ~0.4 | N/A | | ROUGE-L @ 0.5 | 0.723 | 0.570 | ~0.6 | N/A | | LLM inference cost | $0 | ~$0.02/call | $0 | $0 | | Latency | <50ms | ~300ms | ~200ms | <10ms | | Reversible | Yes | No | No | Yes (manual) | | Content-aware routing | 14 stages | 1 (perplexity) | 1 (self-info) | None | | AST-aware code handling | Yes (tree-sitter) | No | No | No | | JSON schema sampling | Yes | No | No | No | | Log/diff/search stages | Yes | No | No | No | | Required dependencies | 0 | torch, transformers | torch | zlib | | LLM-readable output | Yes | Partial | Partial | No |

Why Claw Compactor wins: LLMLingua-2 drops tokens by perplexity score — effective for natural language, but destroys code identifiers, JSON keys, and log patterns. Claw Compactor uses content-type-aware stages that understand the structure of what they're compressing.

Input
  |
  v
┌─────────────────────────────────────────────────────────────────────────┐
│                         FUSION PIPELINE                                 │
│                                                                         │
│  QuantumLock ─> Cortex ─> Photon ─> RLE ─> SemanticDedup ─> Ionizer    │
│       |            |         |        |          |              |        │
│   KV-cache    auto-detect  base64   path     simhash       JSON         │
│   alignment   16 languages  strip  shorten   dedup        sampling      │
│                                                                         │
│  ─> LogCrunch ─> SearchCrunch ─> DiffCrunch ─> StructuralCollapse      │
│        |              |              |                |                  │
│    log folding    result dedup   context fold    import merge            │
│                                                                         │
│  ─> Neurosyntax ─> Nexus ─> TokenOpt ─> Abbrev ─────────> Output       │
│        |             |          |           |                            │
│    AST compress   ML token   format     NL shorten                      │
│    (tree-sitter)  classify   optimize   (text only)                     │
│                                                                         │
│  [ RewindStore ] ── hash-addressed LRU for reversible retrieval         │
└─────────────────────────────────────────────────────────────────────────┘

Key design principles:

• Immutable data flow — FusionContext is a frozen dataclass. Every stage produces a new FusionResult; nothing is mutated in-place.
• Gate-before-compress — Each stage has should_apply() that inspects context type, language, and role before doing any work. Stages that don't apply are skipped at zero cost.
• Content-aware routing — Cortex auto-detects content type (code, JSON, logs, diffs, search results) and language (Python, Go, Rust, TypeScript, etc.), then downstream stages make type-aware compression decisions.
• Reversible compression — Ionizer stores originals in a hash-addressed RewindStore. The LLM can call a tool to retrieve any compressed section by its marker ID.

Benchmarks

Real-World Compression (FusionEngine v7 vs Legacy Regex)

| Content Type | Legacy | FusionEngine | Improvement | |:-------------|-------:|-------------:|:-----------:| | Python source | 7.3% | 25.0% | 3.4x | | JSON (100 items) | 12.6% | 81.9% | 6.5x | | Build logs | 5.5% | 24.1% | 4.4x | | Agent conversation | 5.7% | 31.0% | 5.4x | | Git diff | 6.2% | 15.0% | 2.4x | | Search results | 5.3% | 40.7% | 7.7x | | Weighted average | 9.2% | 36.3% | 3.9x |

SWE-bench Real Tasks

Tested on real SWE-bench instances with actual repository code:

| Instance | Size | Compression | |:---------|-----:|------------:| | django__django-11620 | 4.5K | 14.5% | | sympy__sympy-14396 | 5.5K | 19.1% | | scikit-learn-25747 | 11.8K | 15.9% | | scikit-learn-13554 | 73K | 11.8% | | scikit-learn-25308 | 81K | 14.4% |

vs LLMLingua-2 (ROUGE-L Fidelity)

| Compression Rate | Claw Compactor | LLMLingua-2 | Delta | |:-----------------|---------------:|------------:|------:| | 0.3 (aggressive) | 0.653 | 0.346 | +88.2% | | 0.5 (balanced) | 0.723 | 0.570 | +26.8% |

Claw Compactor preserves more semantic content at the same compression ratio, with zero LLM inference cost.

Quick Start

Install from PyPI

pip install claw-compactor

Or clone from source

git clone https://github.com/open-compress/claw-compactor.git
cd claw-compactor
pip install -e .

Run

# Benchmark your workspace (non-destructive)
claw-compactor benchmark /path/to/workspace
Full compression pipeline
claw-compactor compress /path/to/workspace

Requirements: Python 3.9+. Optional: pip install claw-compactor[accurate] for exact token counts via tiktoken.

API

FusionEngine — Single Text

from scripts.lib.fusion.engine import FusionEngine
engine = FusionEngine()
result = engine.compress(     text="def hello():\n    # greeting function\n    print('hello')",     c,    # or let Cortex auto-detect     language="python",      # optional hint )
print(result["compressed"])     # compressed output print(result["stats"])          # per-stage timing + token counts print(result["markers"])        # Rewind markers for reversibility

FusionEngine — Chat Messages

messages = [
    {"role": "system", "content": "You are a coding assistant..."},
    {"role": "user", "content": "Fix the auth bug in login.py"},
    {"role": "assistant", "content": "I found the issue. Here's the fix:\n

result = engine.compress_messages(messages)

Cross-message dedup runs first, then per-message pipeline

### Rewind — Reversible Retrieval

LLM sees compressed output with markers like [rewind:abc123...]

When the LLM needs the original, it calls the Rewind tool:

### Custom Stage

class MyStage(FusionStage): name = "my_compressor" order = 22 # runs between StructuralCollapse (20) and Neurosyntax (25)

def should_apply(self, ctx: FusionContext) -> bool: return ctx.content_type == "log"

def apply(self, ctx: FusionContext) -> FusionResult: compressed = mycompressionlogic(ctx.content) return FusionResult( content=compressed, originaltokens=estimatetokens(ctx.content), compressedtokens=estimatetokens(compressed), )

Add to pipeline

---
The 14 Stages
| # | Stage | Order | Purpose | Applies To | |:-:|:------|:-----:|:--------|:-----------| | 1 | QuantumLock | 3 | Isolates dynamic content in system prompts to maximize KV-cache hit rate | system messages | | 2 | Cortex | 5 | Auto-detects content type and programming language (16 languages) | untyped content | | 3 | Photon | 8 | Detects and compresses base64-encoded images | all | | 4 | RLE | 10 | Path shorthand ($WS), IP prefix compression, enum compaction | all | | 5 | SemanticDedup | 12 | SimHash fingerprint deduplication across content blocks | all | | 6 | Ionizer | 15 | JSON array statistical sampling with schema discovery + error preservation | json | | 7 | LogCrunch | 16 | Folds repeated log lines with occurrence counts | log | | 8 | SearchCrunch | 17 | Deduplicates search/grep results | search | | 9 | DiffCrunch | 18 | Folds unchanged context lines in git diffs | diff | | 10 | StructuralCollapse | 20 | Merges import blocks, collapses repeated assertions/patterns | code | | 11 | Neurosyntax | 25 | AST-aware code compression via tree-sitter (safe regex fallback). Never shortens identifiers. | code | | 12 | Nexus | 35 | ML token-level compression (stopword removal fallback without model) | text | | 13 | TokenOpt | 40 | Tokenizer format optimization — strips bold/italic markers, normalizes whitespace | all | | 14 | Abbrev | 45 | Natural language abbreviation. Only fires on text — never touches code, JSON, or structured data. | text |
Each stage is independent and stateless. Stages communicate only through the immutable FusionContext that flows forward through the pipeline.

Workspace Commands

| Command | Description |
|:--------|:-----------|
| full | Run complete compression pipeline |
| benchmark | Dry-run compression report |
| compress | Rule-based compression only |
| dict | Dictionary encoding with auto-learned codebook |
| observe | Session transcript JSONL to structured observations |
| tiers | Generate L0/L1/L2 tiered summaries |
| dedup | Cross-file duplicate detection |
| estimate | Token count report |
| audit | Workspace health check |
| optimize | Tokenizer-level format optimization |
| auto | Watch mode — compress on file changes |
Options: --json, --dry-run, --since YYYY-MM-DD, --quiet

Architecture
See ARCHITECTURE.md for the full technical deep-dive: 
Immutable data flow design
 
Stage execution model and gating
 
Rewind reversible compression protocol
 
Cross-message semantic deduplication
 
How to extend the pipeline

---

Installation