Topics

#bm25 #crewai #docker #document-parsing #document-search #fastapi #haystack #hybrid-search #knowledge-base #langchain #llamaindex #llm #ollama #open-source #python #rag #self-hosted #semantic-search #vector-search

Language breakdown

Python 85.5%

HTML 14.0%

Shell 0.2%

Makefile 0.2%

Dockerfile 0.1%

▸ Files click to expand

README

FLAMEHAVEN FileSearch

Self-hosted RAG search engine. Production-ready in 3 minutes.

Quick Start • Features • Documentation • API Reference • Contributing

🎯 Why FLAMEHAVEN FileSearch?

Stop sending your sensitive documents to third-party services. FLAMEHAVEN FileSearch is a production-grade RAG search engine — BM25+hybrid retrieval, 34 file formats, multi-LLM (Gemini, OpenAI, Claude, Ollama) — running self-hosted in minutes, not days.

# Gemini (cloud) — one command, three minutes
docker run -d -p 8000:8000 -e GEMINIAPIKEY="your_key" flamehaven-filesearch:1.6.4
Ollama — fully local, zero API cost (Gemma, Llama, Mistral, Qwen, Phi …)
Step 1: pull a model  →  ollama pull gemma4:27b
docker run -d -p 8000:8000 \
  -e LLM_PROVIDER=ollama \
  -e LOCAL_MODEL=gemma4:27b \
  -e OLLAMABASEURL=http://host.docker.internal:11434 \
  flamehaven-filesearch:1.6.4

🚀 Fast

Production deployment in 3 minutes
Vector generation in <1ms
Zero ML dependencies

🔒 Private

100% self-hosted
Your data never leaves your infrastructure
Enterprise-grade security

💰 Cost-Effective

Free tier: 1,500 queries/month
No infrastructure costs
Open source & MIT licensed

Features ✨

Core Capabilities

| Capability | Detail | |---|---| | Search Modes | Keyword, semantic, and hybrid (BM25+RRF) with automatic typo correction | | Quality Gate | Confidence-scored hybrid results (PASS/FORGE/INHIBIT). FORGE augments with keyword fallback; INHIBIT flags low_confidence. Self-adapting BM25 pool via EMA meta-learner. Zero new dependencies. | | Obsidian Light Mode | Markdown-first vault ingest with frontmatter, aliases, tags, wikilinks, heading-aware chunking, context enrichment, exact note resolution | | 34 File Formats | PDF, DOCX/DOC, XLSX, PPTX, RTF, HTML, CSV, LaTeX, WebVTT, images + plain text — see Document Parsing | | RAG Pipeline | Structure-aware chunking, KnowledgeAtom 2-level indexing, sliding-window context enrichment, mtime parse cache | | Ultra-Fast Vectors | DSP v2.0 generates embeddings in <1ms — no ML frameworks required | | Source Attribution | Every answer links back to the originating document and chunk | | Framework SDKs | LangChain, LlamaIndex, Haystack, CrewAI adapters out of the box | | Enterprise Auth | API key hashing (SHA256+salt), OAuth2/OIDC, fine-grained permissions | | Admin Dashboard | Real-time metrics, quota management, batch processing (1–100 queries) | | Flexible Storage | SQLite (default) · PostgreSQL + pgvector · Redis cache (optional) |

What changed in each release? See CHANGELOG.md for the full version history.

Quick Start 🚀

Option 1: Docker (Recommended)

The fastest path to production:

docker run -d \
  -p 8000:8000 \
  -e GEMINIAPIKEY="yourgeminiapi_key" \
  -e FLAMEHAVENADMINKEY="secureadminpassword" \
  -v $(pwd)/data:/app/data \
  flamehaven-filesearch:1.6.4

✅ Server running at http://localhost:8000

Option 2: Python SDK

Perfect for integrating into existing applications:

from flamehaven_filesearch import FlamehavenFileSearch, FileSearchConfig
Initialize
config = FileSearchConfig(googleapikey="yourgeminikey")
fs = FlamehavenFileSearch(config)
Upload and search
fs.uploadfile("companyhandbook.pdf", store="docs")
result = fs.search("What is our remote work policy?", store="docs")
print(result['answer']) Output: "Employees can work remotely up to 3 days per week..."

Option 3: REST API

For language-agnostic integration:

# 1. Generate API key
curl -X POST http://localhost:8000/api/admin/keys \
  -H "X-Admin-Key: youradminkey" \
  -d '{"name":"production","permissions":["upload","search"]}'
2. Upload document
curl -X POST http://localhost:8000/api/upload/single \
  -H "Authorization: Bearer skliveabc123..." \
  -F "file=@document.pdf" \
  -F "store=my_docs"
3. Search
curl -X POST http://localhost:8000/api/search \
  -H "Authorization: Bearer skliveabc123..." \
  -H "Content-Type: application/json" \
  -d 
  '{ 
    "query": "What are the main findings?",
    "store": "my_docs",
    "search_mode": "hybrid"
  }'

📦 Installation

# Core package (HTML, CSV, LaTeX, WebVTT, plain-text parsing included — zero extra deps)
pip install flamehaven-filesearch
+ Document parsers: PDF (pymupdf/pypdf), DOCX, XLSX, PPTX, RTF
pip install flamehaven-filesearch[parsers]
+ Image OCR (Pillow + pytesseract; requires Tesseract system binary)
pip install flamehaven-filesearch[vision]
+ Google Gemini API
pip install flamehaven-filesearch[google]
+ REST API server (FastAPI + uvicorn)
pip install flamehaven-filesearch[api]
+ HNSW vector index
pip install flamehaven-filesearch[vector]
+ PostgreSQL backend
pip install flamehaven-filesearch[postgres]
Everything
pip install flamehaven-filesearch[all]
Build from source
git clone https://github.com/flamehaven01/Flamehaven-Filesearch.git
cd Flamehaven-Filesearch
docker build -t flamehaven-filesearch:1.6.4 .

Framework Integrations

Framework SDKs (LangChain, LlamaIndex, etc.) are imported lazily — install only what you need:

# LangChain  (pip install langchain-core)
from flamehaven_filesearch.integrations import FlamehavenLangChainLoader
docs = FlamehavenLangChainLoader("report.pdf", chunk=True).load()
LlamaIndex  (pip install llama-index-core)
from flamehaven_filesearch.integrations import FlamehavenLlamaIndexReader
nodes = FlamehavenLlamaIndexReader(chunk=True).load_data(["report.pdf", "slides.pptx"])
Haystack  (pip install haystack-ai)
from flamehaven_filesearch.integrations import FlamehavenHaystackConverter
result = FlamehavenHaystackConverter().run(sources=["report.pdf"])
CrewAI  (pip install crewai)
from flamehaven_filesearch.integrations import FlamehavenCrewAITool
tool = FlamehavenCrewAITool()           # pass to your agent's tools list

Configuration ⚙️

LLM Provider Selection

FLAMEHAVEN supports four LLM backends — switch with a single env var:

| LLM_PROVIDER | Required variables | Notes | |---|---|---| | gemini (default) | GEMINIAPIKEY | Google Gemini file-search API | | ollama | LOCALMODEL, OLLAMABASE_URL | Local inference via Ollama — Gemma 4/3, Llama 3.2, Qwen 2.5, Mistral, Phi-4 … | | openai | OPENAIAPIKEY | OpenAI or any OpenAI-compatible endpoint | | anthropic | ANTHROPICAPIKEY | Anthropic Claude | | openaicompatible | OPENAIAPIKEY, OPENAIBASE_URL | vLLM, LM Studio, Kimi, etc. |

# Gemini (default)
export GEMINIAPIKEY="yourgooglegeminiapikey"
Ollama (fully local)
export LLM_PROVIDER=ollama
export LOCAL_MODEL=gemma4:27b          # or gemma4:4b, qwen2.5:7b, llama3.2 …
export OLLAMABASEURL=http://localhost:11434
OpenAI
export LLM_PROVIDER=openai
export OPENAIAPIKEY="sk-..."
export DEFAULT_MODEL=gpt-4o-mini       # optional override
Anthropic
export LLM_PROVIDER=anthropic
export ANTHROPICAPIKEY="sk-ant-..."

Required Environment Variables

export FLAMEHAVENADMINKEY="yoursecureadmin_password"
Plus the provider credentials above (at least one provider)

Optional Configuration

export HOST="0.0.0.0"              # Bind address
export PORT="8000"                  # Server port
export REDIS_HOST="localhost"       # Distributed caching
export REDIS_PORT="6379"            # Redis port
export MAXOUTPUTTOKENS="1024"     # Max answer tokens
export TEMPERATURE="0.5"            # Model temperature (0.0–1.0)
export MAX_SOURCES="5"              # Max source documents per answer

Obsidian / Local Vault Configuration

For Markdown-heavy vaults, enable Obsidian light mode:

export OBSIDIANLIGHTMODE=true
export OBSIDIANCHUNKMAX_TOKENS=256
export OBSIDIANCHUNKMIN_TOKENS=32
export OBSIDIANCONTEXTWINDOW=1
export OBSIDIANRESPLITCHUNK_CHARS=1200
export OBSIDIANRESPLITOVERLAP_CHARS=160

This path preserves note structure and improves retrieval on dense vaults with many related notes. Operational details: Obsidian Light Mode

Advanced Configuration

Create a config.yaml for fine-tuned control:

vector_store:
  quantization: int8
  compression: gravitas_pack
  
search:
  default_mode: hybrid
  typo_correction: true
  max_results: 10
  
security:
  rate_limit: 100  # requests per minute
  maxfilesize: 52428800  # 50MB

📊 Performance

Metric	Value	Notes
Vector Generation	`<1ms`	DSP v2.0, zero ML dependencies
Memory Footprint	`75% reduced`	Int8 quantization vs float32
Metadata Size	`90% smaller`	Gravitas-Pack compression
Test Suite	`1200 tests, 81% coverage`	All passing (pytest)
Cold Start	`3 seconds`	Docker container ready

Real-World Benchmarks

Environment: Docker on Apple M1 Mac, 16GB RAM
Document Set: 500 PDFs, ~2GB total
Health Check:           8ms Search (cache hit):     9ms Search (cache miss):    1,250ms  (includes Gemini API call) Batch Search (10):      2,500ms  (parallel processing) Upload (50MB file):     3,200ms  (with indexing)

Architecture 🏗️

flowchart TD
    Client(["Client\n(HTTP / SDK)"])
subgraph API["REST API Layer (FastAPI)"]         Upload["/api/upload"]         Search["/api/search"]         Admin["/api/admin"]     end
subgraph Engine["Engine Layer"]         FP["FileParser\n+ BackendRegistry\n(34 formats)"]         Cache["ParseCache\n(mtime-based)"]         Chunker["TextChunker\n+ KnowledgeAtom\n(chunk atoms)"]         DSP["DSP v2.0\nEmbedding Generator\n(&lt;1ms, zero-ML)"]         BM25["BM25 + RRF\nHybrid Search\n(v1.6.0)"]         Scorer["SemanticScorer\n+ TypoCorrector"]     end
subgraph Storage["Storage Layer"]         SQLite[("SQLite\nMetadata Store")]         Vec[("Vector Store\n(local / pgvector)")]         Redis[("Redis Cache\n(optional)")]     end
subgraph LLM["LLM Provider (env: LLM_PROVIDER)"]         Gemini["Gemini\n(cloud)"]         Ollama["Ollama\n(local)"]         OAI["OpenAI /\nAnthropic /\nCompatible"]     end     Metrics["Metrics Logger"]
Client --> Upload & Search & Admin     Upload --> FP     FP <-->|"cache hit/miss"| Cache     FP --> Chunker     Chunker --> DSP     DSP --> Vec     FP --> SQLite
Search --> Scorer     Scorer --> DSP     DSP --> Vec     Scorer -->|"gemini"| Gemini     Scorer -->|"ollama"| Ollama     Scorer -->|"openai/anthropic"| OAI     LLM --> Client
Admin --> Metrics     Admin --> SQLite     Storage <-->|"read / write"| Redis

Full layer detail: Architecture.md

Security 🔒

FLAMEHAVEN takes security seriously:

✅ API Key Hashing - SHA256 with salt
✅ Rate Limiting - Per-key quotas (default: 100/min)
✅ Permission System - Granular access control
✅ Audit Logging - Complete request history
✅ OWASP Headers - Security headers enabled by default
✅ Input Validation - Strict file type and size checks

Security Best Practices

# Use strong admin keys
export FLAMEHAVENADMINKEY=$(openssl rand -base64 32)
Enable HTTPS in production
(use nginx/traefik as reverse proxy)
Rotate API keys regularly
curl -X DELETE http://localhost:8000/api/admin/keys/oldkeyid \
  -H "X-Admin-Key: $FLAMEHAVENADMINKEY"

Roadmap 🗺️

Full roadmap: ROADMAP.md

v1.4.x (Completed)

[x] Multimodal search (image + text)
[x] HNSW vector indexing for faster search
[x] OAuth2/OIDC integration
[x] PostgreSQL backend (metadata + pgvector)
[x] Usage-budget controls and reporting
[x] pgvector tuning and reliability hardening
[x] CI/CD — ruff replaces flake8; pipelines fully green

v1.5.x (Completed)

[x] Universal Document Parser — 34 formats, zero doc-AI dependency (v1.5.0)
[x] Internal text chunker — structure-aware + token-aware, zero ML deps (v1.5.0)
[x] Framework integrations — LangChain, LlamaIndex, Haystack, CrewAI (v1.5.0)
[x] Backend Plugin Architecture — AbstractFormatBackend + BackendRegistry (v1.5.2)
[x] Parse cache — mtime-based, extracttext(usecache=True) (v1.5.2)
[x] ContextExtractor — sliding-window RAG chunk enrichment (v1.5.2)
[x] Multi-provider LLM support — OpenAI, Claude, Ollama, Gemini (v1.5.3)

v1.6.0 (Completed)

[x] BM25 + RRF hybrid search — Korean+English tokenizer, lazy per-store index
[x] KnowledgeAtom 2-level indexing — chunk atoms with fragment URIs
[x] Stable URI scheme — local://<store>/<quote(abs_path)>, collision-free
[x] core.py mixin segmentation — 1258 → 221 lines, 3 focused modules
[x] Fix: search_stream double intent-refine bug

v1.6.1 (Completed)

[x] CC reduction — seekvectorresonance CC 8→2, getadmin_user CC 10→1
[x] Dispatch table pattern — transformdict unifies GravitasPacker compress/decompress
[x] recordupload_failure helper — eliminates 2× duplicated metrics blocks in api.py
[x] /health exposes llmprovider + llmmodel — frontend can detect active backend
[x] config.todict() exposes llmprovider, localmodel, ollamabase_url
[x] Frontend: provider-aware model selector (Gemini dropdown ↔ local model badge)
[x] Frontend: upload accept list expanded to all 34 supported formats
[x] Frontend: store datalist auto-populated from /api/metrics
[x] Frontend: version badge synced to v1.6.1 across all 6 dashboard pages
[x] Ruff F401/F841 — 5 lint errors resolved, CI green
[x] Admin: Stores tab — create / list / delete stores (POST|GET|DELETE /api/stores)
[x] Admin: Ops tab — usage stats (GET /api/admin/usage) + vector ops (stats / reindex / vacuum)
[x] Landing: "Manage" deep-link to admin.html#stores with hash-based tab routing

v1.6.2 (Completed)

[x] engine/qualitygate.py — SearchQualityGate (PASS/FORGE/INHIBIT), SearchMetaLearner (EMA alpha adaptation), computesearchconfidence (BM25/semantic agreement score with residual floor — rawrrf × max(floor, (overlap+coverage)/2) — zero new deps)
[x] Hybrid search: confidence-scored results with FORGE keyword augmentation and INHIBIT flag
[x] searchconfidence + lowconfidence fields in search response schema
[x] BM25 pool size self-adapts via meta-learner alpha (keyword-dominant → larger pool)
[x] 25 tests, 99% coverage on quality_gate.py

v1.6.3 (Completed)

[x] P4 — Snapshot persistence (persistence.py, core.py): atomic JSON snapshots, cold-start restore
[x] P3 — Embedding provider abstraction: OllamaEmbeddingProvider + DSP fallback (EMBEDDING_PROVIDER=ollama|dsp)
[x] P6 — Non-neural query expansion (engine/query_expansion.py): optional synonym map, zero-ML recall lever
[x] P5 — Auto re-ingest watcher (tools/watch_ingest.py): content-fingerprint dedup, stdlib polling fallback
[x] P1 — searchconfidence in REST response: provider_search now computes confidence signal
[x] P2 — Configurable rate limits via env vars (UPLOADRATELIMIT, SEARCHRATELIMIT)
[x] P0 bugfixes: live→file typo overcorrection, exactnotematch suppressed by query expansion

v1.6.4 (Completed)

[x] Refactor: restorefrompersistence decomposed into injectintochronos, restorestoredocs, restorestoreatoms — depth 5 → 3, CC 17 → 5 (core.py)
[x] Refactor: listkeys inner try/except extracted to decodepermissions + rowtokey_info static helpers — depth 4 → 2 (auth.py)
[x] Refactor: initsearcher nested store-seed extracted to seeddefault_store — depth 4 → 2 (api.py)

v2.0.0 (Q3 2026)

[ ] Multi-language support (15+ languages) — multilingual stopwords + jieba
[ ] Kubernetes Helm charts
[ ] Distributed indexing

Troubleshooting 🐛

❌ 401 Unauthorized Error

Problem: API returns 401 when making requests.

Solutions:

Verify FLAMEHAVENADMINKEY environment variable is set
Check Authorization: Bearer sklive... header format
Ensure API key hasn't expired (check admin dashboard)

# Debug: Check if admin key is set echo $FLAMEHAVENADMINKEY
Regenerate API key
curl -X POST http://localhost:8000/api/admin/keys \
  -H "X-Admin-Key: $FLAMEHAVENADMINKEY" \
  -d '{"name":"debug","permissions":["search"]}'

🐌 Slow Search Performance

Problem: Searches taking >5 seconds.

Solutions:

Check cache hit rate: FLAMEHAVENMETRICSENABLED=1 curl http://localhost:8000/metrics
Enable Redis for distributed caching
Verify Gemini API latency (should be <1.5s)

# Enable Redis caching docker run -d --name redis redis:7-alpine export REDIS_HOST=localhost

💾 High Memory Usage

Problem: Container using >2GB RAM.

Solutions:

Enable Redis with LRU eviction policy
Reduce max file size in config
Monitor with Prometheus endpoint

# Configure Redis memory limit docker run -d \   -p 6379:6379 \   redis:7-alpine \   --maxmemory 512mb \   --maxmemory-policy allkeys-lru

More solutions in our Wiki Troubleshooting Guide.

Documentation 📚

Documentation Hub

Use the links below to jump to the most relevant guide.

| Topic | Description | |-------|-------------| | Document Parsing | Supported formats, internal parsers, RAG chunking | | Hybrid Search | BM25+RRF, KnowledgeAtom indexing, stable URI scheme (v1.6.0) | | Obsidian Light Mode | Markdown-first vault ingest, exact note resolution, dense-note retrieval tuning | | Framework Integrations | LangChain, LlamaIndex, Haystack, CrewAI adapters | | API Reference | REST endpoints, payloads, rate limits | | Architecture | How all layers fit together (v1.6.0) | | Configuration Reference | Full list of environment variables and config fields | | Production Deployment | Docker, systemd, reverse proxy, scaling tips | | Troubleshooting | Step-by-step debugging playbook | | Benchmarks | Performance measurements and methodology | | Release and Tagging | Release checklist, tag policy, and next tag guidance |

These Markdown files live inside the repository so they stay versioned alongside the code. Feel free to contribute improvements via pull requests.

Additional Resources

Interactive API Docs - OpenAPI/Swagger interface (when server is running)
CHANGELOG - Version history and release notes
Docs Hub - Versioned documentation index
CONTRIBUTING - How to contribute code
Examples - Sample integrations and use cases

Contributing 🤝

We love contributions! FLAMEHAVEN is better because of developers like you.

Good First Issues

🟢 [Easy] Add dark mode to admin dashboard (1-2 hours)
🟡 [Medium] PostgreSQL backend for usage tracker (multi-instance deployments)
🔴 [Advanced] Kubernetes Helm charts for production deployment

See CONTRIBUTING.md for development setup and guidelines.