DataFoundry

An enterprise-grade Data Agent workbench — it reads business definitions through unified semantics, runs complex multi-table, multi-step analysis inside read-only boundaries,
and keeps every step auditable and replayable, turning one question into a trustworthy analysis.

28 datasource types out of the box · Enterprise semantics & context · Self-hosted · Multi-model · Fully auditable

English · 简体中文

Quick Start · Docs · Supported Data Sources · Roadmap · Contributing

🤔 What Is DataFoundry

When teams let AI query enterprise databases, the real worry is never "can the model write SQL." It is: does it understand business definitions? Could it mutate production data? Could credentials leak into context? Can a conclusion be verified after the fact?

Most tools reduce the problem to prompt → SQL → answer — impressive in a demo, dead on arrival in the enterprise. DataFoundry takes a different path: it puts the agent inside a semantic, policy-aware, evidence-preserving data task system, upgrading natural-language analytics into controllable, trustworthy, verifiable data work.

✨ Core Capabilities

• 🗄️ 28 datasource types, ready out of the box — From PostgreSQL, MySQL, Snowflake, BigQuery, and ClickHouse to MongoDB, Redis, and Elasticsearch: connect your existing data stack quickly, cut integration cost, and get the agent into real business analysis faster.
• 🧠 Enterprise semantics and context organization — Manage schema, metric definitions, and field relationships in one place, so terms like "GMV" and "retention" resolve to enterprise-approved tables, fields, and definitions — fewer guessed fields, wrong joins, and definition drift, and fundamentally better accuracy.
• 🏠 Self-hosted deployment and multi-model support — Run it inside your own boundary so data never leaves; on the model side, any OpenAI-compatible provider works (Qwen, DeepSeek, GPT, ...), letting you balance security, cost, latency, and quality per scenario.
• 🔒 Safe by default, auditable throughout — Read-only queries, credential isolation, field masking, row limits, and timeouts by default; SQL, tool calls, and event streams are fully persisted and replayable, so every conclusion is backed by evidence.
• 🧩 Deep optimization for complex data tasks — Built for multi-table, multi-field, long-horizon analysis and multi-step reasoning: complex questions get decomposed, verified, and converged into trustworthy conclusions, materialized as tables, charts, and reports the team can reuse.

🚀 Formal deploy

Default path is the formal stack: password auth + build / start (do not run npm run dev). No business database required — a built-in DuckDB demo datasource works out of the box.

Two formal environments share the same startup commands:

| Environment | Use for | Email | Public URL | | --- | --- | --- | --- | | Formal test | Local / private acceptance | AUTHEMAILDELIVERY=test (links in API console) | http://127.0.0.1:3000 | | Real production | Public service | AUTHEMAILDELIVERY=smtp | Public HTTPS + reverse proxy |

git clone https://github.com/datagallery-lab/datafoundry.git
cd datafoundry
npm install
cp .env.example .env
cp apps/web/.env.example apps/web/.env.local

Configure any OpenAI-compatible model in the root .env, plus auth for formal test or real production (sample defaults lean formal test):

LLM_PROVIDER=openai-compatible
LLM_MODEL=qwen-plus                # or deepseek-chat, gpt-4o, ...
LLMBASEURL=https://dashscope.aliyuncs.com/compatible-mode/v1
LLMAPIKEY=replace-with-your-key
DATAFOUNDRYAUTHMODE=password AUTHSESSIONSECRET=replace-with-at-least-32-random-characters AUTHPUBLICBASE_URL=http://127.0.0.1:3000   # real production: https://your.domain AUTHEMAILDELIVERY=test                     # real production: smtp + AUTHSMTP*

apps/web/.env.local:

NEXTPUBLICDATAFOUNDRYAUTHMODE=password
NEXTPUBLICAGENTRUNTIMEURL=
NEXTPUBLICCONFIGAPIURL=
APIPROXYTARGET=http://127.0.0.1:8787

Build and start:

npm run build
npm run build:web
npm run start:api    # :8787  — /healthz liveness, /ready readiness
npm run start:web    # :3000  — password mode via same-origin BFF

Open http://127.0.0.1:3000/login, register or sign in, go to /data-tasks, and ask:

Show me the tables in this datasource and explain the main fields of each.

You will see the full chain: schema inspection → read-only SQL → SQL audit → table output → replayable run history.

For real production, also configure SMTP and a reverse proxy: deploy/nginx.datafoundry.conf.example (gzip/brotli static assets; keep /api/copilotkit uncompressed and unbuffered for SSE).

Full steps and the two-environment matrix: Quick Start. Contributor hot-reload (npm run dev) is appendix-only. Connect your own PostgreSQL / MySQL / CSV and more: Data Sources guide.

🆚 How It Differs From Coding Agents And SQL Chatbots

Coding agents change code, SQL chatbots answer questions, DataFoundry runs data tasks — three different operating objects, risk boundaries, and outputs:

| | Works on | Main risk | Output | | --- | --- | --- | --- | | Coding agent | Repos, tests, PRs | Breaking code | Patch, commit, PR | | SQL chatbot | Prompt, SQL, answer | Wrong tables, unsafe access, leaked credentials, no replay | A SQL snippet or an answer | | DataFoundry | Datasources, files, knowledge, tools, task state | Production data boundaries, business semantics, audit evidence | Replayable data tasks + SQL audit + tables / charts / reports |

On data tasks specifically, DataFoundry's core advantages over a general-purpose coding agent are:

• Accuracy, from data constraints — Pointed at a database, a coding agent tends to guess tables, fields, and definitions; DataFoundry enforces schema-first analysis and constrains query paths through Data Gateway, cutting guessed fields and wrong joins.
• Safety, from controlled execution — Coding agents run commands and write files: powerful, but high-risk against enterprise data. DataFoundry defaults to read-only SQL, credential isolation, field masking, row limits, timeouts, and audit — built for real data environments.
• Speed, from a converging task path — Not because the model reasons faster, but because datasource selection, schema caching, context budgeting, tool policy, and the artifact pipeline eliminate wasted attempts, so analysis converges on results sooner.
• Complex tasks, from data-workflow design — Coding agents excel at code engineering; DataFoundry is built for analysis across many tables, fields, and metrics plus knowledge bases, files, and report outputs, chaining "query, verify, explain, materialize" into one complete flow.
• Adoption, from an enterprise runtime — This is not a demo: the Web workbench, TUI, REST API, CopilotKit / AG-UI, Data Gateway, Skills, MCP, Files, Artifacts, and Metadata combine into an operating foundation for data agents.

⚙️ How A Data Task Runs

Ask → Align semantics → Execute under control → Materialize output → Replay & review

• Define the task — Pick datasources, files, knowledge, and tools, then describe the business question in natural language.
• Align meaning with structure — The agent inspects schema and available context first, grounding terms like "GMV" or "retention" in real tables and fields.
• Execute under control — Data Gateway runs queries inside a read-only boundary with SQL guardrails, row limits, timeouts, and masking; every SQL statement leaves an audit record.
• Materialize output — Results become tables, charts, reports, or files — assets the team can cite.
• Replay and review — Web, TUI, and API share one run history, so every step's evidence is always one click away.

🖥️ More Than A Chat Box

The Web workbench fits day-to-day analysis and demos, the TUI fits terminals and remote servers, and the API / CopilotKit / AG-UI path lets you embed the same trusted runtime into your own product.

🗄️ Bring Your Data Stack, No Rebuild

Connect through Data Gateway adapters: the built-in DuckDB demo works out of the box; SQLite, CSV, Excel, PostgreSQL, and MySQL fit local trials; cloud warehouses, search engines, and NoSQL systems plug in with their own services and credentials.

See the full list in Supported Data Sources.

🛡️ Security Boundary

• The model only receives governed context; datasource credentials, model API keys, and MCP tokens never enter messages, context, or forwardedProps.
• All datasource access goes through Data Gateway — read-only by default, with SQL guardrails, row limits, timeouts, and field masking.
• SQL audit logs, tool-call records, and event streams are fully persisted for after-the-fact review.
• Production-grade multi-tenant auth, centralized secret management, monitoring, and deployment operations require a dedicated design for your environment — see Security.

🗺️ Roadmap

• [x] Governed data-task workbench — Web and TUI share one TypeScript agent runtime, CopilotKit / AG-UI event stream, replayable run history, SQL audit trail, and unified asset layer.
• [x] Safe data access foundation — Datasource registration, connection testing, schema introspection, table preview, read-only SQL, masking, knowledge imports, MCP resources, skill packages, and model configuration.
• [ ] Unified semantic layer — Durable metrics, entities, relationships, lineage, and policies, moving agents from "guessing fields" to "understanding business definitions" and from one-off SQL to a governable data operating layer.
• [ ] Autonomous analyst loops — Agents that plan investigations, run controlled experiments, critique findings, and converge on evidence-backed conclusions.
• [ ] Evaluation and reliability lab — NL2SQL, retrieval, tool-use, and end-to-end task benchmarks with regression gates and failure forensics.
• [ ] Enterprise control plane — Identity, RBAC, approvals, audit export, policy-as-code, and cost limits.

📚 Documentation

| Goal | Read | | --- | --- | | Run the local demo | Quick Start | | Understand positioning and scope | Overview · Capabilities | | Use the Web / TUI | Web workbench guide · TUI guide | | Connect data sources | Data sources guide · Supported data sources | | Integrate via API and runtime | REST API · Agent Runtime & AG-UI | | Understand architecture and security | Architecture overview · Security |

🤝 Contributing

DataFoundry moves quickly, so small, well-scoped PRs are the easiest to merge:

• Open an issue or discussion first for behavioral, protocol, datasource-adapter, or agent-policy changes.
• Keep each PR focused on one runtime boundary or feature area.
• Run npm run build plus the targeted smoke checks for what you touched (e.g. npm run smoke:data-gateway).
• Update docs when a change affects setup, APIs, datasource configuration, or user-visible output.
• Do not commit credentials, local databases, generated storage, or private benchmark data.

💬 Community

Join the DataFoundry community to discuss the product, roadmap, and real-world adoption with us. Issues and discussions are always welcome too.

Slack Community	QQ Group
Join Slack	Group ID 1048076064

🙏 Acknowledgements

DataFoundry is inspired by and built with ideas from excellent open-source projects and communities:

• Thanks to the LINUX DO community for their support and discussions.
• Mastra for agent runtime patterns.
• AG-UI for event stream protocol design.
• CopilotKit for agent-native UX patterns.
• Ink for terminal UI foundations.
• MCP for the tool ecosystem and integration model.

📄 License

Apache License 2.0. See LICENSE.

DataFoundry is under active development. Current code, public docs, and passing smoke checks are the source of truth.