Crypto Vision

A production-grade crypto intelligence platform — 200+ API endpoints, 37 data sources, 58 AI agents, real-time WebSocket feeds, anomaly detection, RAG-powered semantic search, BigQuery data warehouse, and a complete AI model training pipeline.

---

Table of Contents

• Overview
• Quick Start
• Environment Variables
• Architecture
• API Reference
• Data Sources
• Packages
• Apps
• AI Agents
• Workers & Ingestion
• Telegram Bot (crypto vision)
• Infrastructure
• Training & Fine-Tuning
• BigQuery Data Warehouse
• Testing
• Monitoring & Observability
• Self-Hosting
• Performance
• Project Structure
• Documentation

---

Overview

Crypto Vision is a comprehensive crypto data aggregation, intelligence, and AI platform built on TypeScript and the Hono framework. It provides:

• 200+ REST API endpoints across 35+ route domains (market data, DeFi, on-chain analytics, derivatives, news, AI, governance, macroeconomics, and more)
• 37 data source adapters (CoinGecko, DeFiLlama, Binance, Bybit, OKX, Deribit, dYdX, Hyperliquid, L2Beat, Messari, Token Terminal, GeckoTerminal, Jupiter, DePin Scan, etc.)
• 58 specialized AI agents for DeFi strategy, trading, portfolio management, security analysis, and education
• Real-time WebSocket feeds for prices, Bitcoin mempool, and DEX trades
• Anomaly detection engine using Modified Z-Score and EWMA algorithms across 16 anomaly types
• RAG-powered semantic search with multi-provider embeddings (Vertex AI, OpenAI)
• Multi-provider AI with automatic fallback chain: Groq → Gemini → OpenAI → Anthropic → OpenRouter
• Two-tier caching (in-memory LRU + Redis) with stampede protection and stale-while-revalidate
• Circuit breaker pattern per upstream host with exponential backoff
• BigQuery data warehouse with 22+ tables and materialized views
• Complete ML pipeline for model training, fine-tuning (Gemini + open-source), evaluation, and inference serving
• Full GCP infrastructure as code (Terraform, Cloud Run, Pub/Sub, Scheduler, Kubernetes)

---

Quick Start

Prerequisites

• Node.js ≥ 22 (ESM required)
• PostgreSQL 16 (for crypto vision)
• Redis 7 (optional, for distributed caching)
• Google Cloud credentials (optional, for BigQuery/Pub/Sub/Vertex AI)

Local Development

# Clone the repository
git clone https://github.com/nirholas/crypto-vision.git
cd crypto-vision
Install dependencies
npm install
Copy and configure environment variables
cp .env.example .env  # Edit with your API keys
Start development server (hot-reload via tsx)
npm run dev
Server starts on http://localhost:8080

Docker

# Build and run the main API server
docker build -t crypto-vision .
docker run -p 8080:8080 --env-file .env crypto-vision
Or use Docker Compose for full stack (API + Redis)
docker compose up

Docker Compose — Ingestion Workers

# Start ingestion workers (market, defi, news, dex, derivatives, governance, macro, onchain)
docker compose -f docker-compose.ingest.yml up

NPM Scripts

| Script | Description | |---|---| | npm run dev | Start dev server with hot-reload (tsx) | | npm run build | Compile TypeScript (tsc -p tsconfig.build.json && tsc-alias) | | npm start | Run compiled build (node dist/index.js) | | npm test | Run all tests (Vitest) | | npm run test:coverage | Run tests with coverage | | npm run test:e2e | Run end-to-end tests | | npm run lint | Run ESLint | | npm run typecheck | TypeScript type checking | | npm run db:generate | Generate Drizzle ORM migrations | | npm run db:migrate | Apply database migrations | | npm run db:push | Push schema changes | | npm run db:studio | Open Drizzle Studio | | npm run export:all | Export all data from BigQuery | | npm run export:embeddings | Export embeddings | | npm run import:postgres | Import exported data to PostgreSQL |

---

Environment Variables

📖 See docs/SECURITY_GUIDE.md for secrets management best practices.

Core

| Variable | Required | Default | Description | |---|---|---|---| | PORT | No | 8080 | Server port | | NODE_ENV | No | development | Environment (development / production) | | LOG_LEVEL | No | info | Pino log level |

API Keys — Data Sources

| Variable | Required | Description | |---|---|---| | COINGECKOAPIKEY | No | CoinGecko Pro/Demo API key | | COINGECKOAPITYPE | No | pro or demo (default: demo) | | CRYPTOCOMPAREAPIKEY | No | CryptoCompare API key | | ETHERSCANAPIKEY | No | Etherscan API key | | MESSARIAPIKEY | No | Messari API key | | TOKENTERMINALAPIKEY | No | Token Terminal API key | | COINGLASSAPIKEY | No | CoinGlass API key | | NEWSDATAAPIKEY | No | NewsData.io API key | | CALENDARIFICAPIKEY | No | Calendarific API key | | GOPLUSAPPKEY | No | GoPlus security API key | | GOPLUSAPPSECRET | No | GoPlus security API secret |

AI Providers

| Variable | Required | Description | |---|---|---| | GROQAPIKEY | No | Groq API key (primary AI provider) | | GEMINIAPIKEY | No | Google Gemini API key (fallback #1) | | OPENAIAPIKEY | No | OpenAI API key (fallback #2) | | ANTHROPICAPIKEY | No | Anthropic API key (fallback #3) | | OPENROUTERAPIKEY | No | OpenRouter API key (final fallback) |

Database & Cache

| Variable | Required | Description | |---|---|---| | DATABASE_URL | For Bot | PostgreSQL connection string | | REDIS_URL | No | Redis connection string |

Google Cloud

| Variable | Required | Description | |---|---|---| | GCPPROJECTID | For GCP features | Google Cloud project ID | | GCPBIGQUERYDATASET | For BigQuery | BigQuery dataset name | | GCPPUBSUBENABLED | No | Enable Pub/Sub publishing (true/false) | | VERTEXAILOCATION | No | Vertex AI region (default: us-central1) |

Authentication

| Variable | Required | Description | |---|---|---| | APIKEYPUBLIC | No | Public tier key (30 rpm) | | APIKEYBASIC | No | Basic tier key (200 rpm) | | APIKEYPRO | No | Pro tier key (2000 rpm) | | APIKEYENTERPRISE | No | Enterprise tier key (10000 rpm) |

Telegram Bot

| Variable | Required | Description | |---|---|---| | SECTBOT_ENABLED | No | Enable crypto vision (true/false) | | BOT_TOKEN | For Bot | Telegram bot token |

---

Architecture

📖 See docs/ARCHITECTURE.md for the full architecture deep-dive.

┌──────────────────────────────────────────────────────────────────┐
│                        Clients                                   │
│   REST API  │  WebSocket  │  Telegram Bot  │  Dashboard/Apps     │
└──────┬──────┴──────┬──────┴───────┬────────┴──────┬──────────────┘
       │             │              │               │
┌──────▼─────────────▼──────────────▼───────────────▼──────────────┐
│                     Hono HTTP Server (src/index.ts)               │
│  ┌────────────────────────────────────────────────────────────┐   │
│  │ Middleware: CORS │ Auth │ Rate-Limit │ Compression │ ETag  │   │
│  │            │ Metrics │ Response Envelope │ Request Timeout │   │
│  └────────────────────────────────────────────────────────────┘   │
│                                                                   │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────────────┐ │
│  │ 35+ Route│  │ WebSocket│  │ Anomaly  │  │ AI + RAG Pipeline│ │
│  │ Modules  │  │ Manager  │  │ Engine   │  │ Multi-Provider   │ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └───────┬──────────┘ │
└───────┼──────────────┼─────────────┼────────────────┼────────────┘
        │              │             │                │
┌───────▼──────────────▼─────────────▼────────────────▼────────────┐
│                        Core Libraries (src/lib/)                  │
│  ┌─────────┐  ┌────────────┐  ┌──────────┐  ┌────────────────┐  │
│  │ Fetcher │  │ Two-Tier   │  │ Bounded  │  │ Vector Store   │  │
│  │ Circuit │  │ Cache      │  │ Queue    │  │ (BigQuery /    │  │
│  │ Breaker │  │ LRU+Redis  │  │ AI:10/500│  │  In-Memory)    │  │
│  └────┬────┘  └─────┬──────┘  └────┬─────┘  └───────┬────────┘  │
└───────┼─────────────┼──────────────┼─────────────────┼───────────┘
        │             │              │                 │
┌───────▼─────────────▼──────────────▼─────────────────▼───────────┐
│               37 Data Source Adapters (src/sources/)              │
│  CoinGecko │ DeFiLlama │ Binance │ Bybit │ OKX │ L2Beat │ ...   │
└───────┬──────────────────────────────────────────────────────────┘
        │
┌───────▼──────────────────────────────────────────────────────────┐
│                    Storage & Messaging                            │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────────────┐   │
│  │PostgreSQL│  │  Redis   │  │ BigQuery │  │  Pub/Sub       │   │
│  │(Drizzle) │  │(ioredis) │  │ (22+ tbl)│  │ (5 tiers)     │   │
│  └──────────┘  └──────────┘  └──────────┘  └────────────────┘   │
└──────────────────────────────────────────────────────────────────┘

Key Architectural Patterns

Two-Tier Cache — In-memory LRU (200,000 entries) with optional Redis L2. Features stampede protection (coalesced requests), stale-while-revalidate (serve stale data while refreshing), per-key TTL, and background refresh.

Circuit Breaker — Per-host circuit breakers with configurable failure thresholds. States: CLOSED → OPEN (after 5 failures) → HALF-OPEN (probe after 30s). Prevents cascade failures to upstream APIs.

Bounded Concurrency Queue — AI requests are bounded to 10 concurrent / 500 max queued. Prevents overloading AI providers and manages backpressure.

Multi-Provider AI Fallback — Automatic failover chain: Groq → Gemini → OpenAI → Anthropic → OpenRouter. If one provider is down, the next in chain handles the request transparently.

4-Tier API Authentication — Public (30 rpm), Basic (200 rpm), Pro (2,000 rpm), Enterprise (10,000 rpm). Header-based API key authentication with per-tier rate limiting.

Response Envelope — All responses wrapped in { success, data, meta: { timestamp, source, cached, latency_ms } } for consistent client consumption.

---

API Reference

📖 See docs/APIREFERENCE.md for endpoint details and docs/APIAUTHENTICATION.md for auth tiers & rate limits.

The full OpenAPI 3.1 specification is available at openapi.yaml.

Health & Metrics

| Method | Path | Description | |---|---|---| | GET | /health | Health check with cache stats, circuit breaker status, queue stats, WebSocket stats, memory usage | | GET | /metrics | Prometheus metrics endpoint |

Market Data (/api/market)

| Method | Path | Description | |---|---|---| | GET | /api/market/prices | Current prices for top cryptocurrencies | | GET | /api/market/prices/:id | Price data for a specific coin | | GET | /api/market/trending | Trending coins | | GET | /api/market/global | Global market statistics | | GET | /api/market/dominance | Market dominance breakdown | | GET | /api/market/fear-greed | Fear & Greed Index | | GET | /api/market/history/:id | Historical price data | | GET | /api/market/categories | Market categories | | GET | /api/market/top-movers | Top gainers and losers | | GET | /api/market/high-volume | High volume coins |

DeFi (/api/defi)

| Method | Path | Description | |---|---|---| | GET | /api/defi/protocols | DeFi protocol rankings | | GET | /api/defi/tvl | Total Value Locked across DeFi | | GET | /api/defi/yields | Yield farming opportunities | | GET | /api/defi/stablecoins | Stablecoin analytics | | GET | /api/defi/chains | Chain-level DeFi metrics | | GET | /api/defi/dexes | DEX volume data | | GET | /api/defi/bridges | Bridge volume and TVL | | GET | /api/defi/liquidations | Liquidation data |

News (/api/news)

| Method | Path | Description | |---|---|---| | GET | /api/news | Latest crypto news | | GET | /api/news/analysis | AI-analyzed news with sentiment |

On-Chain (/api/onchain)

| Method | Path | Description | |---|---|---| | GET | /api/onchain/gas | Gas prices (Ethereum) | | GET | /api/onchain/whale-transactions | Whale transaction alerts | | GET | /api/onchain/token-security/:address | Token security audit | | GET | /api/onchain/contract-info/:address | Smart contract information |

AI & Intelligence (/api/ai)

22 AI-powered endpoints for market analysis, portfolio advice, DeFi strategy, risk assessment, and more.

| Method | Path | Description | |---|---|---| | POST | /api/ai/analyze | AI market analysis | | POST | /api/ai/portfolio-advice | AI portfolio recommendations | | POST | /api/ai/defi-strategy | DeFi strategy generation | | POST | /api/ai/risk-assess | Risk assessment | | POST | /api/ai/trading-signals | AI trading signal analysis | | POST | /api/ai/sentiment | Sentiment analysis | | POST | /api/ai/compare | Comparative token analysis | | POST | /api/ai/predict | Price prediction analysis | | POST | /api/ai/explain | Explain crypto concepts | | POST | /api/ai/news-impact | News impact analysis | | POST | /api/ai/whale-analysis | Whale behavior analysis | | POST | /api/ai/yield-optimize | Yield optimization | | POST | /api/ai/security-audit | Smart contract security audit | | POST | /api/ai/gas-optimize | Gas optimization recommendations | | POST | /api/ai/narrative | Market narrative detection | | POST | /api/ai/correlation | Cross-asset correlation analysis | | POST | /api/ai/tokenomics | Tokenomics analysis | | POST | /api/ai/chain-compare | Multi-chain comparison | | POST | /api/ai/defi-risk-score | DeFi protocol risk scoring | | POST | /api/ai/portfolio-rebalance | Portfolio rebalancing suggestions | | POST | /api/ai/entry-exit | Entry/exit point analysis | | POST | /api/ai/macro-impact | Macro-economic impact on crypto |

AI Agents (/api/agents)

| Method | Path | Description | |---|---|---| | GET | /api/agents | List all 58 available agents | | GET | /api/agents/:agentId | Get agent details | | POST | /api/agents/:agentId/invoke | Invoke an agent with a query | | POST | /api/agents/multi | Invoke multiple agents in parallel | | POST | /api/agents/orchestrate | Multi-agent orchestration (planner → executor → synthesizer) | | GET | /api/agents/categories | Agent categories | | GET | /api/agents/search | Search agents |

CEX Data (/api/cex)

| Method | Path | Description | |---|---|---| | GET | /api/cex/orderbook/:symbol | Order book depth | | GET | /api/cex/trades/:symbol | Recent trades | | GET | /api/cex/ticker/:symbol | 24h ticker | | GET | /api/cex/klines/:symbol | Candlestick data | | GET | /api/cex/funding/:symbol | Funding rates |

DEX Data (/api/dex)

| Method | Path | Description | |---|---|---| | GET | /api/dex/trending | Trending pools | | GET | /api/dex/new-pairs | Newly created pairs | | GET | /api/dex/pools/:network/:address | Pool details | | GET | /api/dex/tokens/:network/:address | Token info from DEXes | | GET | /api/dex/search | Search DEX pools |

Derivatives (/api/derivatives)

| Method | Path | Description | |---|---|---| | GET | /api/derivatives/options | Options market data | | GET | /api/derivatives/futures | Futures market data | | GET | /api/derivatives/open-interest | Open interest aggregation | | GET | /api/derivatives/funding-rates | Cross-exchange funding rates | | GET | /api/derivatives/liquidations | Liquidation data |

Perps (/api/perps)

| Method | Path | Description | |---|---|---| | GET | /api/perps/markets | Perpetual markets overview | | GET | /api/perps/funding | Perpetual funding rates |

Bitcoin (/api/bitcoin)

| Method | Path | Description | |---|---|---| | GET | /api/bitcoin/stats | Bitcoin network statistics | | GET | /api/bitcoin/mempool | Mempool status | | GET | /api/bitcoin/hashrate | Hash rate data | | GET | /api/bitcoin/fees | Fee estimates | | GET | /api/bitcoin/lightning | Lightning Network stats |

Gas (/api/gas)

| Method | Path | Description | |---|---|---| | GET | /api/gas/ethereum | Ethereum gas prices | | GET | /api/gas/multi-chain | Multi-chain gas prices | | GET | /api/gas/history | Gas price history |

Governance (/api/governance)

| Method | Path | Description | |---|---|---| | GET | /api/governance/proposals | Active DAO proposals | | GET | /api/governance/spaces | Snapshot spaces | | GET | /api/governance/votes/:id | Votes for a proposal |

Macro (/api/macro)

| Method | Path | Description | |---|---|---| | GET | /api/macro/indicators | Macroeconomic indicators (from Yahoo Finance) | | GET | /api/macro/correlation | Crypto-macro correlation |

Solana (/api/solana)

| Method | Path | Description | |---|---|---| | GET | /api/solana/tokens | Solana token data (via Jupiter) | | GET | /api/solana/defi | Solana DeFi metrics |

Additional Route Domains

| Domain | Path Prefix | Description | |---|---|---| | DePIN | /api/depin | Decentralized Physical Infrastructure Networks | | Exchanges | /api/exchanges | Exchange rankings and details | | NFT | /api/nft | NFT collections and marketplaces | | Whales | /api/whales | Whale wallet tracking | | Staking | /api/staking | Staking yields and validators | | Calendar | /api/calendar | Crypto events calendar | | Oracles | /api/oracles | Oracle network data | | Unlocks | /api/unlocks | Token unlock schedules | | ETF | /api/etf | Crypto ETF data | | Portfolio | /api/portfolio | Portfolio tracking and analysis | | Social | /api/social | Social metrics and sentiment | | L2 | /api/l2 | Layer 2 scaling data (L2Beat) | | Security | /api/security | Token/contract security analysis (GoPlus) | | Research | /api/research | Research reports and analysis | | Analytics | /api/analytics | Platform analytics | | Ecosystem | /api/ecosystem | Ecosystem mapping | | Aggregate | /api/aggregate | Multi-source data aggregation | | Export | /api/export | Data export (CSV, JSON, Parquet) | | Search | /api/search | Semantic search across all data | | Anomalies | /api/anomalies | Anomaly detection alerts | | News Feed | /api/news-feed | Aggregated news feed (12+ sources) |

WebSocket Endpoints

📖 See docs/WEBSOCKET.md for message schemas, throttling, and client examples.

| Path | Description | |---|---| | /ws/prices | Real-time price stream (CoinCap, 5Hz throttle) | | /ws/bitcoin | Bitcoin mempool & block updates (Mempool.space) | | /ws/trades | Live DEX trades (DexScreener) | | /ws/alerts | Anomaly detection alerts (real-time) |

---

Data Sources

📖 See docs/DATA_SOURCES.md for adapter configuration and rate limit details.

37 adapters in src/sources/ connect to external APIs:

| Source | File | Data | |---|---|---| | CoinGecko | coingecko.ts | Prices, market data, trending, categories, history | | DeFiLlama | defillama.ts | TVL, yields, stablecoins, DEX volumes, bridges | | Alternative.me | alternative.ts | Fear & Greed Index | | Binance | binance.ts | Order books, trades, klines, ticker, funding | | Bybit | bybit.ts | Order books, trades, klines, funding rates | | OKX | okx.ts | Order books, trades, klines, funding rates | | Deribit | deribit.ts | Options, futures, volatility surface | | dYdX | dydx.ts | Perpetual markets, funding rates | | Hyperliquid | hyperliquid.ts | Perpetual markets, funding, open interest | | CoinCap | coincap.ts | Real-time prices (WebSocket upstream) | | CoinGlass | coinglass.ts | Open interest, liquidations, funding | | CoinLore | coinlore.ts | Global market stats | | CryptoCompare | cryptocompare.ts | Social stats, historical data | | GeckoTerminal | geckoterminal.ts | DEX pools, trending pairs, OHLCV | | Jupiter | jupiter.ts | Solana token prices, DEX aggregation | | L2Beat | l2beat.ts | Layer 2 TVL and risk data | | Messari | messari.ts | Fundamental metrics, profiles | | Token Terminal | tokenterminal.ts | Revenue, earnings, P/E ratios | | Bitcoin/Blockchain | bitcoin.ts, blockchain.ts | Network stats, mempool, hashrate | | EVM | evm.ts | EVM chain RPC data | | GoPlus | goplus.ts | Token security audits | | Snapshot | snapshot.ts | Governance proposals, voting | | DePin Scan | depinscan.ts | DePIN project data | | Yahoo Finance | macro.ts | Macro indicators (DXY, S&P 500, VIX, bonds) | | NFT Sources | nft.ts | NFT collections and marketplaces | | Oracles | oracles.ts | Chainlink, Band, Pyth data | | Social | social.ts | Social metrics | | Staking | staking.ts | Staking yields, validators | | Unlocks | unlocks.ts | Token unlock schedules | | Whales | whales.ts | Whale transaction tracking | | Portfolio | portfolio.ts | Portfolio analytics | | Calendar | calendar.ts, coinmarketcal.ts | Crypto events | | News | crypto-news.ts, news-aggregator.ts | News from 12+ outlets | | ETF | etf.ts | Crypto ETF flow data |

---

Packages

📖 See docs/PACKAGES.md for full package API documentation.

Eight packages in packages/ provide standalone functionality:

@nirholas/erc8004-agent-runtime — packages/agent-runtime

ERC-8004 compliant agent runtime for decentralized AI agents with A2A (Agent-to-Agent) messaging and x402 micropayments.

Key exports:

• ERC8004Agent — Base agent class with lifecycle management
• A2AHandler — Agent-to-Agent communication protocol handler
• TaskManager — Task queue management for agent workloads
• x402Middleware — HTTP 402 micropayment middleware
• IdentityManager — On-chain identity management
• ReputationManager — Agent reputation scoring
• searchAgents, connectToAgent — Agent discovery and connection
• Middleware: authMiddleware, rateLimitMiddleware, loggingMiddleware

Stack: TypeScript, Hono, ethers.js, EventEmitter3

@nirholas/binance-mcp-server — packages/binance-mcp

MCP (Model Context Protocol) server for Binance exchange. Provides AI-accessible tools for spot trading, staking, wallet management, NFTs, Binance Pay, and mining.

Stack: TypeScript, @modelcontextprotocol/sdk, @binance/* SDK packages

@nirholas/bnbchain-mcp — packages/bnbchain-mcp

MCP server for BNB Chain with dual support for BSC (EVM smart contracts) and Greenfield (decentralized storage).

Stack: TypeScript, @modelcontextprotocol/sdk, @bnb-chain/greenfield-js-sdk, ethers.js

@nirholas/crypto-market-data — packages/market-data

Standalone crypto market data service, Edge Runtime compatible. Aggregates CoinGecko, DeFiLlama, and Fear & Greed data with built-in caching and rate limiting.

Stack: TypeScript, Edge Runtime compatible

@crypto-vision/mcp-server — packages/mcp-server

MCP server exposing Crypto Vision intelligence tools to AI models. Includes Solana integration.

Stack: TypeScript, @modelcontextprotocol/sdk, @solana/web3.js

sweep — packages/sweep

Multi-chain dust sweeper with DeFi routing. Sweeps small token balances across chains, consolidating them via DEX routes.

Stack: TypeScript, Hono, ethers.js, @solana/web3.js, bullmq, drizzle-orm

abi-to-mcp (UCAI) — packages/ucai

Universal Contract AI Interface — Python tool that generates MCP servers from Ethereum ABI files. Published on PyPI as abi-to-mcp. Web builder at mcp.ucai.tech.

Stack: Python, mcp, web3.py

📖 See docs/PACKAGES.md for full package API documentation.

Eight packages in packages/ provide standalone functionality:

@nirholas/erc8004-agent-runtime — packages/agent-runtime

ERC-8004 compliant agent runtime for decentralized AI agents with A2A (Agent-to-Agent) messaging and x402 micropayments.

Key exports:

• ERC8004Agent — Base agent class with lifecycle management
• A2AHandler — Agent-to-Agent communication protocol handler
• TaskManager — Task queue management for agent workloads
• x402Middleware — HTTP 402 micropayment middleware
• IdentityManager — On-chain identity management
• ReputationManager — Agent reputation scoring
• searchAgents, connectToAgent — Agent discovery and connection
• Middleware: authMiddleware, rateLimitMiddleware, loggingMiddleware

Stack: TypeScript, Hono, ethers.js, EventEmitter3

@nirholas/binance-mcp-server — packages/binance-mcp

MCP (Model Context Protocol) server for Binance exchange. Provides AI-accessible tools for spot trading, staking, wallet management, NFTs, Binance Pay, and mining.

Stack: TypeScript, @modelcontextprotocol/sdk, @binance/* SDK packages

@nirholas/bnbchain-mcp — packages/bnbchain-mcp

MCP server for BNB Chain with dual support for BSC (EVM smart contracts) and Greenfield (decentralized storage).

Stack: TypeScript, @modelcontextprotocol/sdk, @bnb-chain/greenfield-js-sdk, ethers.js

@nirholas/crypto-market-data — packages/market-data

Standalone crypto market data service, Edge Runtime compatible. Aggregates CoinGecko, DeFiLlama, and Fear & Greed data with built-in caching and rate limiting.

Stack: TypeScript, Edge Runtime compatible

@crypto-vision/mcp-server — packages/mcp-server

MCP server exposing Crypto Vision intelligence tools to AI models. Includes Solana integration.

Stack: TypeScript, @modelcontextprotocol/sdk, @solana/web3.js

Stack: TypeScript, Hono, ethers.js, grammy, bullmq, drizzle-orm

sweep — packages/sweep

Multi-chain dust sweeper with DeFi routing. Sweeps small token balances across chains, consolidating them via DEX routes.

Stack: TypeScript, Hono, ethers.js, @solana/web3.js, bullmq, drizzle-orm

abi-to-mcp (UCAI) — packages/ucai

Universal Contract AI Interface — Python tool that generates MCP servers from Ethereum ABI files. Published on PyPI as abi-to-mcp. Web builder at mcp.ucai.tech.

Stack: Python, mcp, web3.py

---

Apps

Three frontend applications in apps/:

Dashboard — apps/dashboard

crypto-data-aggregator — Next.js dashboard for real-time crypto market data visualization. Features DeFi analytics, portfolio tracking, and watchlists.

Stack: Next.js, React, TypeScript

News — apps/news

free-crypto-news — Next.js crypto news aggregator with AI-powered analysis. Aggregates 12+ sources with sentiment tracking. Includes MCP integration, mobile app, CLI tool, browser extension, and embeddable widget. README available in 50+ languages.

Stack: Next.js, React, TypeScript, MCP

Video — apps/video

@crypto-vision/video — Remotion project for generating x402 Gas Station promotional videos programmatically.

Stack: Remotion, React, TypeScript

---

AI Agents

📖 See docs/AI_AGENTS.md for the full agent catalog and prompt engineering guide.

58 specialized AI agents in the agents/ directory (@nirholas/ai-agents-library v1.42.0):

Agent Categories

| Category | Count | Examples | |---|---|---| | Trading | 8 | Swing Trader, Position Sizer, MEV Shield, Leverage Calculator | | DeFi | 12 | Yield Farmer, Protocol Comparator, Insurance Advisor, Risk Scoring Engine | | Portfolio | 5 | Portfolio Optimizer, Tax Strategist, Dollar-Cost Averaging | | Security | 4 | Bridge Security Analyst, Rug Pull Detector, Smart Contract Auditor | | Education | 6 | DeFi Onboarding Mentor, APY vs APR Educator, Impermanent Loss Calculator | | Research | 8 | Alpha Leak Detector, Whale Tracker, On-Chain Forensics, Governance Analyst | | Ecosystem | 15 | Sperax-specific agents (SPA Staker, USDs Navigator, Demeter Farmer, etc.) |

Agent Invocation

# List all agents
curl http://localhost:8080/api/agents
Invoke a specific agent
curl -X POST http://localhost:8080/api/agents/defi-yield-farmer/invoke \
  -H "Content-Type: application/json" \
  -d '{"query": "Find the best stablecoin yields above 10% APY"}'
Multi-agent orchestration
curl -X POST http://localhost:8080/api/agents/orchestrate \
  -H "Content-Type: application/json" \
  -d '{"query": "Analyze ETH risk and suggest a DeFi strategy", "agents": ["defi-risk-scoring-engine", "defi-yield-farmer"]}'

Supported Languages

Agent prompts are translated into 18 languages: English, Spanish, French, German, Portuguese, Italian, Russian, Chinese (Simplified & Traditional), Japanese, Korean, Arabic, Turkish, Hindi, Vietnamese, Thai, Indonesian, Dutch.

---

Workers & Ingestion

📖 See docs/WORKERS.md for worker configuration, scheduling, and troubleshooting.

15 worker files in src/workers/ provide background data ingestion:

Worker Architecture

All workers extend WorkerBase which provides:

• Periodic fetching with configurable intervals
• Dual-write to BigQuery (streaming inserts) and Pub/Sub
• Exponential backoff on failures
• Graceful shutdown handling
• Health reporting

Available Workers

| Worker | File | Interval | Description | |---|---|---|---| | Market Ingest | ingest-market.ts | 1-2 min | Prices, market cap, volume from CoinGecko | | DeFi Ingest | ingest-defi.ts | 5 min | TVL, yields, protocol data from DeFiLlama | | News Ingest | ingest-news.ts | 10 min | Crypto news from multiple outlets | | DEX Ingest | ingest-dex.ts | 2 min | DEX trades, pools, trending pairs | | Derivatives Ingest | ingest-derivatives.ts | 2 min | Options, futures, funding rates | | Governance Ingest | ingest-governance.ts | 30 min | DAO proposals, votes | | Macro Ingest | ingest-macro.ts | 1 hour | Macroeconomic indicators | | On-Chain Ingest | ingest-onchain.ts | 5 min | On-chain metrics, gas prices |

Indexing Workers

| Worker | File | Description | |---|---|---| | Agent Indexer | index-agents.ts | Indexes AI agents into vector store | | Governance Indexer | index-governance.ts | Indexes governance proposals | | News Indexer | index-news.ts | Indexes news articles for RAG | | Protocol Indexer | index-protocols.ts | Indexes DeFi protocol data | | Historical Backfill | backfill-historical.ts | Backfills historical data |

Running Workers

# Via Docker Compose
docker compose -f docker-compose.ingest.yml up
Individual workers
npm run worker:market
npm run worker:defi
npm run worker:news
All workers
npm run workers

---

Telegram Bot (crypto vision)

📖 See docs/TELEGRAM_BOT.md for bot commands, configuration, and deployment.

A full-featured Telegram trading bot in src/bot/ built on the grammy framework.

Features

• Trading Calls — Post buy/sell calls with P&L tracking
• Leaderboard — Global and group leaderboards ranked by ROI
• Hardcore Mode — Gamified challenge mode with elimination and prizes
• Portfolio Tracking — Track positions across tokens
• PNL Cards — Visual profit/loss summary cards
• Insider Alerts — Smart money movement notifications
• Premium Tiers — Subscription-based premium features
• Referral System — User referral tracking and rewards
• Multi-Group Support — Works across multiple Telegram groups

Database Schema

PostgreSQL schema at src/bot/db/schema.ts (Drizzle ORM) includes tables for:

• Users, groups, channels
• Trading calls with entry/exit prices
• Leaderboard snapshots
• Hardcore mode sessions and participants
• PNL history
• Premium subscriptions
• Referral tracking
• Insider alerts

Enabling the Bot

# Set environment variables
SECTBOT_ENABLED=true
BOT_TOKEN=your-telegram-bot-token
DATABASE_URL=postgresql://user:pass@host:5432/sectbot
The bot starts automatically with the main server
npm run dev

---

Infrastructure

📖 See docs/INFRASTRUCTURE.md for Terraform modules and docs/DEPLOYMENT.md for CI/CD pipelines.

Google Cloud Platform (GCP)

The primary deployment target is GCP Cloud Run with supporting services:

| Service | Purpose | |---|---| | Cloud Run | Main API + worker containers | | BigQuery | Data warehouse (22+ tables) | | Pub/Sub | Event messaging (5-tier topic architecture) | | Cloud Scheduler | Cron jobs for workers, retraining | | Memorystore (Redis) | Distributed cache | | Secret Manager | API keys and credentials | | Vertex AI | Embeddings and model training | | VPC | Private networking |

Terraform

17 Terraform files in infra/terraform/ define the complete infrastructure:

cd infra/terraform
terraform init
terraform plan
terraform apply

Key files: main.tf, cloud_run.tf, redis.tf, bigquery.tf, pubsub.tf, scheduler.tf, secrets.tf, network.tf, iam.tf, monitoring.tf, vertex.tf, gke-gpu.tf, export.tf, apis.tf

Kubernetes

12 Kubernetes manifests in infra/k8s/ for GKE deployment:

• deployment.yaml — Main API deployment
• hpa.yaml — Horizontal Pod Autoscaler
• redis.yaml — Redis StatefulSet
• cronjobs.yaml — Ingestion worker CronJobs
• network-policy.yaml — Network policies
• training-job.yaml — ML training Job
• inference-deployment.yaml — Model inference serving

Pub/Sub Topic Architecture

5-tier messaging defined in infra/pubsub/topics.yaml:

| Tier | Latency | Topics | |---|---|---| | Realtime | < 1s | market-prices-realtime, dex-trades-realtime | | Frequent | 1-2 min | market-data-frequent, dex-pools-frequent | | Standard | 5-10 min | defi-metrics-standard, news-articles-standard | | Hourly | 1 hour | governance-hourly, macro-indicators-hourly | | Daily | 24 hours | analytics-daily, embeddings-daily |

CI/CD

Google Cloud Build pipelines:

• cloudbuild.yaml — Main API build and deploy
• cloudbuild-workers.yaml — Worker containers build and deploy

Quick Setup

# Full GCP setup (interactive)
bash infra/setup.sh
Teardown
bash infra/teardown.sh

---

Training & Fine-Tuning

📖 See docs/TRAINING.md for the full training pipeline guide.

Complete ML pipeline in scripts/training/:

Training Data Generation

# Generate training data from BigQuery
npx tsx scripts/training/generate-training-data.ts
Validate generated data
npx tsx scripts/training/validate-data.ts

Gemini Fine-Tuning

# Fine-tune Gemini via Vertex AI
npx tsx scripts/training/finetune-gemini.ts
Evaluate fine-tuned models
npx tsx scripts/training/eval-models.ts

Open-Source Model Training

# Prepare data in Alpaca format
npx tsx scripts/training/opensource/prepare-data.ts
Train with LoRA (QLoRA 4-bit quantization)
python scripts/training/opensource/train.py
Benchmark against baselines
python scripts/training/opensource/benchmark.py
Export to GGUF/ONNX
python scripts/training/opensource/export.py

Inference Serving

# Serve fine-tuned model with vLLM
python scripts/inference/serve.py
Health check
python scripts/inference/healthcheck.py

Kubernetes resources: infra/k8s/training-job.yaml, infra/k8s/inference-deployment.yaml

---

BigQuery Data Warehouse

📖 See docs/BIGQUERY.md for schema details, partitioning, and query examples.

22+ tables defined in infra/bigquery/tables.sql:

Table Categories

| Category | Tables | |---|---| | Market | marketprices, marketglobal, feargreedindex | | DeFi | defiprotocols, defiyields, defitvlchains, defistablecoins, dexpools | | Derivatives | derivativesoptions, derivativesfutures, derivativesfunding, derivativesoi | | Blockchain | onchaingas, bitcoinstats, bitcoin_mempool | | Governance | governanceproposals, governancevotes | | News | news_articles | | Macro | macro_indicators | | AI | embeddings, searchqueries, aicompletions |

Materialized Views

Defined in infra/bigquery/views.sql:

• Hourly/daily price aggregations
• Protocol TVL rankings
• Cross-exchange funding rate comparisons
• Market anomaly summaries

Data Export

# Export all BigQuery data
npm run export:all
Export embeddings
npm run export:embeddings
Import to PostgreSQL
npm run import:postgres

---

Testing

📖 See docs/TESTING.md for test writing guidelines and CI integration.

Comprehensive test suite using Vitest:

Test Categories

| Type | Location | Count | Description | |---|---|---|---| | Unit | tests/lib/ | 33 files | Every core library module | | Route | tests/routes/ | 10 files | API route handlers | | Integration | tests/integration/ | 1 file | End-to-end API flows | | E2E | tests/e2e/ | 1 file | Smoke tests against running server | | Benchmarks | tests/benchmarks/ | 1 file | Performance benchmarks | | Fuzz | tests/fuzz/ | 1 file | Fuzz testing for API inputs | | Load | tests/load/ | 3 files | Load tests (smoke, soak, stress) |

Running Tests

# All tests
npm test
With coverage
npm run test:coverage
E2E tests (requires running server)
npm run test:e2e
Specific test file
npx vitest tests/lib/cache.test.ts
Watch mode
npx vitest --watch

Key Test Files

• tests/lib/cache.test.ts — Two-tier cache with stampede protection
• tests/lib/fetcher.test.ts — Circuit breaker and retry logic
• tests/lib/anomaly.test.ts — Anomaly detection algorithms
• tests/lib/queue.test.ts — Bounded concurrency queue
• tests/lib/auth.test.ts — API key authentication tiers
• tests/lib/search.test.ts — Semantic search and intent detection
• tests/lib/embeddings.test.ts — Multi-provider embedding generation
• tests/lib/rag.test.ts — RAG pipeline
• tests/lib/orchestrator.test.ts — Multi-agent orchestration

---

Monitoring & Observability

📖 See docs/MONITORING.md for Grafana dashboards, alerting rules, and docs/ANOMALYDETECTION.md for the anomaly detection engine.

Prometheus Metrics

Available at /metrics:

| Metric | Type | Description | |---|---|---| | httprequeststotal | Counter | Total HTTP requests by method, path, status | | httprequestduration_seconds | Histogram | Request latency distribution | | upstreamrequeststotal | Counter | Upstream API calls by source, status | | upstreamrequestduration_seconds | Histogram | Upstream latency | | cachehitstotal / cachemissestotal | Counter | Cache hit/miss rates | | cache_size | Gauge | Current cache size | | websocket_connections | Gauge | Active WebSocket connections | | queue_size | Gauge | Current queue depth | | queue_concurrency | Gauge | Active concurrent tasks | | circuitbreakerstate | Gauge | Circuit breaker states per host |

Structured Logging

Pino-based structured JSON logging with:

• Request ID tracking
• Latency measurement
• Error context
• Configurable log levels via LOG_LEVEL

Health Check

GET /health returns:

{
  "status": "healthy",
  "uptime": 3600,
  "cache": { "size": 1500, "hitRate": 0.85 },
  "circuitBreakers": { "coingecko": "CLOSED", "defillama": "CLOSED" },
  "queue": { "pending": 2, "active": 1 },
  "websocket": { "connections": 15 },
  "memory": { "heapUsed": "150MB", "rss": "250MB" }
}

---

Self-Hosting

See docs/SELF_HOSTING.md for the complete self-hosting guide, including:

• Minimal setup (Node.js only, no cloud dependencies)
• Docker deployment
• Full GCP deployment with Terraform
• Scaling and production hardening
• Cost estimation

---

Performance

See docs/PERFORMANCE.md for detailed benchmarks and optimization guide, including:

• Response time targets (< 200ms cached, < 2s upstream)
• Cache optimization strategies
• Connection pooling
• Memory management
• Load testing methodology

---

Project Structure

crypto-vision/
├── src/                        # Main application source
│   ├── index.ts                # Entry point — Hono server, middleware, route mounting
│   ├── routes/                 # 40 route modules (market, defi, ai, agents, etc.)
│   ├── sources/                # 37 data source adapters
│   ├── lib/                    # Core libraries (cache, fetcher, queue, auth, anomaly, etc.)
│   ├── workers/                # 15 background ingestion & indexing workers
│   └── bot/                    # Telegram crypto vision (grammy)
│       └── db/schema.ts        # PostgreSQL schema (Drizzle ORM)
├── packages/                   # Standalone packages
│   ├── agent-runtime/          # ERC-8004 agent runtime (A2A + x402)
│   ├── binance-mcp/            # Binance MCP server
│   ├── bnbchain-mcp/           # BNB Chain MCP server
│   ├── market-data/            # Standalone market data service
│   ├── mcp-server/             # Crypto Vision MCP server
│   ├── pump-agent-swarm/       # Pump.fun agent swarm
│   ├── sweep/                  # Multi-chain dust sweeper
│   └── ucai/                   # ABI-to-MCP generator (Python)
├── apps/                       # Frontend applications
│   ├── dashboard/              # Next.js market dashboard
│   ├── news/                   # Next.js crypto news aggregator
│   └── video/                  # Remotion video project
├── agents/                     # 58 AI agents library
│   ├── prompts/                # Agent system prompts
│   ├── locales/                # 18 language translations
│   ├── schema/                 # Agent JSON schemas
│   └── docs/                   # Agent documentation
├── infra/                      # Infrastructure as Code
│   ├── terraform/              # 17 Terraform configs (GCP)
│   ├── k8s/                    # 12 Kubernetes manifests
│   ├── pubsub/                 # Pub/Sub topic definitions
│   ├── bigquery/               # BigQuery schema & views
│   └── scheduler/              # Cloud Scheduler jobs
├── scripts/                    # Operational scripts
│   ├── training/               # ML training pipeline
│   ├── inference/              # Model serving
│   ├── demo/                   # Demo scripts
│   ├── export-all.ts           # BigQuery data export
│   └── import-to-postgres.ts   # PostgreSQL import
├── tests/                      # Comprehensive test suite
│   ├── lib/                    # 33 unit test files
│   ├── routes/                 # 10 route test files
│   ├── e2e/                    # End-to-end smoke tests
│   ├── integration/            # API integration tests
│   ├── benchmarks/             # Performance benchmarks
│   ├── fuzz/                   # Fuzz testing
│   └── load/                   # Load tests (smoke, soak, stress)
├── docs/                       # Documentation
│   ├── AGENTS.md               # 58 AI agents guide
│   ├── API_REFERENCE.md        # Complete 300+ endpoint reference
│   ├── ARCHITECTURE.md         # System architecture & data flow
│   ├── COINGECKORATELIMITING.md # CoinGecko rate limiting strategy
│   ├── CONFIGURATION.md        # Environment variables reference
│   ├── DATABASE.md             # PostgreSQL & BigQuery schemas
│   ├── DATA_PIPELINE.md        # Workers, ingestion, Pub/Sub
│   ├── DATA_SOURCES.md         # 37+ upstream data sources
│   ├── DEPLOYMENT.md           # Docker, Cloud Run, K8s deployment
│   ├── DEVELOPER_WORKFLOW.md   # Day-to-day dev commands
│   ├── INFRASTRUCTURE.md       # Terraform, K8s, CI/CD
│   ├── ML_TRAINING.md          # Model training & fine-tuning
│   ├── PACKAGES.md             # All 8 packages deep-dive
│   ├── PERFORMANCE.md          # Performance & optimization
│   ├── REPOSITORY_GUIDE.md     # Full repo structure
│   ├── SELF_HOSTING.md         # Self-hosting guide
│   ├── TELEGRAM_BOT.md         # crypto vision documentation
│   ├── TESTING.md              # Test strategy & coverage
│   └── X402_PAYMENTS.md        # x402 micropayment system
├── prompts/                    # Engineering prompt guides
├── docker-compose.yml          # API + Redis stack
├── docker-compose.ingest.yml   # Ingestion workers
├── Dockerfile                  # Main API container
├── Dockerfile.worker           # Worker container
├── Dockerfile.train            # Training container
├── cloudbuild.yaml             # CI/CD — API
├── cloudbuild-workers.yaml     # CI/CD — Workers
├── drizzle.config.ts           # Drizzle ORM config
├── openapi.yaml                # OpenAPI 3.1 specification
├── vitest.config.ts            # Test configuration
└── tsconfig.json               # TypeScript configuration

---

Documentation

Comprehensive documentation is available in the docs/ directory:

| Document | Description | |----------|-------------| | AGENTS.md | 58 AI agents — categories, architecture, orchestration, localization | | ANOMALY_DETECTION.md | Real-time statistical anomaly detection — 16 event types, algorithms, processors | | API_AUTHENTICATION.md | 4-tier API key authentication, rate limiting, key management API | | API_REFERENCE.md | Complete 538+ endpoint reference across 40 route modules | | ARCHITECTURE.md | System architecture, data flow, caching, circuit breakers | | COINGECKORATE_LIMITING.md | CoinGecko API rate limiting strategy and optimization | | CONFIGURATION.md | All environment variables, API keys, secrets management | | DATABASE.md | PostgreSQL schema (Drizzle ORM) and BigQuery table definitions | | DATA_PIPELINE.md | Ingestion workers, Pub/Sub tiers, scheduling | | DATA_SOURCES.md | 37+ upstream data source adapters and their capabilities | | DEPLOYMENT.md | Docker, Cloud Run, Kubernetes deployment guides | | DEVELOPER_WORKFLOW.md | Day-to-day development commands and workflows | | INFRASTRUCTURE.md | Terraform, GCP resources, K8s manifests, CI/CD pipelines | | ML_TRAINING.md | Model fine-tuning (Gemini, Llama, Mistral), GPU training, vLLM inference | | MONITORING.md | Prometheus metrics, structured logging, health checks, alerting | | PACKAGES.md | All 8 packages — agent-runtime, MCP servers, market-data, sweep, ucai | | PERFORMANCE.md | Benchmarks, optimization techniques, caching strategies | | REPOSITORY_GUIDE.md | Full repository structure walkthrough | | SECURITY_GUIDE.md | Security architecture, threat mitigations, best practices | | SELF_HOSTING.md | Complete self-hosting guide with data migration | | TELEGRAM_BOT.md | crypto vision — Telegram bot commands, features, database | | TESTING.md | Test strategy, coverage, unit/integration/e2e/fuzz/load tests | | TROUBLESHOOTING.md | Common issues, diagnostics, and solutions | | WEBSOCKET.md | Real-time WebSocket feeds — prices, Bitcoin, trades, alerts | | X402_PAYMENTS.md | x402 micropayment protocol integration |

---

License

AGPL-3.0

---

Contributing

See CONTRIBUTING.md for contribution guidelines and SECURITY.md for security policy.

---

Built with TypeScript, Hono, and an unhealthy obsession with crypto data.