AI Context

 

Capabilities • Installation • Usage • Tips & Notes

Generate AI-friendly markdown files from local code or GitHub repos using a multi-arch, multi-OS Go CLI tool to make your interactions with LLMs (like ChatGPT, Claude, etc.) easy.

Capabilities

| Category | Commands | Description | |----------|----------|-------------| | Processing | ai-context [url/path] | Process local directories or GitHub repositories | | Batch | ai-context -f [file] | Process multiple directories/repositories concurrently from a list file | | Stats | ai-context stats [file] | View lines, words, chars, and estimated LLM tokens for a file |

Installation

Binary

Download from releases:

# Linux/macOS
ARCH=$(uname -m); [ "$ARCH" = "x86_64" ] && ARCH=amd64; [ "$ARCH" = "aarch64" ] && ARCH=arm64
curl -sL https://github.com/tanq16/ai-context/releases/latest/download/ai-context-$(uname -s | tr '[:upper:]' '[:lower:]')-$ARCH -o ai-context
chmod +x ai-context
sudo mv ai-context /usr/local/bin/

Build from Source

git clone https://github.com/tanq16/ai-context.git
cd ai-context
make build

Usage

Processing

Generate context from a single source.

# Process a single path (local directory) with additional exclude patterns
ai-context /path/to/directory  -e "tests,docs,doc."
Only include specific file types with max size limit
ai-context /path/to/directory -i ".go,.md" -s 5242880
Process a public GitHub repository
ai-context https://github.com/tanq16/ai-context
Process private GitHub repository
GH_TOKEN=$(cat /secrets/GH.PAT) ai-context https://github.com/ORG/REPO

Flags:

• --include, -i - Include files matching globs (e.g., '.go,.md')
• --exclude, -e - Exclude files matching globs (e.g., 'tests,docs')
• --max-size, -s - Maximum file size in bytes to include (default 10MB)
• --debug - Enable debug logging
• --for-ai - AI-friendly output (plain text, piped input)

Batch Processing

Generate context from multiple sources listed in a file.

# Make a list of paths
cat << EOF > listfile
../notif
/working/cybernest
https://github.com/assetnote/h2csmuggler
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html
EOF
Process URL list concurrently
ai-context -f listfile

Flags:

• --file, -f - File with list of URLs to process
• --threads, -t - Number of threads to use for processing (default: 10)

File Stats & Token Estimation

Analyze any generated context file (or any local file) to see its lines, words, characters, size, and an estimated LLM token count. The token heuristic is mathematically tuned for BPE tokenizers (like GPT-4 and Claude) and is highly accurate for both prose and code.

ai-context stats context/example.md

Tips and Notes

• For directory path (in URL or listfile mode), the path should either start with / (absolute) or with ./ or ../ (relative). For current directory, always use ./ for correct regex matching.
• Do a head -n 200 context/FILE.md (or 500 lines) to view the content tree of the processed code base or directory to see what's been included. Then refine your -e flag arguments to exclude additional patterns.
• The --for-ai flag produces plain text without ANSI colors, which is easier for AI agents to parse.

Acknowledgments

This project takes inspiration from, uses, or references:

• repomix: inspiration for turning code into context
• innertube: inspiration for code to get transcript from YouTube video
• html-to-markdown: used to convert HTML to MD
• go-git: git operations in Go