#Robots-txt
Showing 9 of 9 repositories tagged #robots-txt, ranked by stars
🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Ultimate Website Sitemap Parser
Open-source GEO / AEO skill for AI visibility readiness: check robots.txt, llms.txt, schema, citation signals, competitor gaps, and buyer prompts
Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.
🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs
ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.
If you're using Vercel and Google hates your site, this is for you.
gorobots - robots.txt recon & path discovery in Go. Structured parsing, 29-category sensitivity classification, concurrent path probing, bulk domain scanning.