#Scraper
Showing 60 of 175 repositories tagged #scraper, ranked by stars
The API to search, scrape, and interact with the web at scale. 🔥
Create agents that monitor and act on your behalf. Your agents are standing by!
A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。
👾 Fast and simple video download library and CLI tool written in Go
AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.
Elegant Scraper and Crawler Framework for Golang
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
:orange_book: 中华新华字典数据库。包括歇后语，成语，词语，汉字。
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
A Chrome DevTools Protocol driver for web automation and scraping.
Turn any webpage into structured data using LLMs
A social networking service scraper in Python
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
Swiss-army tool for scraping and extracting data from online assets, made for hackers
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
A community-driven way to read and chat with AI bots - powered by chatGPT.
Movie metadata scraper
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Linkedin Automation Tool: Describe your product. Define your target market. The AI finds the leads for you.
Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App
📖 The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!
Completely free and open-source human-like Instagram bot. Powered by UIAutomator2 and compatible with basically any Android device 5.0+ that can run Instagram - real or emulated.
Information Gathering Instagram.
电视剧/番剧自动化重命名工具, 一键批量改名. 可配合QBittorrent下载后自动重命名, 方便Emby自动刮削. 支持Windows, Linux, MacOS, Docker 和 群晖套件环境运行
An ergonomic, censorship-resistant Python HTTP Client
Scrapes Reddit to download media of your choice.
Scalable Python web scraping scripts for +40 popular domains
Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.
Google play scraper for Python inspired by <facundoolano/google-play-scraper>
Scrape content from OnlyFans and Fansly
An ergonomic, censorship-resistant Rust HTTP Client
Fetch X/Twitter tweets, replies, timelines, and articles without login or API keys — field tool for AI agents.
First ever tool to view "Instagram private posts" anonymously
HTTP API for Scrapy spiders
A Kotlin-based testing/scraping/parsing library providing the ability to analyze and extract data from HTML (server & client-side rendered). It places particular emphasis on ease of use and a high level of readability by providing an intuitive DSL. It aims to be a testing lib, but can also be used to scrape websites in a convenient fashion.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
[Unmaintained] A simple and clean video/music/image downloader 👾
Make a ZIM file from any Web site and surf offline!
:bar_chart: Financial markets data library implemented in go.
:scissors: High performance, multi-threaded image scraper
operative framework is a rust investigation OSINT framework, you can interact with multiple targets, execute multiple modules, create links with target, export rapport to PDF file, add note to target or results, interact with RESTFul API, write your own modules.
Extract structured data from web sites. Web sites scraping.
A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user
The Slash OSINT Tool
Automated Tinder bot and scraper using selenium in python.
OSINT Swiss Army Knife
Complete-Life-Cycle-of-a-Data-Science-Project
Linkedin Automation Bot with every possible scraping! Valid for 2022 used by Linvo.io
Universal Reddit Scraper - Works on any Subreddit or User
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
Web scraper made for AI and simplicity in mind. It runs as a CLI that can be parallelized and outputs high-quality markdown content.
Little tool in python to watch and download anime from the terminal (the better way to watch anime). Also applicable as an API