#Scraping
Showing 60 of 159 repositories tagged #scraping, ranked by stars
The API to search, scrape, and interact with the web at scale. π₯
π·οΈ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!
Scrapy, a fast high-level web crawling & scraping framework for Python.
π΅οΈββοΈ Collect a dossier on a person by username from 3000+ sites
AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.
Python scraper based on AI
Elegant Scraper and Crawler Framework for Golang
CrawleeβA web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Pythonic HTML Parsing for Humansβ’
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
CrawleeβA web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Stealth headless browser for AI agents β bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Self-hosted webscraper.
Swiss-army tool for scraping and extracting data from online assets, made for hackers
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
Collection of useful data science topics along with articles, videos, and code
Up-to-date simple useragent faker with real world database
Snoop β ΠΈΠ½ΡΡΡΡΠΌΠ΅Π½Ρ ΡΠ°Π·Π²Π΅Π΄ΠΊΠΈ Π½Π° ΠΎΡΠ½ΠΎΠ²Π΅ ΠΎΡΠΊΡΡΡΡΡ Π΄Π°Π½Π½ΡΡ (OSINT world)
AnyCrawl π: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Web data
A curated list of awesome puppeteer resources.
Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Web Scraping Framework
Linkedin Automation Tool: Describe your product. Define your target market. The AI finds the leads for you.
OSINT cheat sheet, list OSINT tools, wiki, dataset, article, book , red team OSINT for hackers and OSINT tips and OSINT branch. This repository will grow every time will research, there is a research, science and technology, tutorial. Please use it wisely.
Anti-Detect Browser that passes every bot detection test. Drop-in Playwright replacement.
Example end to end data engineering project.
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.
π° DiΓ‘rios oficiais brasileiros acessΓveis a todos | π° Brazilian government gazettes, accessible to everyone.
Open Source Bulk Auto Gmail Creator Bot with Selenium & Seleniumwire ( Python ). Feel free to contact me with Django/Flask, ML, AI, GPT, Automation, Scraping.
Lightweight library for scraping web-sites with LLMs
Watch everything from your terminal.
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
Automatically archive links to videos, images, and social media content from Google Sheets (and more).
Your browser anime experience from the terminal
modular service framework to move and transform network packets
Scalable Python web scraping scripts for +40 popular domains
Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.
Generate Free Edu Mail(s) within minutes
The agent that turns websites into APIs!
A CLI toolset to generate table of contents for PDF files automatically.
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
[Unmaintained] A simple and clean video/music/image downloader πΎ
:scissors: High performance, multi-threaded image scraper
Google Search Results via SERP API pip Python Package
Extract structured data from web sites. Web sites scraping.
Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
The simplest demo of chrome automation by python and selenium in AWS Lambda
Python 3 script to dump/scrape/extract company employees from LinkedIn API
Free Web Scraping Tool with Java
Universal Reddit Scraper - Works on any Subreddit or User