#Webcrawling

scrapinghub

scrapyrt

HTTP API for Scrapy spiders

Aavache

LLMWebCrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

Skumarr53

Stock-Fundamental-data-scraping-and-analysis

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go