Moroccan housing data pipeline using scrapy, mongodb , zyte and digitalocean cloud
Last updated May 1, 2024
11
Stars
0
Forks
0
Issues
0
Stars/day
Attention Score
0
Language breakdown
Python 100.0%
▸ Files
click to expand
README
MoroccanHousing-ETL
This project is a data pipeline built with scrapy spiders that crawl the famous housing moroccan website avito.ma , the pipeline extract data from the website and it's deployed on Zyte (Scrapy cloud) and the data collected will be loaded to MongoDB on Digital ocean cloud
Requirements
- Python >= 3.8
- Installing other requirements with the following command:
pip install -r requirements.txtUsage
First you need to change the value of the variable MONGO_URI with your connection string of your MongoDB, Then you can simply run the spider :
scrapy crawl avitoSpider
