Jobsite Scraper and Analyzer

Overview

A comprehensive web scraping and data analysis platform that extracts job listings from theprotocol.it to analyze technology demand trends for developers in the Polish job market. The system provides automated data collection, intelligent analysis, and interactive visualizations to help developers understand market requirements and prioritize their learning efforts.

🚀 Features

Core Functionality

• Multi-Position Support: Configurable job position targeting (Python, Java, JavaScript, etc.)
• Automated Scraping: Scheduled data collection using Celery and Redis
• Intelligent Data Processing: Natural language processing with NLTK and fuzzy matching
• Interactive Visualizations: Comprehensive charts and graphs using Matplotlib
• RESTful API: Flask-based web service for data access
• Containerized Deployment: Full Docker and Docker Compose support

Technical Capabilities

• Advanced Web Scraping: Scrapy + Selenium integration for JavaScript-rendered content
• Robust Data Pipeline: Custom ItemPipelines and Middlewares
• Database Integration: MySQL with SQLAlchemy ORM
• Task Queue Management: Celery with Redis broker
• Production-Ready: Nginx reverse proxy configuration

🛠 Technology Stack

| Category | Technologies | |----------|-------------| | Web Scraping | Scrapy, Selenium, scrapy-selenium4 | | Data Processing | Pandas, NLTK, fuzzywuzzy | | Visualization | Matplotlib, Jupyter | | Web Framework | Flask, SQLAlchemy | | Task Queue | Celery, Redis | | Database | MySQL | | Infrastructure | Docker, Docker Compose, Nginx |

📊 Analytics Dashboard

The platform generates comprehensive visualizations including:

• Skills Analysis: Required vs. optional technical skills
• Experience Levels: Distribution of seniority requirements
• Employment Types: Contract types and arrangements
• Geographic Distribution: Location-based job distribution
• Market Trends: Technology demand over time
• Ukraine Support: Companies supporting Ukrainian developers

🚀 Quick Start

Prerequisites

• Docker and Docker Compose installed
• Git

Installation

• Clone the repository

git clone https://github.com/danieladdisonorg/Jobsite-Scraper-and-Analyzer.git
cd Jobsite-Scraper-and-Analyzer

• Configure environment variables

cp .env.sample .env

• Launch the application

docker-compose up --build

• Access the application

📖 API Documentation

Endpoints

| Endpoint | Method | Description | |----------|--------|-------------| | /scraping/diagrams | GET | Initiates data analysis and returns Celery task ID | | /scraping/diagrams/<task_id> | GET | Checks task status and retrieves results |

Configuration

Modify config.py to customize scraping parameters:

# Target job position (lowercase)
POSITION = "python"  # Options: "java", "javascript", "dev", etc.
Scraping frequency
SCRAPINGEVERYDAYS = 7

🏗 Architecture

├── analyzing/          # Data analysis and visualization modules
├── web_server/         # Flask web application
├── main_celery/        # Celery configuration and tasks
├── static/             # Static assets and scraping results
├── config.py           # Application configuration
├── docker-compose.yml  # Container orchestration
└── requirements.txt    # Python dependencies

📈 Sample Visualizations

Skills by Experience Level

Required Technical Skills

Optional Technical Skills

Experience Level Distribution

Employment Types

Geographic Distribution

🔧 Development

Key Learning Outcomes

• Web Scraping Mastery: Advanced techniques with Scrapy and Selenium
• Data Pipeline Development: ETL processes and data transformation
• Asynchronous Task Processing: Celery and Redis implementation
• Containerization: Docker and microservices architecture
• Data Visualization: Statistical analysis and chart generation

Future Enhancements

High Priority

• [ ] Cloud Storage Integration: Migrate from local file storage to cloud solutions (AWS S3, Google Cloud Storage)
• [ ] Database Optimization: Implement proper data warehousing for scraped content
• [ ] Container Optimization: Reduce Docker image sizes and improve build efficiency

Medium Priority

• [ ] Code Architecture: Refactor to follow SOLID principles and improve modularity
• [ ] Comprehensive Testing: Unit, integration, and end-to-end test coverage
• [ ] CI/CD Pipeline: Automated testing and deployment workflows
• [ ] API Documentation: OpenAPI/Swagger integration

Low Priority

• [ ] Real-time Analytics: WebSocket integration for live data updates
• [ ] Machine Learning: Predictive analytics for job market trends
• [ ] Multi-language Support: Expand beyond Polish job market

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Contact

Daniel Addison

• GitHub: @danieladdisonorg
• Project Link: https://github.com/danieladdisonorg/Jobsite-Scraper-and-Analyzer