Y Combinator Directory Scraper

A Python scraper for extracting company data from the Y Combinator directory, featuring an interactive web-based explorer.

Features

• 🚀 User-Friendly Selenium Scraping: No API keys required - just Firefox and geckodriver
• 💾 30-Day URL Caching: Avoid unnecessary re-scraping
• 🔄 Checkpoint/Resume: Recover from interrupted scrapes
• 🎯 Flexible Batch Filtering: Select specific batches or recent N batches
• 🌐 Interactive Web Explorer: Browse and filter companies with a sleek UI
• 📊 Rich Dataset: Includes founder profiles with bios and social links

About Y Combinator

Y Combinator is a startup accelerator that has invested in over 4,000 companies with a combined valuation exceeding $600B. Notable alumni include Airbnb, Stripe, DoorDash, Coinbase, and Reddit.

Requirements

• Python: 3.11 or higher
• Browser: Firefox
• WebDriver: geckodriver

Installing geckodriver

macOS:

brew install geckodriver

Linux:

wget https://github.com/mozilla/geckodriver/releases/download/v0.35.0/geckodriver-v0.35.0-linux64.tar.gz tar -xvzf geckodriver-v0.35.0-linux64.tar.gz sudo mv geckodriver /usr/local/bin/

Windows: Download from GitHub releases and add to PATH.

Installation

• Clone the repository

git clone https://github.com/corralm/yc-scraper.git
   cd yc-scraper

• Create a virtual environment (recommended)

python -m venv .venv
   source .venv/bin/activate  # On Windows: .venv\Scripts\activate

• Install dependencies

pip install -r requirements.txt

Usage

Quick Start

# Step 1: Extract company URLs (takes 10-15 minutes)
python yclinksextractor.py
Step 2: Scrape company data
cd scrapy-project
scrapy runspider ycombinator/spiders/yscraper.py -o output.jl
Step 3: Analyze data with Pandas
python -c "import pandas as pd; df = pd.read_json('output.jl', lines=True); print(df.head())"

Step 1: Extract Company URLs

python yclinksextractor.py

This script:

• Opens the YC directory in headless Firefox
• Iterates through all batch filters (Summer 2007 - present)
• Collects unique company URLs
• Saves to scrapy-project/ycombinator/start_urls.txt

Step 2: Scrape Company Data

cd scrapy-project
scrapy runspider ycombinator/spiders/yscraper.py -o output.jl

Supported output formats:

• JSON Lines (.jl) - Recommended for large datasets
• JSON (.json) - Standard JSON array
• CSV (.csv) - Spreadsheet format

Step 3: Analyze Data

import pandas as pd
Load data
df = pd.read_json('output.jl', lines=True)
Basic statistics
print(f"Total companies: {len(df)}")
print(f"Active companies: {len(df[df['status'] == 'Active'])}")
Top industries
print("\nTop 10 Industries:")
print(df['tags'].explode().value_counts().head(10))
Geographic distribution
print("\nTop 10 Locations:")
print(df['location'].value_counts().head(10))

Optional: Launch Web Explorer

After scraping, you can launch an interactive web-based explorer:

# Convert scraped data to JSON format for the web UI
python scripts/convertoutputto_json.py
Launch local web server
python -m http.server 3000

Then open http://localhost:3000 in your browser.

Web Explorer Features:

• 🔍 Real-time search and filtering by batch, tags, and keywords
• 📊 Sortable directory with ascending/descending options
• 👥 Full founder profiles with bios and social links
• 📥 Export filtered results as JSON
• ⚡ Lazy loading for smooth performance with thousands of companies

Data Schema

| Attribute | Description | Type | |-------------------|-----------------------------------|--------| | company_id | Company ID provided by YC | int | | company_name | Company name | string | | short_description | One-line description | string | | long_description | Full company description | string | | batch | YC batch (e.g., "W23", "S24") | string | | status | Active, Inactive, Public, etc. | string | | tags | Industry tags | list | | location | City | string | | country | Country code | string | | year_founded | Year founded | int | | num_founders | Number of founders | int | | founders_names | List of founder names | list | | founder_details | Extended bios and social links | list | | team_size | Number of employees | int | | website | Company website | string | | cb_url | Crunchbase URL | string | | linkedin_url | LinkedIn URL | string |

Example Output

| companyid | companyname | batch | status | location | yearfounded | teamsize | |------------|--------------|-------|--------|---------------|--------------|-----------| | 240 | Stripe | S09 | Active | San Francisco | 2010 | 7000 | | 271 | Airbnb | W09 | Public | San Francisco | 2008 | 6132 | | 325 | Dropbox | S07 | Public | San Francisco | 2008 | 4000 | | 439 | Coinbase | S12 | Public | San Francisco | 2012 | 6112 | | 531 | DoorDash | S13 | Public | San Francisco | 2013 | 8600 |

Troubleshooting

Issue: WebDriverException: 'geckodriver' executable needs to be in PATH

Solution:

brew install geckodriver  # macOS geckodriver --version     # Verify installation

Issue: FileNotFoundError: Start URLs file not found

Solution: Run python yclinksextractor.py first to generate the URLs file.

Issue: Scraper gets stuck or times out

Solution:

• Check your internet connection
• Verify Firefox isn't already running
• Ensure YC website is accessible: curl https://www.ycombinator.com/companies

Debug Mode

Enable verbose logging:

LOGLEVEL=DEBUG python yclinksextractor.py

Data Analysis Examples

Companies by Batch

import pandas as pd
import matplotlib.pyplot as plt
df = pd.read_json('output.jl', lines=True) active = df[df['status'] == 'Active']
batchcounts = active['batch'].valuecounts().sort_index() batch_counts.plot(kind='bar', figsize=(15, 5)) plt.title('Active YC Companies by Batch') plt.show()

Industry Trends

# Most common industries
tags = df['tags'].explode()
top15 = tags.valuecounts().head(15)
print(top_15)

Geographic Analysis

# Companies by country
countries = df['country'].value_counts()
print(countries.head(10))

Dataset

For a pre-scraped dataset, check out Y Combinator Directory on Kaggle.

License

MIT License - see LICENSE for details.

Contributors

Original Author: Miguel Corral Jr.

• Email: corraljrmiguel@gmail.com
• LinkedIn: linkedin.com/in/imiguel
• GitHub: github.com/corralm

• LinkedIn: linkedin.com/in/tario-you
• Contributions: Interactive web UI, enhanced founder profiles

Last Updated: October 2025