Feedsearch Crawler

feedsearch-crawler is a Python library for discovering RSS, Atom), and JSON feeds on websites.

About

This is a library package designed to be integrated into other Python applications. It provides a simple API for feed discovery that can be embedded into web scrapers, content aggregators, RSS readers, or API services.

It is a continuation of my work on Feedsearch, which is itself a continuation of the work done by Dan Foreman-Mackey on Feedfinder2, which in turn is based on feedfinder - originally written by Mark Pilgrim) and subsequently maintained by Aaron Swartz until his untimely death.

Feedsearch Crawler differs from previous versions in that it is now built as an asynchronous Web crawler using asyncio and aiohttp, allowing much more rapid scanning of potential feed URLs.

Real-World Usage

An implementation using this library to provide a public Feed Search API is available at

Pull requests and suggestions are welcome.

Installation

The library is available on PyPI:

pip install feedsearch-crawler

Requirements:

• Python 3.12 or higher
• No additional system dependencies

Usage

Feedsearch Crawler is called with the single function `search:

If you are already running in an asyncio event loop, then you can import and await searchasync instead. The search function is only a wrapper that runs search_async in a new asyncio event loop.

A search will always return a list of FeedInfo objects, each of which will always have a url property, which is a URL object that can be decoded to a string with str(url). The returned FeedInfo are sorted by the score value from highest to lowest, with a higher score theoretically indicating a more relevant feed compared to the original URL provided. A FeedInfo can also be serialized to a JSON compatible dictionary by calling it's .serialize() method.

If you need detailed error information when a URL fails to load (DNS errors, SSL errors, HTTP errors, timeouts, etc.), use searchwithinfo or searchasyncwith_info instead. These functions return a SearchResult object that includes error details:

You can also retrieve crawl statistics by passing include_stats=True:

The SearchResult object is iterable, so you can iterate over feeds directly:

Note: The original search() and searchasync() functions return an empty list when errors occur. This behavior is maintained for backward compatibility. Use searchwith_info() when you need to distinguish between "no feeds found" and "URL failed to load".

search and search_async` take the following arguments:

python
search(
    url: Union[URL, str, List[Union[URL, str]]],
    crawl_hosts: bool=True,
    try_urls: Union[List[str], bool]=False,
    concurrency: int=10,
    total_timeout: Union[float, aiohttp.ClientTimeout]=10,
    request_timeout: Union[float, aiohttp.ClientTimeout]=3,
    user_agent: str="Feedsearch Bot",
    maxcontentlength: int=1024  1024  10,
    max_depth: int=10,
    headers: dict={"X-Custom-Header": "Custom Header"},
    favicondatauri: bool=True,
    delay: float=0
)

• url: Union[str, List[str]]: The initial URL or list of URLs at which to search for feeds. You may also provide URL objects.
• crawl_hosts: bool: (default True): An optional argument to add the site host origin URL to the list of initial crawl URLs. (e.g. add "example.com" if crawling "example.com/path/rss.xml"). If False, site metadata and favicon data may not be found.
• tryurls: Union[List[str], bool]: (default False): An optional list of URL paths to query for feeds. Takes the origins of the url parameter and appends the provided paths. If no list is provided, but tryurls is True, then a list of common feed locations will be used.
• concurrency: int: (default 10): An optional argument to specify the maximum number of concurrent HTTP requests.
• total_timeout: float: (default 30.0): An optional argument to specify the time this function may run before timing out.
• request_timeout: float: (default 3.0): An optional argument that controls how long before each individual HTTP request times out.
• user_agent: str: An optional argument to override the default User-Agent header.
• maxcontentlength: int: (default 10Mb): An optional argument to specify the maximum size in bytes of each HTTP Response.
• max_depth: int: (default 10): An optional argument to limit the maximum depth of requests while following urls.
• headers: dict: An optional dictionary of headers to pass to each HTTP request.
• favicondatauri: bool: (default True): Optionally control whether to fetch found favicons and return them as a Data Uri.
• delay: float: (default 0.0): An optional argument to delay each HTTP request by the specified time in seconds. Used in conjunction with the concurrency setting to avoid overloading sites.

FeedInfo Values

In addition to the url, FeedInfo objects may have the following values:

• bozo: int: Set to 1 when feed data is not well formed or may not be a feed. Defaults 0.
• content_length: int: Current length of the feed in bytes.
• contenttype: str: Content-Type value of the returned feed.
• description: str: Feed description.
• favicon: URL: URL of feed or site Favicon.
• favicondatauri: str: Data Uri of Favicon.
• hubs: List[str]: List of Websub hubs of feed if available.
• ispodcast: bool: True if the feed contains valid podcast elements and enclosures.
• is_push: bool: True if feed contains valid Websub data.
• item_count: int: Number of items currently in the feed.
• last_updated: datetime: Date of the latest published entry.
• score: int: Computed relevance of feed url value to provided URL. May be safely ignored.
• self_url: URL: ref="self" value returned from feed links. In some cases may be different from feed url.
• site_name: str: Name of feed's website.
• siteurl: URL: URL of feed's website.
• title: str: Feed Title.
• url: URL: URL location of feed.
• velocity: float: Mean number of items per day in the feed at the current time.
• version: str: Feed version XML values,

uv sync
uv run ruff check
uv run ruff format
uv run pytest

# Use default URLs from file
uv run main.py
Crawl single URL
uv run main.py https://example.com
Crawl single URL with domain only
uv run main.py example.com
Crawl multiple URLs
uv run main.py https://site1.com https://site2.com
Use comma-separated format
uv run main.py --urls "https://site1.com,https://site2.com"
Get help
uv run main.py --help