Arche

pip install arche

Arche (pronounced Arkey) helps to verify scraped data using set of defined rules, for example: * Validation with JSON schema * Coverage (items, fields, categorical data, including booleans and enums) * Duplicates * Garbage symbols * Comparison of two jobs We use it in Scrapinghub, among the other tools, to ensure quality of scraped data

Installation

Arche requires Jupyter environment, supporting both JupyterLab and Notebook UI

For JupyterLab, you will need to properly install plotly extensions

Then just pip install arche

Why

To check the quality of scraped data continuously. For example, if you scraped a website, a typical approach would be to validate the data with Arche. You can also create a schema and then set up Spidermon

Developer Setup

pipenv install --dev pipenv shell tox

Contribution

Any contributions are welcome! See https://github.com/scrapinghub/arche/issues if you want to take on something or suggest an improvement/report a bug.