A modern template for data science projects with all the necessary tools for experiment, development, testing, and deployment. From notebooks to production.
Data science project template
A modern template for data science projects with all the necessary tools for experiment, development, testing, and deployment. From notebooks to production.
โจ๐โจ Documentation:
Source Code:
Features
- Dependency management with [UV]
- Virtual environment management with [UV]
- Linting with [pre-commit] and [Ruff]
- Continuous integration with [GitHub Actions]
- Documentation with [mkdocs] and [mkdocstrings] using the mkdocs-materialtheme
- Automated dependency updates with [Dependabot]
- Code formatting with [Ruff]
- Import sorting with [Ruff] using isort rule.
- Testing with [pytest]
- Code coverage with [Coverage.py]
- Coverage reporting with [Codecov]
- Static type-checking with [mypy]
- Security audit with [Ruff] using bandit rule.
- Manage project labels with [GitHub Labeler]
Table of Contents - Features - ๐ Creating a New Project - ๐ Recommendations - ๐ช๐ฅ Via Cruft - (recommended) - ๐ช Via Cookiecutter - ๐ Linking an Existing Project - ๐๏ธ Project structure - โจ Features and Tools - ๐ Project Standardization and Automation - ๐จ Developer Workflow Automation - ๐ฑ Conditionally Rendered Python Package or Project Boilerplate - ๐ง Maintainability - ๐ท๏ธ Type Checking and Data Validation - โ ๐งช Testing/Coverage - ๐จ Linting - ๐ Code quality - ๐จ Code formatting - ๐ท CI/CD - Automatic Dependency updates - Dependency Review in PR - Pre-commit automatic updates - ๐ Security - ๐ Static Application Security Testing (SAST) - โจ๏ธ Accessibility - ๐จ Automation tool (Makefile) - ๐ Project Documentation - ๐๏ธ Templates - Good practices - References
๐ Creating a New Project
๐ Recommendations
It is highly recommended to use managers for the python versions, dependencies and virtual environments.
This project uses [UV], a extremely fast tool to replace pip, pip-tools, [Pipx], [Poetry], [Pyenv], twine, virtualenv, and more.
๐ Check how to setup your environment:
๐ช๐ฅ Via [Cruft] - (recommended)
title="install cruft"
Install cruft in a isolated environment using uv
uv tool install cruft
Or Install with pip
pip install --user cruft # Install cruft on your path for easy access
title="create project"
cruft create https://github.com/JoseRZapata/data-science-project-templatethen inside the project folder, init git and uv environment using [Make]:
title="install project"
make init_git
make install_env
source .venv/bin/activate๐ช Via [Cookiecutter]
title="install cookiecutter"
uv tool install cookiecutter # Install cruft in a isolated environment
Or Install with pip
pip install --user cookiecutter # Install cookiecutter on your path for easy access
title="create project"
cookiecutter gh:JoseRZapata/data-science-project-templateNote: Cookiecutter uses gh: as short-hand for https://github.com/
๐ Linking an Existing Project
If the project was originally installed via [Cookiecutter], you must first use [Cruft] to link the project with the original template:
cruft link https://github.com/JoseRZapata/data-science-project-templateThen/else:
cruft update๐๏ธ Project structure
Folder structure for data science projects why?
.
โโโ .code_quality
โย ย โโโ mypy.ini # mypy configuration
โย ย โโโ ruff.toml # ruff configuration
โโโ .github # github configuration
โย ย โโโ actions
โย ย โย ย โโโ python-poetry-env
โย ย โย ย โโโ action.yml # github action to setup python environment
โย ย โโโ dependabot.md # github action to update dependencies
โย ย โโโ pullrequesttemplate.md # template for pull requests
โย ย โโโ workflows # github actions workflows
โย ย โโโ ci.yml # run continuous integration (tests, pre-commit, etc.)
โย ย โโโ dependency_review.yml # review dependencies
โย ย โโโ docs.yml # build documentation (mkdocs)
โย ย โโโ pre-commit_autoupdate.yml # update pre-commit hooks
โโโ .vscode # vscode configuration
| โโโ extensions.json # list of recommended extensions
| โโโ launch.json # vscode launch configuration
| โโโ settings.json # vscode settings
โโโ conf # folder configuration files
โย ย โโโ config.yaml # main configuration file
โโโ data
โย ย โโโ 01_raw # raw immutable data
โย ย โโโ 02_intermediate # typed data
โย ย โโโ 03_primary # domain model data
โย ย โโโ 04_feature # model features
โย ย โโโ 05modelinput # often called 'master tables'
โย ย โโโ 06_models # serialized models
โย ย โโโ 07modeloutput # data generated by model runs
โย ย โโโ 08_reporting # reports, results, etc
โย ย โโโ README.md # description of the data structure
โโโ docs # documentation for your project
โย ย โโโ index.md # documentation homepage
โโโ models # store final models
โโโ notebooks
โย ย โโโ 1-data # data extraction and cleaning
โย ย โโโ 2-exploration # exploratory data analysis (EDA)
โย ย โโโ 3-analysis # Statistical analysis, hypothesis testing.
โย ย โโโ 4-feat_eng # feature engineering (creation, selection, and transformation.)
โย ย โโโ 5-models # model training, evaluation and hyperparameter tuning.
โย ย โโโ 6-interpretation # model interpretation
โย ย โโโ 7-deploy # model packaging, deployment strategies.
โย ย โโโ 8-reports # story telling, summaries and analysis conclusions.
โย ย โโโ notebook_template.ipynb # template for notebooks
โย ย โโโ README.md # information about the notebooks
โโโ src # source code for use in this project
โ โโโ README.md # description of src structure
โ โโโ tmp_mock.py # example python file
โ โโโ data # data extraction, validation, processing, transformation
โ โโโ model # model training, evaluation, validation, export
โ โโโ inference # model prediction, serving, monitoring
โ โโโ pipelines # orchestration of pipelines
โ โโโ feature_pipeline # transforms raw data into features and labels
โ โโโ training_pipeline # transforms features and labels into a model
โ โโโ inference_pipeline # takes features and a trained model for predictions
โโโ tests # test code for your project
โ โโโ test_mock.py # example test file
โ โโโ data # tests for data module
โ โโโ model # tests for model module
โ โโโ inference # tests for inference module
โ โโโ pipelines # tests for pipelines module
โโโ .editorconfig # editor configuration
โโโ .gitignore # files to ignore in git
โโโ .pre-commit-config.yaml # configuration for pre-commit hooks
โโโ codecov.yml # configuration for codecov
โโโ Makefile # useful commands to setup environment, run tests, etc.
โโโ mkdocs.yml # configuration for mkdocs documentation
โโโ pyproject.toml # dependencies and configuration project file
โโโ uv.lock # locked dependencies
โโโ README.md # description of your projectโจ Features and Tools
๐ Project Standardization and Automation
๐จ Developer Workflow Automation
- Python packaging, dependency management and environment management
why use a management, (uv is a replacement for poetry)
- Project workflow orchestration
make on the command line to display auto-generated documentation on available
targets:
- Automated Cookiecutter template synchronization with [Cruft] -
why? - Code quality tooling automation and management with [pre-commit]
- Continuous integration and deployment with [GitHub Actions]
- Project configuration files with [Hydra] -
why?
๐ฑ Conditionally Rendered Python Package or Project Boilerplate
- Optional: [Jupyter] support
๐ง Maintainability
๐ท๏ธ Type Checking and Data Validation
- Static type-checking with [Mypy]
โ ๐งช Testing/Coverage
- Testing with [Pytest]
- Code coverage with [Coverage.py]
- Coverage reporting with [Codecov]
๐จ Linting
๐ Code quality
- [Ruff] An extremely fast (10x-100x faster) Python linter and code formatter, written in Rust.
- ShellCheck
- Unsanitary commits:
detect-secrets
- Large files with check-added-large-files
- Files that contain merge conflict strings.check-merge-conflict
๐จ Code formatting
- [Ruff] An extremely fast (10x-100x faster) Python linter and code formatter, written in Rust.
- General file formatting:
end-of-file-fixer
- pretty-format-json
- (trim) trailing-whitespace
- check-yaml
๐ท CI/CD
Automatic Dependency updates
- Dependency updates with [Dependabot], Automated [Dependabot] PR merging with the Dependabot Auto Merge GitHub Action
- This is a replacement for pip-audit , In your local environment, If you want to check for vulnerabilities in your dependencies you can use [pip-audit]_.
Dependency Review in PR
- Dependency Review with [dependency-review-action], This action scans your pull requests for dependency changes, and will raise an error if any vulnerabilities or invalid licenses are being introduced.
Pre-commit automatic updates
- Automatic updates with [GitHub Actions] workflow
.github/workflows/pre-commit_autoupdate.yml
๐ Security
๐ Static Application Security Testing (SAST)
- Code vulnerabilities with [Bandit] using [Ruff]
โจ๏ธ Accessibility
๐จ Automation tool (Makefile)
Makefile to automate the setup of your environment, the installation of dependencies, the execution of tests, etc. in terminal type make to see the available commands
Target Description
------------------- ----------------------------------------------------
check Run code quality tools with pre-commit hooks.
docs_test Test if documentation can be built without warnings or errors
docs_view Build and serve the documentation
init_env Install dependencies with uv and activate env
init_git Initialize git repository
installdatalibs Install pandas, scikit-learn, Jupyter, seaborn
pre-commit_update Update pre-commit hooks
test Test the code with pytest and coverage๐ Project Documentation
- Documentation building
๐๏ธ Templates
- [Pull Request template]
- [Notebook template]
Good practices
References
[Bandit]: https://github.com/PyCQA/bandit [Black]: https://github.com/psf/black [Codecov]: https://codecov.io/ [Cookiecutter]:https://cookiecutter.readthedocs.io/en/stable/ [Coverage.py]: https://coverage.readthedocs.io/ [Cruft]: https://cruft.github.io/cruft/ [Dependabot]: https://github.com/dependabot/dependabot-core [dependency-review-action]: https://github.com/actions/dependency-review-action [Flake8]:https://github.com/PyCQA/flake8 [GitHub Actions]: https://github.com/features/actions [Github Labeler]: https://github.com/marketplace/actions/github-labeler [hydra]: https://hydra.cc/ [isort]: https://github.com/PyCQA/isort [Jupyter]: https://jupyter.org/ [Make]: https://www.gnu.org/software/make/manual/make.html [mkdocs]: https://www.mkdocs.org/ [mkdocstrings]: https://mkdocstrings.github.io/ [Mypy]: http://mypy-lang.org/ [Notebook template]: https://github.com/JoseRZapata/data-science-project-template/blob/main/{{cookiecutter.reponame}}/notebooks/notebooktemplate.ipynb [Pipx]:https://pipx.pypa.io/stable/ [Poetry]: https://python-poetry.org/ [pre-commit]: https://pre-commit.com/ [Pull Request template]: https://github.com/JoseRZapata/data-science-project-template/blob/main/{{cookiecutter.reponame}}/.github/pullrequest_template.md [Pyenv]:https://github.com/pyenv/pyenv [Pylint]:https://github.com/PyCQA/pylint [Pytest]: https://docs.pytest.org/en/latest/ [pyupgrade]: https://github.com/asottile/pyupgrade [Ruff]: https://docs.astral.sh/ruff/ [UV]: https://docs.astral.sh/uv/
