Topics

#gpu #jupyter #kubeflow #kubernetes #nvidia #odh

Language breakdown

Python 69.5%

Shell 15.4%

Jupyter Notebook 8.5%

TypeScript 1.8%

JavaScript 1.8%

Makefile 1.7%

▸ Files click to expand

README

OpenDataHub Notebooks

GitHub License

Welcome to the OpenDataHub Notebooks repository! This repository provides a collection of notebooks tailored for data analysis, machine learning, research and coding within the OpenDataHub ecosystem. Designed to streamline data science workflows, these notebooks offer an integrated environment equipped with the latest tools and libraries. These notebooks were created to be used with OpenDataHub ecosystem with the ODH Notebook Controller as the launcher.

These workbenches are available at: quay.io/repository/opendatahub/workbench-images

Getting Started

For a deeper understanding of the architecture underlying this repository, please explore our wiki page Wiki

Prerequisites

Make sure the following tools are installed in your environment: - podman/docker - python 3.14 - uv - make (on macOS: brew install make and add /opt/homebrew/opt/make/libexec/gnubin to PATH) - curl - git, git-lfs (for *.vsix files)

Installation

Clone this repository to your local machine:

git clone https://github.com/opendatahub-io/notebooks.git
cd notebooks

Quick Start Guide

Build a Notebook

To build a workbench image, you can execute the following command:

make ${WORKBENCHNAME} -e  IMAGEREGISTRY=quay.io/${YOUR_USER}/workbench-images  -e  RELEASE=2023x

Using IMAGE_REGISTRY and RELEASE variables you can overwrite the default values and use a different registry or release tag

Using CONTAINERBUILDCACHEARGS (default: --no-cache), and PUSHIMAGES variables you can further customize the build process.

Hermetic prefetch (`prefetch-input/`)

Jupyter images that build hermetically (Konflux and local builds with Cachi2) share repo-root prefetch-input/: RPM lock inputs (odh/, rhds/), repos/, and the mongocli submodule. Each hermetic image directory under jupyter/*/ubi9-python-3.12/ contains a symlink prefetch-input → ../../../prefetch-input. Code-server keeps its own codeserver/ubi9-python-3.12/prefetch-input/ (different dependency set). To prefetch dependencies locally, see scripts/lockfile-generators/README.md.

Local Execution

The notebook can be run as container on the local systems.

Use podman/docker to execute the workbench images as container.

podman  run -it -p  8888:8888  quay.io/opendatahub/workbench-images:jupyter-minimal-ubi9-python-3.9-2024a-20240317-6f4c36b

Lock file generation

Image lock files (pylock.toml / uv.lock.d/) are regenerated with make refresh-lock-files. For details on the dual uv version policy (dev vs image locks), see the Deploy & Test section below.

Update base image versions

Use the root versions_config.yml file as the operator entry point for base image updates.

The sync flow validates that config, rewrites managed build-args/*.conf BASE_IMAGE and RELEASE values, updates root Makefile RELEASE defaults, and resolves RHDS channel: fast tags from published Quay tags with skopeo.

For the full configuration contract, RHDS and ODH resolution rules, Mermaid flow diagram, and worked examples, see docs/baseimageversionsupdate_configuration.md.

Quick commands:

make sync-build-args-from-versions
make sync-build-args-from-versions SYNCBUILDARGS_ARGS=--dry-run
make sync-build-args-from-versions SYNCBUILDARGS_ARGS=--check

If the version update also requires lockfile refreshes, run make refresh-lock-files after the sync.

Deploy & Test

Prepare Python + uv + pytest env

Root dev tooling uses [tool.uv] required-version in pyproject.toml (currently >=0.10,<0.12). Install any uv in that range and use it directly for sync, tests, and pre-commit:

# Linux
sudo dnf install python3.14
pip install --user 'uv>=0.10,<0.12'
macOS
brew install python@3.14 uv
uv venv --python $(which python3.14) uv sync --locked

Per-image lock files (pylock.toml / uv.lock.d/) are generated with a stricter uv pin (dependencies/uv-image-lock-version, currently 0.11.18) via the ./uv wrapper and make refresh-lock-files. Do not use ./uv for everyday dev commands unless you are refreshing image locks.

Image lock generation with ./uv

make refresh-lock-files
or: ./uv run scripts/pylocks_generator.py public-index <dir>

The ./uv script runs uv tool run "uv@$(cat dependencies/uv-image-lock-version)" when your system uv does not already match that exact version.

Running tests

For the full test catalog (types, markers, CI parity), see docs/agents/testing.md.

By completing configuration in the previous section, you can run tests using the following targets:

make test              # Quick static tests (pytest + Dockerfile alignment check)
make test-unit         # Python unit tests + doctests + Go tests (no container runtime needed)
make test-integration PYTEST_ARGS="--image=<image>"  # Container integration tests

Container selftests

We're using Testcontainers.com to run containers from Python tests.

Ideally, these tests should allow switching between Kubernetes and Docker/Podman. Running on Kubernetes (OpenShift) is the easiest way to manage GPU access for testing TensorFlow/Pytorch images. This improvement is tracked as a future work.

We also considered Dagger.io. It has very nice verbose logging by default for every action that is running. The main difference between the two is that Dagger creates more abstractions over the container engine. Dagger limitations that stem from tight BuildKit integration discouraged us from using it. Images present in a local store cannot be directly used in Dagger, they have to be imported into its BuildKit container first. In addition, Dagger does not allow bind-mounting local directories to the container but always copies files in and out.

Running testcontainers tests in Pytest

# Podman/Docker config
Linux
sudo dnf install podman
systemctl --user start podman.service
systemctl --user status podman.service
systemctl --user status podman.socket
DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock uv run pytest tests/containers -m 'not openshift and not cuda and not rocm' --image quay.io/opendatahub/workbench-images@sha256:e98d19df346e7abb1fa3053f6d41f0d1fa9bab39e49b4cb90b510ca33452c2e4
Mac OS
brew install podman
podman machine init
podman machine set --rootful=false
sudo podman-mac-helper install
podman machine start
uv run pytest tests/containers -m 'not openshift' --image quay.io/opendatahub/workbench-images@sha256:e98d19df346e7abb1fa3053f6d41f0d1fa9bab39e49b4cb90b510ca33452c2e4

When using lima on macOS, it might be useful to give yourself access to rootful podman socket

lima sudo systemctl start podman
lima sudo dnf install acl
lima sudo setfacl -m u:${USER}:x /var/run/podman
lima sudo setfacl -m u:${USER}:rw /var/run/podman/podman.sock

Running Playwright tests

tests/browser/README.md

Notebooks

Deploy the notebook images in your Kubernetes environment using: deploy8-${NOTEBOOKNAME} for ubi8 or deploy9-${NOTEBOOKNAME} for ubi9

make  deployX-${NOTEBOOK_NAME}

Run the test suite against this notebook:

make  test-${NOTEBOOK_NAME}

You can overwrite NOTEBOOKREPOBRANCH_BASE variable to use a different repository and branch for testing scripts. This is useful when you debug your changes.

make  test-${NOTEBOOKNAME} -e  NOTEBOOKREPOBRANCHBASE="https://raw.githubusercontent.com/${YOURUSER}/notebooks/${YOURBRANCH}"

Clean up the environment when the tests are finished:

make  undeployX-${NOTEBOOK_NAME}

Runtimes

The runtimes image requires to have curl and python installed, so that on runtime additional packages can be installed.

Deploy the runtime images in your Kubernetes environment using: deploy8-${WORKBENCHNAME} for ubi8 or deploy9-${WORKBENCHNAME} for ubi9

make  deployX-${WORKBENCH_NAME}

Run the validate test suit for checking compatabilty of runtime images:

make  validate-runtime-image  image=<runtime-image>

Clean up the environment when the tests are finished:

make  undeployX-${WORKBENCH_NAME}

Image Inventory List

This table provides a concise overview of the support status for various container images.

| Workbench | Version | Python | OS | x86_64 | aarch64 | ppc64le | s390x | Container | |------------|----------|--------|------|---------|----------|----------|--------|-------------| | CPU Minimal | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-workbench-jupyter-minimal-cpu-py312-ubi9 | | CPU DataScience | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-workbench-jupyter-datascience-cpu-py312-ubi9 | | CPU TrustyAI | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-workbench-jupyter-trustyai-cpu-py312-ubi9 | | CPU CodeServer | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-workbench-codeserver-datascience-cpu-py312-ubi9 | | CUDA Minimal | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ✅ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-minimal-cuda-py312-ubi9 | | CUDA PyTorch | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ⏰ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-pytorch-cuda-py312-ubi9 | | CUDA PyTorch LLMCompressor | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ⏰ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-pytorch-llmcompressor-cuda-py312-ubi9 | | CUDA TensorFlow | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ✅ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-tensorflow-cuda-py312-ubi9 | | ROCM Minimal | 6.3 | 3.12 | UBI9/RHEL9.6 | ✅ | ❌ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-minimal-rocm-py312-ubi9 | | ROCM PyTorch | 6.3 | 3.12 | UBI9/RHEL9.6 | ✅ | ❌ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-pytorch-rocm-py312-ubi9 | | ROCM TensorFlow | 6.3 | 3.12 | UBI9/RHEL9.6 | ✅ | ❌ | ❌ | ❌ | quay.io/repository/opendatahub/odh-workbench-jupyter-tensorflow-rocm-py312-ubi9 |

| Runtime | Version | Python | UBI | x86_64 | aarch64 | ppc64le | s390x | Container | |----------|----------|--------|------|---------|----------|----------|--------|-------------| | CPU Minimal | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-pipeline-runtime-minimal-cpu-py312-ubi9 | | CPU DataScience | - | 3.12 | UBI9 | ✅ | ✅ | ✅ | ✅ | quay.io/repository/opendatahub/odh-pipeline-runtime-datascience-cpu-py312-ubi9 | | CUDA PyTorch | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ⏰ | ❌ | ❌ | quay.io/repository/opendatahub/odh-pipeline-runtime-pytorch-cuda-py312-ubi9 | | CUDA PyTorch LLMCompressor | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ⏰ | ❌ | ❌ | quay.io/repository/opendatahub/odh-pipeline-runtime-pytorch-llmcompressor-cuda-py312-ubi9 | | CUDA TensorFlow | 12.8 | 3.12 | UBI9/RHEL9.6 | ✅ | ⏰ | ❌ | ❌ | quay.io/repository/opendatahub/odh-pipeline-runtime-tensorflow-cuda-py312-ubi9 | | ROCM PyTorch | 6.3 | 3.12 | UBI9/RHEL9.6 | ✅ | ❌ | ❌ | ❌ | quay.io/repository/opendatahub/odh-pipeline-runtime-pytorch-rocm-py312-ubi9 | | ROCM TensorFlow | 6.3 | 3.12 | UBI9/RHEL9.6 | ✅ | ❌ | ❌ | ❌ | quay.io/repository/opendatahub/odh-pipeline-runtime-tensorflow-rocm-py312-ubi9 |

✅ supported
⚠️ under development
⏰ planned in the future
❌ not supported by vendor

Contributing

Whether you're fixing bugs, adding new notebooks, or improving documentation, your contributions are welcome. Please refer to our Contribution Guidelines.

For AI Agents

If you're an AI agent working with this repository, please refer to our Agents Guide for comprehensive instructions on project structure, development workflows, and best practices.