This repo covers Kubeflow Environment with LABs: Kubeflow GUI, Jupyter Notebooks on pods, Kubeflow Pipelines, Experiments, KALE, KATIB (AutoML: Hyperparameter Tuning), KFServe (Model Serving), Training Operators (Distributed Training), Projects, etc.
Last updated Jun 16, 2026
97
Stars
25
Forks
0
Issues
0
Stars/day
Attention Score
27
Topics
Language breakdown
Python 93.6%
Dockerfile 6.4%
▸ Files
click to expand
README
Fast-Kubeflow
This repo covers Kubeflow Environment with LABs: Kubeflow GUI, Jupyter Notebooks running on Kubernetes Pod, Kubeflow Pipeline, KALE (Kubeflow Automated PipeLines Engine), KATIB (AutoML: Finding Best Hyperparameter Values), KFServe (Model Serving), Training Operators (Distributed Training), Projects, etc. Possible usage scenarios are aimed to update over time.Kubeflow is powerful tool that runs on Kubernetes (K8s) with containers (process isolation, scaling, distributed and parallel training). Kubeflow can be installed on-premise (WSL2 or MiniKF), and Cloud (AWS, Azure, GCP; ref: https://www.kubeflow.org/docs/started/installing-kubeflow/)
This repo makes easy to learn and apply projects on your local machine with MiniKF, Virtualbox 6.1.40 and Vagrant without any FEE (Min: 16GB RAM, 6 CPU cores, 70-80 GB Disk space).
Prerequisite
- Have a knowledge of
Quick Look (HowTo): Scenarios - Hands-on LABs
- LAB: Creating LAB Environment (WSL2), Installing Kubeflow with MicroK8s, Juju on Ubuntu 20.04
- LAB: Creating LAB Environment, Installing MiniKF with Vagrant (Preffered for Easy Usage)
- LAB/Project: Kubeflow Pipeline (From Scratch) with Kubeflow SDK (DSL Compiler) and Custom Docker Images (Decision Tree, Logistic Regression, SVM, Naive Bayes, Xg Boost)
- LAB/Project: KALE (Kubeflow Automated PipeLines Engine) and KATIB (AutoML: Finding Best Hyperparameter Values)
- LAB/Project: KALE (Kubeflow Automated PipeLines Engine) and KServe (Model Serving) for Model Prediction
- LAB/Project: Distributed Training with Tensorflow (MNIST data)
Table of Contents
- Motivation
- What is Kubelow?
- How Kubeflow Works?
- What is Container (Docker)?
- What is Kubernetes?
- Installing Kubeflow
- Kubeflow Basics
- Kubeflow Jupyter Notebook
- Kubeflow Pipeline
- KALE (Kubeflow Automated PipeLines Engine)
- KATIB (AutoML: Finding Best Hyperparameter Values)
- KServe (Model Serving)
- Training-Operators (Distributed Training)
- Minio (Object Storage) and ROK (Data Management Platform)
- Project 1: Creating ML Pipeline with Custom Docker Images (Decision Tree, Logistic Regression, SVM, Naive Bayes, Xg Boost)
- Project 2: KALE (Kubeflow Automated PipeLines Engine) and KATIB (AutoML: Finding Best Hyperparameter Values)
- Project 3: KALE (Kubeflow Automated PipeLines Engine) and KServe (Model Serving) for Model Prediction
- Project 4: Distributed Training with Training Operator
- Other Useful Resources Related Kubeflow
- References
Motivation
Why should we use / learn Kubeflow?
- Kubeflow uses containers on Kubernetes to run steps of Machine Learning and Deep Learning algorithms on the computer clusters.
- Kubeflow provides Machine Learning (ML) data pipeline.
- It saves pipelines, experiments, runs (experiment tracking on Kubeflow), models (model deployment).
- It provides easy, repeatable, portable deployments on a diverse infrastructure (for example, experimenting on a laptop, then moving to an on-premises cluster or to the cloud).
- Kubeflow provides deploying and managing loosely-coupled microservices and scaling based on demand.
- Kubeflow is free, open source platform that runs on on-premise or any cloud (AWS, Google Cloud, Azure, etc.).
- It includes Jupyter Notebook to develop ML algorithms, user interface to show pipeline.
- "Kubeflow started as an open sourcing of the way Google ran TensorFlow internally, based on a pipeline called TensorFlow Extended. It began as just a simpler way to run TensorFlow jobs on Kubernetes, but has since expanded to be a multi-architecture, multi-cloud framework for running entire machine learning pipelines." (ref: kubeflow.org)
- Kubeflow applies to become a CNCF incubating project, it is announced on 24 October 2022 (ref: opensource.googleblog.com).
- Distributed and Parallel training become more important day by day, because the number of the parameters is increasing (especially deep learning models: billions to trillion parameters). Increasing parameter provides better results but it also causes the longer training and it needs more computing power. With Kubeflow, Kubernetes and containers, distributed learning is achieved with many GPUs. Please have look Training-Operators (Distributed Training) part for details.
- CERN uses Kubeflow and Training operators to speed up the training (3D-GAN) on parallel multiple GPUs (1 single training time: From 2.5 days = 60 hours to 30 minutes, video/presentation: https://www.youtube.com/watch?v=HuWt1N8NFzU)
What is Kubelow
- "The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable." (ref: kubeflow.org)
- "Kubeflow has developed into an end-to-end, extendable ML platform, with multiple distinct components to address specific stages of the ML lifecycle: model development (Kubeflow Notebooks), model training (Kubeflow Pipelines and Kubeflow Training Operator), model serving (KServe), and automated machine learning (Katib)" (ref: opensource.googleblog.com).
- Kubeflow is a type of ML data pipeline application that provides to create ML data pipeline (saving model and artifacts, running multiple times) like Airflow
How Kubeflow Works?
- Kubeflow works on Kubernetes platform with Docker Containers.
- Kubernetes creates the node clusters with many servers and PCs. Kubeflow is a distributed application (~35 pods) running on the Kubernetes platform. Kubeflow pods are running on the different nodes if there are several nodes connected to the Kubernetes cluster.
- Containers include Python Machine learning (ML) codes that are each step of the ML pipeline (e.g. Dowloading data function, decision tree classifier, linear regression classifier, evaluation part, etc.)
- Containers' outputs can be able to connect to the other containers' inputs. With this feature, it is possible to create DAG (Directed Acyclic Graph) with containers. Each function can be able to run on the seperate containers.
(ref: kubeflow-pipelines towardsdatascience)
- If you want to learn the details of the working of Kubeflow, you should learn:
What is Container (Docker)?
- Docker is a tool that reduces the gap between Development/Deployment phase of a software development cycle.
- Docker is like VM but it has more features than VMs (no kernel, only small app and file systems, portable)
- Without Docker containers, each VM consumes 30% resources (Memory, CPU)
(Ref: Docker.com)
(Ref: docs.docker.com)
- To learn about Docker and Containers, please go to this repo: Fast-Docker
What is Kubernetes?
- "Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available." (Ref: Kubernetes.io)
(Ref: Kubernetes.io)
(Ref: Kubernetes.io)
- To learn about Kubernetes, please go to this repo: https://github.com/omerbsezer/Fast-Kubernetes
Installing Kubeflow
- How to install Kubeflow on WSL2 with Juju:
- To get more features like KALE, and to install in easy way: Use Kubeflow with MiniKF below (preferred)
- Kubeflow with MiniKF: How to install MiniKF with Vagrant and VirtualBox:
Kubeflow Basics
- Kubeflow is an ML distributed application that contains following parts:
Kubeflow Jupyter Notebook
- Kubeflow creates Notebook using containers and K8s pod.
- When user wants to run new notebook, user can configure:
- After launching notebook pod, it creates pod and we can connect it to open the notebook.
- After creating notebook pod, in MiniKF, it triggers to create volume automatically (with ROK storage class), user can reach files and even downloads the files.
Kubeflow Pipeline
- Kubeflow Pipelines is based on Argo Workflows which is a container-native workflow engine for kubernetes.
- Kubeflow Pipelines consists of (ref: Kubeflow-Book):
- Have a look it:
KALE (Kubeflow Automated PipeLines Engine)
- KALE (Kubeflow Automated pipeLines Engine) is a project that aims at simplifying the Data Science experience of deploying Kubeflow Pipelines workflows.
- Kale bridges this gap by providing a simple UI to define Kubeflow Pipelines workflows directly from you JupyterLab interface, without the need to change a single line of code (ref: https://github.com/kubeflow-kale/kale).
- With KALE, each cells are tagged and worklow can be created by connecting cells, then after compiling, Kubeflow Pipeline is created and run.
- KALE feature helps data scientist to run on Kubeflow quickly without creating any container manually.
(ref: KALE Tags)
- Have a look to KALE and KATIB Project:
KATIB (AutoML: Finding Best Hyperparameter Values)
- Katib is a Kubernetes-native project for automated machine learning (AutoML). Katib supports Hyperparameter Tuning, Early Stopping and Neural Architecture Search.
- Katib has search methods (ref: https://github.com/kubeflow/katib):
- Have a look it:
KFServe (Model Serving)
- KServe enables serverless inferencing on Kubernetes and provides performant, high abstraction interfaces for common machine learning (ML) frameworks like TensorFlow, XGBoost, scikit-learn, PyTorch, and ONNX to solve production model serving use cases (ref: https://github.com/kserve/kserve).
- Have a look it:
Training-Operators (Distributed Training)
- It is great advantage to run distributed and parallel jobs (training) on Kubernetes with Training-Operators. User can determine the number of worker trainer pods.
- Training operator provides Kubernetes custom resources that makes it easy to run distributed or non-distributed TensorFlow / PyTorch / Apache MXNet / XGBoost / MPI jobs on Kubernetes (ref: https://github.com/kubeflow/training-operator).
- Distributed Training become more important day by day, because the number of the parameters is increasing (especially deep learning, deep neural networks). Increasing parameter provides better results but it also causes the longer training and it needs more computing power.
- CERN uses Kubeflow and Training operators to speed up the training (3D-GAN) on parallel multiple GPUs (1 single training time: From 2.5 days = 60 hours to 30 minutes):
- Have a look it:
Minio (Object Storage) and ROK (Data Management Platform)
- Minio is object storage (like AWS S3, Azure Blob Storage), but it also works on-premise, Kubeflow uses minio to save Kubeflow object data. To get more info and printscreen of the Minio, please have a look below:
- ROK is data management platform on Kubernetes that is developed by Arrikto. In the MiniKF, ROK makes easy to use K8s data management (e.g. automatically managed PV, storage classes)
Project 1: Creating ML Pipeline with Custom Docker Images (Decision Tree, Logistic Regression, SVM, Naive Bayes, Xg Boost)
- Have a look it:
Project 2: KALE (Kubeflow Automated PipeLines Engine) and KATIB (AutoML: Finding Best Hyperparameter Values)
- Have a look it:
Project 3: KALE (Kubeflow Automated PipeLines Engine) and KServe (Model Serving) for Model Prediction
- Have a look it:
Project 4: Distributed Training with Tensorflow (MNIST data)
- Have a look it:
Other Useful Resources Related Kubeflow
- https://www.kubeflow.org/
- https://www.kubeflow.org/docs/components/central-dash/overview/
- https://github.com/kubeflow/
References
- kubeflow.org: (kubeflow documentation) https://v0-7.kubeflow.org/docs/
- opensource.googleblog.com: https://opensource.googleblog.com/2022/10/kubeflow-applies-to-become-a-cncf-incubating-project.html
- kubeflow-pipelines towardsdatascience: https://towardsdatascience.com/kubeflow-pipelines-how-to-build-your-first-kubeflow-pipeline-from-scratch-2424227f7e5
- Kubernetes.io: https://kubernetes.io/docs/concepts/overview/
- docs.docker.com: https://docs.docker.com/get-started/overview/
- Argo Worflow: https://github.com/argoproj/argo-workflows
- Kubeflow-Book: https://www.amazon.com.mx/Kubeflow-Machine-Learning-Lab-Production/dp/1492050121
- KALE: https://github.com/kubeflow-kale/kale
- KATIB: https://github.com/kubeflow/katib,
- KALE Tags: https://medium.com/kubeflow/automating-jupyter-notebook-deployments-to-kubeflow-pipelines-with-kale-a4ede38bea1f
- KServe: https://github.com/kserve/kserve
- https://www.technologyreview.com/2021/12/21/1042835/2021-was-the-year-of-monster-ai-models/
- https://www.technologyreview.com/2021/12/08/1041557/deepmind-language-model-beat-others-25-times-size-gpt-3-megatron/
- https://indico.cern.ch/event/924283/contributions/4105328/attachments/2153724/3632143/2020-12-01-Kubeflow-FastML.pdf
- CERN Distributed Training Video: https://www.youtube.com/watch?v=HuWt1N8NFzU
