Terraform DigitalOcean HA K3S Module

Features

• [x] High Availability K3s Cluster provisioned on the DigitalOcean platform
• [x] Managed PostgreSQL/MySQL database provisioned. Serves as the datastore for the cluster's state (configurable options: size & node count)
• [x] Dedicated VPC provisioned for cluster use (IP Range: 10.10.10.0/24)
• [x] Number of provisioned Servers (Masters) and Agents (Workers) is configurable
• [x] Cluster API/Server(s) are behind a provisioned load balancer for high availability
• [x] All resources assigned to a dedicated DigitalOcean project (expect Load Balancers provisioned by app deployments)
• [x] Flannel backend is configurable. Choose from vxlan (default), ipsec or wireguard
• [x] DigitalOcean's CCM (Cloud Controller Manager) and CSI (Container Storage Interface) plugins are pre-installed. Enables the cluster to leverage DigitalOcean's load balancer and volume resources
• [x] Option to make Servers (Masters) schedulable. Default is false i.e. CriticalAddonsOnly=true:NoExecute
• [x] Cluster database engine is configurable. Choose between PostgreSQL (v11, default) or MySQL (v8)
• [x] Deploy System Upgrade Controller to manage automated upgrades of the cluster [default: false]
• [x] Deploy the Kubernetes Dashboard [default: false]
• [x] Deploy Jetstack's cert-manager [default: false]
• [x] Firewalled Nodes & Database
• [x] Deploy an ingress controller from Kong (Postgres or DB-less mode), Nginx or Traefik v2 [default: none]
• [ ] Generate custom kubeconfig file (optional)

Compatibility/Requirements

• Requires Terraform 0.15 or higher.
• A DigitalOcean account and personal access token for accessing the DigitalOcean API - Use this referral link for $100 free credit

Tutorial

Deploy a HA K3s Cluster on DigitalOcean in 10 minutes using Terraform

Architecture

A default deployment of this module provisions an architecture similar to that illustrated below (minus the external traffic Load Balancer). 2x Servers, 1x Agent and a load balancer in front of the servers providing a fixed registration address for the Kubernetes API. Additional Servers or Agents can be provisioned via the servercount and agentcount variables respectively.

K3s Architecture with a High-availability Servers - Source

Usage

module "do-ha-k3s" {
  source = "github.com/aigisuk/terraform-digitalocean-ha-k3s"
do_token                  = "7f5ef8eb151e3c81cd893c6...."   sshkeyfingerprints      = ["00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff"] }

cluster_summary = {   "agents" = [     {       "id" = "246685594"       "ip_private" = "10.10.10.4"       "ip_public" = "203.0.113.10"       "name" = "k3s-agent-fra1-1a9f-1"       "price" = 10     },   ]   "apiserverip" = "198.51.100.10"   "cluster_region" = "fra1"   "servers" = [     {       "id" = "246685751"       "ip_private" = "10.10.10.5"       "ip_public" = "203.0.113.11"       "name" = "k3s-server-fra1-55b4-1"       "price" = 10     },     {       "id" = "246685808"       "ip_private" = "10.10.10.6"       "ip_public" = "203.0.113.12"       "name" = "k3s-server-fra1-d6e7-2"       "price" = 10     },   ] }

To manage K3s from outside the cluster, SSH into any Server node and copy the contents of /etc/rancher/k3s/k3s.yaml to ~/.kube/config on an external machine with kubectl installed.

sudo scp -i .ssh/yourprivatekey root@203.0.113.11:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Then replace 127.0.0.1 with the API Load Balancer IP address of your K3s Cluster (value for the apiserverip key from the Terraform cluster_summary output).

sudo sed -i -e "s/127.0.0.1/198.51.100.10/g" ~/.kube/config

Functional examples are included in the examples directory.

Inputs

| Name | Description | Type | Default | Required | |------|-------------|:----:|:-----:|:-----:| | do_token | DigitalOcean Personal Access Token | string | N/A | yes | | sshkeyfingerprints | List of SSH Key fingerprints | list(string) | N/A | yes | | region | Region in which to deploy cluster | string | fra1 | no | | vpcnetworkrange | Range of IP addresses for the VPC in CIDR notation | string | 10.10.10.0/24 | no | | k3s_channel | K3s release channel. stable, latest, testing or a specific channel or version e.g. v1.22, v1.19.8+k3s1 | string | "stable" | no | | databaseuser | Database username | string | "k3sdefault_user" | no | | database_engine | Database engine. postgres (v13) or mysql (v8) | string | "postgres" | no | | database_size | Database Droplet size associated with the cluster e.g. db-s-1vcpu-1gb | string |"db-s-1vcpu-1gb" | no | | databasenodecount | Number of nodes that comprise the database cluster | number | 1| no | | flannel_backend | Flannel Backend Type. Valid options include vxlan, ipsec or wireguard-native | string | vxlan| no | | server_size | Server droplet size. e.g. s-1vcpu-2gb | string | s-1vcpu-2gb| no | | agent_size | Agent droplet size. e.g. s-1vcpu-2gb | string | s-1vcpu-2gb| no | | server_count | Number of server (master) nodes to provision | number | 2| no | | agent_count | Number of agent (worker) nodes to provision | number | 1| no | | servertaintcriticalonly | Allow only critical addons to be scheduled on server nodes? (thus preventing workloads from being launched on them) | bool | true| no | | sysupgradectrl | Deploy the System Upgrade Controller | bool | false| no | | ingress | Deploy an ingress controller. none, traefik, kong, kong_pg | string | "none"| no | | k8sdashboard | Deploy Kubernetes Dashboard | bool | false| no | | k8sdashboardversion | Kubernetes Dashboard version | string | 2.7.0| no | | certmanager | Deploy cert-manager | bool | false| no | | certmanagerversion | cert-manager version | string | 1.11.0| no | | traefikversion | traefik version | string | 2.9.7| no |

kubectl -n kubernetes-dashboard describe secret admin-user-token | awk '$1=="token:"{print $2}'

eyJhbGciOiJSUzI1NiI....JmL-nP-x1SPjOCNfZkg

kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 8080:443

https://localhost:8080

kubectl port-forward -n traefik $(kubectl get pods -n traefik --selector=app=traefik --output=name) 9000:9000

http://localhost:9000/dashboard/