Jenkins X EKS Module

This repository contains a Terraform module for creating an EKS cluster and all the necessary infrastructure to install Jenkins X as described in https://jenkins-x.io/v3/admin/platforms/eks/.

• Jenkins X EKS Module

What is a Terraform module

A Terraform module refers to a self-contained package of Terraform configurations that are managed as a group. For more information about modules refer to the Terraform documentation.

How do you use this module

Prerequisites

This Terraform module allows you to create an EKS cluster ready for the installation of Jenkins X. You need the following binaries locally installed and configured on your PATH:

• terraform (>= 1.0.0, < 2.0.0)
• kubectl (>= 1.10)
• aws-cli
• helm (>= 3.0)

Cluster provisioning

From version 3.0.0 this module creates neither the EKS cluster nor the VPC.

We recommend using the Terraform modules terraform-aws-modules/eks/aws to create the cluster and terraform-aws-modules/vpc/aws to create the VPC.

A Jenkins X ready cluster can be provisioned using the configuration in jx3-terraform-eks as described in https://jenkins-x.io/v3/admin/platforms/eks/.

All s3 buckets created by the module use Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3) by default. You can set the value of usekmss3 to true to use server-side encryption with AWS KMS (SSE-KMS). If you don't specify the value of s3kmsarn, then the default aws managed cmk is used (aws/s3)

:warning: Note: Using AWS KMS with customer managed keys has cost considerations.

You should have your AWS CLI configured correctly.

AWS_REGION

In addition, you should make sure to specify the region via the AWS_REGION environment variable. e.g. export AWS_REGION=us-east-1 and the region variable (make sure the region variable matches the environment variable)

The IAM user does not need any permissions attached to it.

Once you have your initial configuration, you can apply it by running:

terraform init
terraform apply

This creates an EKS cluster with all possible configuration options defaulted.

:warning: Note: This example is for getting up and running quickly. It is not intended for a production cluster. Refer to Production cluster considerations for things to consider when creating a production cluster.

Migrating to current version of module from a version prior to 3.0.0

If you already have created an EKS cluster using a pre 3.0.0 version of this module there is unfortunately no easy way to upgrade without recreating the cluster. If you already create the cluster in some other way and now set create_eks = false you only n eed to remove some inputs. I won't cover that much simpler case here.

While it would be a bit easier if you started using the same version of terraform-aws-modules/eks/aws as previously used by this module we would advise against that. The reason is that this version is very old and doesn't support a lot of feature currently available with AWS.

Let's say you created your cluster using an old version of the template and change your configuration to a current version. If you then run terraform plan you will see that basically everything would be destroyed and then created. To mitigate that you can move resources in the terraform state to the new addresses. In some cases there are no corresponding new address, instead you are better off removing resources to avoid that they get destroyed before the new resources are created. This means that you need to remove those cloud resources manually later. You can also tweak configurations to prevent resources from be replaced. If you check the output from terraform plan you will see that resources marked as "must be replaced" have one or more inputs with the comment "# forces replacement". If it is a resource that you need to keep to prevent disruption or data loss you should try to tweak the configuration so that the inputs value is reverted to what it was before.

terraform state mv module.eks-jx.randompet.current  randompet.current # Only needed if cluster_name wasn't specified
terraform state mv module.eks-jx.module.cluster.module.vpc module.vpc # Only needed if create_vpc wasn't false
terraform state mv module.eks-jx.module.cluster.module.eks module.eks
terraform state mv 'module.eks.awsiamrole.cluster[0]' 'module.eks.awsiamrole.this[0]'
If the following two commands fail it is because you are migrating from a version of this module that didn't 
create these resource. That is not a problem, but if you have installed the add on in some other way you will 
need to issue some other terraform command: either "terraform state mv" command or "terragrunt import"
terraform state mv 'module.eks-jx.module.cluster.awseksaddon.ebsaddon[0]' 'module.eks.awseks_addon.this["aws-ebs-csi-driver"]'
terraform state mv module.eks-jx.module.cluster.module.ebscsiirsarole module.ebscsiirsarole
Removing the following resources from the state prevent terraform apply from destroying existing node groups and 
related resources before new ones are created. But this means that ypu need to delete the resources manually later.  
terraform state rm module.eks.module.node_groups 
terraform state rm 'module.eks.awsiamrolepolicyattachment.workersAmazonEC2ContainerRegistryReadOnly[0]' 'module.eks.awsiamrolepolicyattachment.workersAmazonEKSWorkerNodePolicy[0]' 'module.eks.awsiamrolepolicyattachment.workersAmazonEKSCNI_Policy[0]'
terraform state rm 'module.eks.awssecuritygroup.workers[0]'  'module.eks.awsiamrole.workers[0]'
terraform state rm $(terraform state list | grep awssecuritygroup_rule.workers)
terraform state rm $(terraform state list | grep awssecuritygroup_rule.cluster)

In main.tf some tweaks are needed. Add the following inputs to the module eks

prefix_separator                   = ""   iamrolename                      = local.cluster_name   clustersecuritygroupname        = local.clustername   clustersecuritygroup_description = "EKS cluster security group."

Cluster add ons

If you already create cluster addons with terraform you can either remove the corresponding addon from the cluster_addons input of the eks module or use terraform state mv to change the address in the state file and thus prevent destroying and creating the add-on.

aws-auth config map

If you have configured the config map aws-auth by setting any of the inputs mapaccounts, maproles or map_users you will need to either configure aws-auth ins some other way, see https://registry.terraform. io/modules/terraform-aws-modules/eks/aws/20.20.0/submodules/aws-auth or switch to using access entries. See the documentation for the input accessentries in terraform-aws-modules/eks/aws and the AWS documentation.

terraform state rm 'module.eks.kubernetesconfigmap.aws_auth[0]'

jx gitops helmfile add --chart  autoscaler/cluster-autoscaler --repository https://kubernetes.github.io/autoscaler  --namespace kube-system

- chart: autoscaler/cluster-autoscaler
  values:
  - image:
      tag: v1.30.0

You can choose to create S3 buckets for long term storage of Jenkins X build artefacts with enablelogsstorage, enablereportsstorage and enablerepository_storage.

During terraform apply the enabled S3 buckets are created, and the jxrequirements_ output will contain the following section:

storage:
%{ if enablelogsstorage }
  - name: logs
    url: s3://${logsstoragebucket}
%{ endif }
%{ if enablereportsstorage }
  - name: reports
    url: s3://${reportsstoragebucket}}
%{ endif }
%{ if enablerepositorystorage }
  - name: repository
    url: s3://${repositorystoragebucket}
%{ endif }

If you just want to experiment with Jenkins X, you can set the variable forcedestroy_ to true. This allows you to remove all generated buckets when running terraform destroy.

:warning: Note: If you set force_destroy to false, and run a terraform destroy, it will fail. In that case empty the s3 buckets from the aws s3 console, and re run terraform destroy.

:warning: Note: A notice from Amazon: Amazon S3 will automatically enable S3 Block Public Access and disable access control lists for all new buckets starting in April 2023. To accomodate this acl setting was removed for buckets and the enableacl variable was introduced and set to false (default). If the requirement is to provide ACL with bucket ownership conrols for the bucket, then set the enable_acl variable to true.

Secrets Management

Vault is the default tool used by Jenkins X for managing secrets. Part of this module's responsibilities is the installation of Vault Operator which in turn install vault.

You can also configure an existing Vault instance for use with Jenkins X. In this case

• provide the Vault URL via the vaulturl_ input variable
• set the boot_secrets in main.tf to this value:

boot_secrets = [
    {
      name  = "jxBootJobEnvVarSecrets.EXTERNAL_VAULT"
      value = "true"
      type  = "string"
    },
    {
      name  = "jxBootJobEnvVarSecrets.VAULT_ADDR"
      value = "https://enter-your-vault-url:8200"
      type  = "string"
    }
  ]

• follow the Jenkins X documentation around the installation of an external Vault instance.

NGINX

The module can install the nginx chart by setting create_nginx flag to true. Example can be found here. You can specify a nginxvalues.yaml file or the module will use the default one stored here. If you are using terraform to create nginx resources, do not use the chart specified in the versionstream. Remove the entry in the helmfile.yaml referencing the nginx chart

path: helmfiles/nginx/helmfile.yaml

You can enable ExternalDNS with the enableexternaldns variable. This modifies the generated jx-requirements.yml_ file to enable External DNS when running jx boot.

If enableexternaldns is true, additional configuration is required.

If you want to use a domain with an already existing Route 53 Hosted Zone, you can provide it through the apex_domain variable:

This domain will be configured in the jxrequirements_ output in the following section:

ingress:
  domain: ${domain}
  ignoreLoadBalancer: true
  externalDNS: ${enableexternaldns}

If you want to use a subdomain and have this module create and configure a new Hosted Zone with DNS delegation, you can provide the following variables:

subdomain: This subdomain is added to the apex domain and configured in the resulting jx-requirements.yml file.

createandconfigure_subdomain: This flag instructs the script to create a new Route53 Hosted Zone for your subdomain and configure DNS delegation with the apex domain.

By providing these variables, the script creates a new Route 53 HostedZone that looks like <subdomain>.<apex_domain>, then it delegates the resolving of DNS to the apex domain. This is done by creating a NS RecordSet in the apex domain's Hosted Zone with the subdomain's HostedZone nameservers.

This ensures that the newly created HostedZone for the subdomain is instantly resolvable instead of having to wait for DNS propagation.

cert-manager

You can enable cert-manager to use TLS for your cluster through LetsEncrypt with the enabletls variable.

LetsEncrypt has two environments, staging and production.

If you use staging, you will receive self-signed certificates, but you are not rate-limited, if you use the production environment, you receive certificates signed by LetsEncrypt, but you can be rate limited.

You can choose to use the production environment with the production_letsencrypt variable:

You need to provide a valid email to register your domain in LetsEncrypt with tls_email.

Customer's CA certificates

Customer has got signed certificates from CA and want to use it instead of LetsEncrypt certificates. Terraform creates k8s tls-ingress-certificates-ca secret with tlskey and tlscert in default namespace. User should define:

enableexternaldns = true
apex_domain         = "office.com"
subdomain           = "subdomain"
enable_tls          = true
tls_email           = "custome@office.com"
// Signed Certificate must match the domain: *.subdomain.office.com tls_cert            = "/opt/CA/cert.crt" tls_key             = "LS0tLS1C....BLRVktLS0tLQo="

Production cluster considerations

The configuration, as seen in Cluster provisioning, is not suited for creating and maintaining a production Jenkins X cluster. The following is a list of considerations for a production use case.

• Specify the version attribute of the module, for example:

module "eks-jx" {
    source  = "github.com/jenkins-x/terraform-aws-eks-jx"
    version = "1.0.0"
    # insert your configuration
  }
output "jx_requirements" {     value = module.eks-jx.jx_requirements   }

Specifying the version ensures that you are using a fixed version and that version upgrades cannot occur unintended.

• Keep the Terraform configuration under version control by creating a dedicated repository for your cluster configuration or by adding it to an already existing infrastructure repository.

• Setup a Terraform backend to securely store and share the state of your cluster. For more information refer to Configuring a Terraform backend.

• Disable public API for the EKS cluster.

Configuring a Terraform backend

A "backend" in Terraform determines how state is loaded and how an operation such as apply_ is executed. By default, Terraform uses the local backend, which keeps the state of the created resources on the local file system. This is problematic since sensitive information will be stored on disk and it is not possible to share state across a team. When working with AWS a good choice for your Terraform backend is the s3_ backend which stores the Terraform state in an AWS S3 bucket. The examples directory of this repository contains configuration examples for using the s3_ backed.

To use the s3 backend, you will need to create the bucket upfront. You need the S3 bucket as well as a Dynamo table for state locks. You can use terraform-aws-tfstate-backend to create these required resources.

Examples

You can find examples for different configurations in the examples folder.

Each example generates a valid jx-requirements.yml file that can be used to boot a Jenkins X cluster.

Module configuration

Providers

| Name | Version | |------|---------| | aws | 5.60.0 |

Modules

| Name | Source | Version | |------|--------|---------| | cluster | ./modules/cluster | n/a | | dns | ./modules/dns | n/a | | health | ./modules/health | n/a | | nginx | ./modules/nginx | n/a | | vault | ./modules/vault | n/a |

Requirements

| Name | Version | |------|---------| | terraform | >= 0.13.0, < 2.0.0 | | aws | > 4.0 | | helm | ~> 2.0 | | kubernetes | ~> 2.0 | | null | ~> 3.0 |

Inputs

| Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| | additional\tekton\role\policy\arns | Additional Policy ARNs to attach to Tekton IRSA Role | list(string) | [] | no | | apex\_domain | The main domain to either use directly or to configure a subdomain from | string | "" | no | | asm\role | DEPRECATED: Use the new bot\iam\_role input with he same semantics instead. | string | "" | no | | boot\iam\role | Specify arn of the role to apply to the boot job service account | string | "" | no | | boot\_secrets | n/a |

list(object({
    name  = string
    value = string
    type  = string
  }))

Outputs

| Name | Description | |------|-------------| | cert\manager\iam\_role | The IAM Role that the Cert Manager pod will assume to authenticate | | cluster\asm\iam\_role | The IAM Role that the External Secrets pod will assume to authenticate (Secrets Manager) | | cluster\autoscaler\iam\_role | The IAM Role that the Jenkins X UI pod will assume to authenticate | | cluster\_name | The name of the created cluster | | cluster\ssm\iam\_role | The IAM Role that the External Secrets pod will assume to authenticate (Parameter Store) | | cm\cainjector\iam\_role | The IAM Role that the CM CA Injector pod will assume to authenticate | | connect | The cluster connection string to use once Terraform apply finishes. You may have to provide the region and
profile (as options or environment variables) | | controllerbuild\iam\role | The IAM Role that the ControllerBuild pod will assume to authenticate | | external\dns\iam\_role | The IAM Role that the External DNS pod will assume to authenticate | | jx\_requirements | The jx-requirements rendered output | | lts\logs\bucket | The bucket where logs from builds will be stored | | lts\reports\bucket | The bucket where test reports will be stored | | lts\repository\bucket | The bucket that will serve as artifacts repository | | pipeline\viz\iam\_role | The IAM Role that the pipeline visualizer pod will assume to authenticate | | subdomain\_nameservers | ---------------------------------------------------------------------------- DNS ---------------------------------------------------------------------------- | | tekton\bot\iam\_role | The IAM Role that the build pods will assume to authenticate |

FAQ: Frequently Asked Questions

IAM Roles for Service Accounts

This module sets up a series of IAM Policies and Roles. These roles will be annotated into a few Kubernetes Service accounts. This allows us to make use of IAM Roles for Sercive Accounts to set fine-grained permissions on a pod per pod basis. There is no way to provide your own roles or define other Service Accounts by variables, but you can always modify the modules/cluster/irsa.tf Terraform file.

How can I contribute

Contributions are very welcome! Check out the Contribution Guidelines for instructions.