• CKS Simulator Kubernetes 1.25

* Pre Setup * Question 1 | Contexts * Question 2 | Runtime Security with Falco * Question 3 | Apiserver Security * Question 4 | Pod Security Standard * Question 5 | CIS Benchmark * Question 6 | Verify Platform Binaries * Question 7 | Open Policy Agent * Question 8 | Secure Kubernetes Dashboard * Question 9 | AppArmor Profile * Question 10 | Container Runtime Sandbox gVisor * Question 11 | Secrets in ETCD * Question 12 | Hack Secrets * Question 13 | Restrict access to Metadata Server * Question 14 | Syscall Activity * Question 15 | Configure TLS on Ingress * Question 16 | Docker Image Attack Surface * Question 17 | Audit Log Policy * Question 18 | Investigate Break-in via Audit Log * Question 19 | Immutable Root FileSystem * Question 20 | Update Kubernetes * Question 21 | Image Vulnerability Scanning * Question 22 | Manual Static Security Analysis

• CKS Simulator Preview Kubernetes 1.25

* Preview Question 1 - Answer: * Part 1 - check existing RBAC rules * Part 2 - create additional RBAC rules * Preview Question 2 - Answer: * Preview Question 3 - Answer:

• CKS Tips Kubernetes 1.25

* Knowledge + Pre-Knowledge + Knowledge + Approach + Content

• CKS Exam Info

* Read the Curriculum * Read the Handbook * Read the important tips * Read the FAQ

• Kubernetes documentation
• CKS clusters
• The Test Environment / Browser Terminal

* Laggin * Kubectl autocompletion and commands * Copy & Paste * Percentages and Score * Notepad & Skipping Questions * Contexts

• PSI Bridge
• Browser Terminal Setup

* Minimal Setup + Alias + Vim * Optional Setup + Fast dry-run output + Fast pod delete + Persist bash settings + Alias Namespace * Be fast * Vim + toggle vim line numbers + copy&paste + Indent multiple lines * Split terminal screen

CKS Simulator Kubernetes 1.25

https://killer.sh

Pre Setup

Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:

alias k=kubectl                         # will already be pre-configured
export do="--dry-run=client -o yaml"    # k create deploy nginx --image=nginx $do
export now="--force --grace-period 0"   # k delete pod x $now

Vim The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

set tabstop=2
set expandtab
set shiftwidth=2

More setup suggestions are in the tips section.

Question 1 | Contexts

Task weight: 1%

You have access to multiple clusters from your main terminal through kubectl contexts. Write all context names into /opt/course/1/contexts, one per line.

From the kubeconfig extract the certificate of user restricted@infra-prod and write it decoded to /opt/course/1/cert.

Answer:

Maybe the fastest way is just to run:

k config get-contexts # copy by hand
k config get-contexts -o name > /opt/course/1/contexts

Or using jsonpath:

k config view -o js
k config view -o js | tr " " "\n" # new lines
k config view -o js | tr " " "\n" > /opt/course/1/contexts

The content could then look like:

#/opt/course/1/contexts
gianna@infra-prod
infra-prod
restricted@infra-prod
workload-prod
workload-stage

For the certificate we could just run

k config view --raw

And copy it manually. Or we do:

k config view --raw -ojs | base64 -d > /opt/course/1/cert

Or even:

k config view --raw -ojs | base64 -d > /opt/course/1/cert
/opt/course/1/cert
-----BEGIN CERTIFICATE-----
MIIDHzCCAgegAwIBAgIQN5Qe/Rj/PhaqckEI23LPnjANBgkqhkiG9w0BAQsFADAV
MRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTIwMDkyNjIwNTUwNFoXDTIxMDkyNjIw
NTUwNFowKjETMBEGA1UEChMKcmVzdHJpY3RlZDETMBEGA1UEAxMKcmVzdHJpY3Rl
ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL/Jaf/QQdijyJTWIDij
qa5p4oAh+xDBX3jR9R0G5DkmPU/FgXjxej3rTwHJbuxg7qjTuqQbf9Fb2AHcVtwH
gUjC12ODUDE+nVtap+hCe8OLHZwH7BGFWWscgInZOZW2IATK/YdqyQL5OKpQpFkx
iAknVZmPa2DTZ8FoyRESboFSTZj6y+JVA7ot0pM09jnxswstal9GZLeqioqfFGY6
YBO/Dg4DDsbKhqfUwJVT6Ur3ELsktZIMTRS5By4Xz18798eBiFAHvgJGq1TTwuPM
EhBfwYwgYbalL8DSHeFrelLBKgciwUKjr1lolnnuc1vhkX1peV1J3xrf6o2KkyMc
lY0CAwEAAaNWMFQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMC
MAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUPrspZIWR7YMN8vT5DF3s/LvpxPQw
DQYJKoZIhvcNAQELBQADggEBAIDq0Zt77gXI1s+uW46zBw4mIWgAlBLl2QqCuwmV
kd86eH5bD0FCtWlb6vGdcKPdFccHh8Z6z2LjjLu6UoiGUdIJaALhbYNJiXXi/7cf
M7sqNOxpxQ5X5hyvOBYD1W7d/EzPHV/lcbXPUDYFHNqBYs842LWSTlPQioDpupXp
FFUQPxsenNXDa4TbmaRvnK2jka0yXcqdiXuIteZZovp/IgNkfmx2Ld4/Q+Xlnscf
CFtWbjRa/0W/3EW/ghQ7xtC7bgcOHJesoiTZPCZ+dfKuUfH6d1qxgj6Jwt0HtyEf
QTQSc66BdMLnw5DMObs4lXDo2YE6LvMrySdXm/S7img5YzU=
-----END CERTIFICATE-----

Question 2 | Runtime Security with Falco

Task weight: 4%

Use context: kubectl config use-context workload-prod

Falco is installed with default configuration on node cluster1-node1. Connect using ssh cluster1-node1. Use it to:

Find a Pod running image nginx which creates unwanted package management processes inside its container. Find a Pod running image httpd which modifies /etc/passwd. Save the Falco logs for case 1 under /opt/course/2/falco.log in format:

time-with-nanosconds,container-id,container-name,user-name

No other information should be in any line. Collect the logs for at least 30 seconds.

Afterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.

Answer: Falco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine.

NOTE: Other tools you might have to be familar with are sysdig or tracee

Use Falco as service

First we can investigate Falco config a little:

➜ ssh cluster1-node1
➜ root@cluster1-node1:~# service falco status ● falco.service - LSB: Falco syscall activity monitoring agent    Loaded: loaded (/etc/init.d/falco; generated)    Active: active (running) since Sat 2020-10-10 06:36:15 UTC; 2h 1min ago ...
➜ root@cluster1-node1:~# cd /etc/falco
➜ root@cluster1-node1:/etc/falco# ls
falco.yaml  falcorules.local.yaml  falcorules.yaml  k8sauditrules.yaml  rules.available  rules.d

This is the default configuration, if we look into falco.yaml we can see:

# /etc/falco/falco.yaml
... 
Where security notifications should go.
 
Multiple outputs can be enabled.

syslog_output:   enabled: true ...

This means that Falco is writing into syslog, hence we can do:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco
Sep 15 08:44:04 ubuntu2004 falco: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Sep 15 08:44:04 ubuntu2004 falco: Falco initialized with configuration file /etc/falco/falco.yaml
Sep 15 08:44:04 ubuntu2004 falco: Loading rules from file /etc/falco/falco_rules.yaml:
...

Yep, quite some action going on in there. Let's investigate the first offending Pod:

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep nginx | grep process
Sep 16 06:23:47 ubuntu2004 falco: 06:23:47.376241377: Error Package management process launched in container (user=root userloginuid=-1 command=apk containerid=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)
...
➜ root@cluster1-node1:~# crictl ps -id 7a5ea6a080d1 CONTAINER ID        IMAGE              NAME        ...         POD ID 7a5ea6a080d1b       6f715d38cfe0e      nginx       ...         7a864406b9794
root@cluster1-node1:~# crictl pods -id 7a864406b9794 POD ID              ...          NAME                             NAMESPACE        ... 7a864406b9794       ...          webapi-6cfddcd6f4-ftxg4          team-blue        ...

First Pod is webapi-6cfddcd6f4-ftxg4 in Namespace team-blue.

➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep httpd | grep passwd
Sep 16 06:23:48 ubuntu2004 falco: 06:23:48.830962378: Error File below /etc opened for writing (user=root userloginuid=-1 command=sed -i $d /etc/passwd parent=sh pcmdline=sh -c echo hacker >> /etc/passwd; sed -i '$d' /etc/passwd; true file=/etc/passwdngFmAl program=sed gparent=<NA> ggparent=<NA> gggparent=<NA> containerid=b1339d5cc2de image=docker.io/library/httpd)
➜ root@cluster1-node1:~# crictl ps -id b1339d5cc2de CONTAINER ID        IMAGE              NAME        ...         POD ID b1339d5cc2dee       f6b40f9f8ad71      httpd       ...         595af943c3245
root@cluster1-node1:~# crictl pods -id 595af943c3245 POD ID              ...          NAME                             NAMESPACE          ... 595af943c3245       ...          rating-service-68cbdf7b7-v2p6g   team-purple        ...

Second Pod is rating-service-68cbdf7b7-v2p6g in Namespace team-purple.

Eliminate offending Pods

The logs from before should allow us to find and "eliminate" the offending Pods:

➜ k get pod -A | grep webapi
team-blue              webapi-6cfddcd6f4-ftxg4                      1/1     Running
➜ k -n team-blue scale deploy webapi --replicas 0 deployment.apps/webapi scaled
➜ k get pod -A | grep rating-service team-purple            rating-service-68cbdf7b7-v2p6g               1/1     Running
➜ k -n team-purple scale deploy rating-service --replicas 0 deployment.apps/rating-service scaled

Use Falco from command line

We can also use Falco directly from command line, but only if the service is disabled:

➜ root@cluster1-node1:~# service falco stop
➜ root@cluster1-node1:~# falco Thu Sep 16 06:33:11 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) Thu Sep 16 06:33:11 2021: Falco initialized with configuration file /etc/falco/falco.yaml Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.yaml: Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Thu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/k8sauditrules.yaml: Thu Sep 16 06:33:12 2021: Starting internal webserver, listening on port 8765 06:33:17.382603204: Error Package management process launched in container (user=root userloginuid=-1 command=apk containerid=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine) ...

We can see that rule files are loaded and logs printed afterwards.

Create logs in correct format

The task requires us to store logs for "unwanted package management processes" in format time,container-id,container-name,user-name. The output from falco shows entries for "Error Package management process launched" in a default format. Let's find the proper file that contains the rule and change it:

➜ root@cluster1-node1:~# cd /etc/falco/
➜ root@cluster1-node1:/etc/falco# grep -r "Package management process launched" . ./falcorules.yaml:    Package management process launched in container (user=%user.name userloginuid=%user.loginuid
➜ root@cluster1-node1:/etc/falco# cp falcorules.yaml falcorules.yaml_ori
➜ root@cluster1-node1:/etc/falco# vim falco_rules.yaml

Find the rule which looks like this:

# Container is supposed to be immutable. Package management should be done in building the image.
rule: Launch Package Management Process in Container
  desc: Package management process ran inside container
  condition: >
    spawned_process
    and container
    and user.name != "_apt"
    and packagemgmtprocs
    and not packagemgmtancestor_procs
    and not userknownpackagemanagerin_container
  output: >
    Package management process launched in container (user=%user.name user_loginuid=%user.loginuid
    command=%proc.cmdline containerid=%container.id containername=%container.name image=%container.image.repository:%container.image.tag)
  priority: ERROR
  tags: [process, mitre_persistence]

Should be changed into the required format:

# Container is supposed to be immutable. Package management should be done in building the image.
rule: Launch Package Management Process in Container
  desc: Package management process ran inside container
  condition: >
    spawned_process
    and container
    and user.name != "_apt"
    and packagemgmtprocs
    and not packagemgmtancestor_procs
    and not userknownpackagemanagerin_container
  output: >
    Package management process launched in container %evt.time,%container.id,%container.name,%user.name
  priority: ERROR
  tags: [process, mitre_persistence]

For all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.

Next we check the logs in our adjusted format:

➜ root@cluster1-node1:/etc/falco# falco | grep "Package management"
06:38:28.077150666: Error Package management process launched in container 06:38:28.077150666,090aad374a0a,nginx,root 06:38:33.058263010: Error Package management process launched in container 06:38:33.058263010,090aad374a0a,nginx,root 06:38:38.068693625: Error Package management process launched in container 06:38:38.068693625,090aad374a0a,nginx,root 06:38:43.066159360: Error Package management process launched in container 06:38:43.066159360,090aad374a0a,nginx,root 06:38:48.059792139: Error Package management process launched in container 06:38:48.059792139,090aad374a0a,nginx,root 06:38:53.063328933: Error Package management process launched in container 06:38:53.063328933,090aad374a0a,nginx,root

This looks much better. Copy&paste the output into file /opt/course/2/falco.log on your main terminal. The content should be cleaned like this:

# /opt/course/2/falco.log
06:38:28.077150666,090aad374a0a,nginx,root
06:38:33.058263010,090aad374a0a,nginx,root
06:38:38.068693625,090aad374a0a,nginx,root
06:38:43.066159360,090aad374a0a,nginx,root
06:38:48.059792139,090aad374a0a,nginx,root
06:38:53.063328933,090aad374a0a,nginx,root
06:38:58.070912841,090aad374a0a,nginx,root
06:39:03.069592140,090aad374a0a,nginx,root
06:39:08.064805371,090aad374a0a,nginx,root
06:39:13.078109098,090aad374a0a,nginx,root
06:39:18.065077287,090aad374a0a,nginx,root
06:39:23.061012151,090aad374a0a,nginx,root

For a few entries it should be fast to just clean it up manually. If there are larger amounts of entries we could do:

cat /opt/course/2/falco.log.dirty | cut -d" " -f 9 > /opt/course/2/falco.log

The tool cut will split input into fields using space as the delimiter (-d""). We then only select the 9th field using -f 9.

Local falco rules

There is also a file /etc/falco/falco_rules.local.yaml in which we can override existing default rules. This is a much cleaner solution for production. Choose the faster way for you in the exam if nothing is specified in the task.

Question 3 | Apiserver Security

Task weight: 3%

Use context: kubectl config use-context workload-prod

You received a list from the DevSecOps team which performed a security investigation of the k8s cluster1 (workload-prod). The list states the following about the apiserver setup:

• Accessible through a NodePort Service

Change the apiserver setup so that:

• Only accessible through a ClusterIP Service

Answer:

In order to modify the parameters for the apiserver, we first ssh into the master node and check which parameters the apiserver process is running with:

➜ ssh cluster1-controlplane1
➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver root     13534  8.6 18.1 1099208 370684 ?      Ssl  19:55   8:40 kube-apiserver --advertise-address=192.168.100.11 --allow-privileged=true --anonymous-auth=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubernetes-service-node-port=31000 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key- ...

We may notice the following argument:

--kubernetes-service-node-port=31000

We can also check the Service and see its of type NodePort:

➜ root@cluster1-controlplane1:~# kubectl get svc
NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kubernetes   NodePort   10.96.0.1    <none>        443:31000/TCP   5d2h

The apiserver runs as a static Pod, so we can edit the manifest. But before we do this we also create a copy in case we mess things up:

➜ root@cluster1-controlplane1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/3_kube-apiserver.yaml
➜ root@cluster1-controlplane1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml

We should remove the unsecure settings:

# /etc/kubernetes/manifests/kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.11:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.100.11
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
   - --kubernetes-service-node-port=31000   # delete or set to 0
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
...

Once the changes are made, give the apiserver some time to start up again. Check the apiserver's Pod status and the process parameters:

➜ root@cluster1-controlplane1:~# kubectl -n kube-system get pod | grep apiserver
kube-apiserver-cluster1-controlplane1            1/1     Running        0          38s
➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver | grep node-port The apiserver got restarted without the unsecure settings. However, the Service kubernetes will still be of type NodePort:
➜ root@cluster1-controlplane1:~# kubectl get svc NAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE kubernetes   NodePort   10.96.0.1    <none>        443:31000/TCP   5d3h We need to delete the Service for the changes to take effect:
➜ root@cluster1-controlplane1:~# kubectl delete svc kubernetes service "kubernetes" deleted

After a few seconds:

➜ root@cluster1-controlplane1:~# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6s

This should satisfy the DevSecOps team.

Question 4 | Pod Security Standard

Task weight: 8%

Use context: kubectl config use-context workload-prod

There is Deployment container-host-hacker in Namespace team-red which mounts /run/containerd as a hostPath volume on the Node where its running. This means that the Pod can access various data about other containers running on the same Node.

To prevent this configure Namespace team-red to enforce the baseline Pod Security Standard. Once completed, delete the Pod of the Deployment mentioned above.

Check the ReplicaSet events and write the event/log lines containing the reason why the Pod isn't recreated into /opt/course/4/logs.

Answer:

Making Namespaces use Pod Security Standards works via labels. We can simply edit it:

k edit ns team-red

Now we configure the requested label:

# kubectl edit namespace team-red
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: team-red
    pod-security.kubernetes.io/enforce: baseline # add
  name: team-red
...

This should already be enough for the default Pod Security Admission Controller to pick up on that change. Let's test it and delete the Pod to see if it'll be recreated or fails, it should fail!

➜ k -n team-red get pod
NAME                                    READY   STATUS    RESTARTS   AGE
container-host-hacker-dbf989777-wm8fc   1/1     Running   0          115s
➜ k -n team-red delete pod container-host-hacker-dbf989777-wm8fc  pod "container-host-hacker-dbf989777-wm8fc" deleted
➜ k -n team-red get pod No resources found in team-red namespace.

Usually the ReplicaSet of a Deployment would recreate the Pod if deleted, here we see this doesn't happen. Let's check why:

➜ k -n team-red get rs
NAME                              DESIRED   CURRENT   READY   AGE
container-host-hacker-dbf989777   1         0         0       5m25s
➜ k -n team-red describe rs container-host-hacker-dbf989777 Name:           container-host-hacker-dbf989777 Namespace:      team-red ... Events:   Type     Reason            Age                   From                   Message   ----     ------            ----                  ----                   ------- ...   Warning  FailedCreate      2m41s                 replicaset-controller  Error creating: pods "container-host-hacker-dbf989777-bjwgv" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata")   Warning  FailedCreate      2m2s (x9 over 2m40s)  replicaset-controller  (combined from similar events): Error creating: pods "container-host-hacker-dbf989777-kjfpn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata")

There we go! Finally we write the reason into the requested file so that Mr Scoring will be happy too!

# /opt/course/4/logs
Warning  FailedCreate      2m2s (x9 over 2m40s)  replicaset-controller  (combined from similar events): Error creating: pods "container-host-hacker-dbf989777-kjfpn" is forbidden: violates PodSecurity "baseline:latest": hostPath volumes (volume "containerdata")
Pod Security Standards can give a great base level of security! But when one finds themselves wanting to deeper adjust the levels like baseline or restricted... this isn't possible and 3rd party solutions like OPA could be looked at.

Question 5 | CIS Benchmark

Task weight: 3%

Use context: kubectl config use-context infra-prod

You're ask to evaluate specific settings of cluster2 against the CIS Benchmark recommendations. Use the tool kube-bench which is already installed on the nodes.

Connect using ssh cluster2-controlplane1 and ssh cluster2-node1.

On the master node ensure (correct if necessary) that the CIS recommendations are set for:

• The --profiling argument of the kube-controller-manager

• The ownership of directory /var/lib/etcd

On the worker node ensure (correct if necessary) that the CIS recommendations are set for:

• The permissions of the kubelet configuration /var/lib/kubelet/config.yaml
• The --client-ca-file argument of the kubelet

Answer:

Number 1

First we ssh into the master node run kube-bench against the master components:

➜ ssh cluster2-controlplane1
➜ root@cluster2-controlplane1:~# kube-bench run --targets=master ... == Summary == 41 checks PASS 13 checks FAIL 11 checks WARN 0 checks INFO

We see some passes, fails and warnings. Let's check the required task (1) of the controller manager:

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3
1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold,
for example:
--terminated-pod-gc-threshold=10
--
1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
on the master node and set the below parameter.
--profiling=false
1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. --feature-gates=RotateKubeletServerCertificate=true

There we see 1.3.2 which suggests to set --profiling=false, so we obey:

➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/manifests/kube-controller-manager.yaml

Edit the corresponding line:

# /etc/kubernetes/manifests/kube-controller-manager.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --port=0
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --use-service-account-credentials=true
    - --profiling=false            # add
...

We wait for the Pod to restart, then run kube-bench again to check if the problem was solved:

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3
1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
on the master node and set the --terminated-pod-gc-threshold to an appropriate threshold,
for example:
--terminated-pod-gc-threshold=10
--
1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml
on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true.
--feature-gates=RotateKubeletServerCertificate=true

Problem solved and 1.3.2 is passing:

root@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.3.2
[PASS] 1.3.2 Ensure that the --profiling argument is set to false (Scored)

Number 2

Next task (2) is to check the ownership of directory /var/lib/etcd, so we first have a look:

➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd
drwx------  3 root      root      4.0K Sep 11 20:08 etcd

Looks like user root and group root. Also possible to check using:

➜ root@cluster2-controlplane1:~# stat -c %U:%G /var/lib/etcd
root:root

But what has kube-bench to say about this?

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep "/var/lib/etcd" -B5
1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command:
ps -ef | grep etcd Run the below command (based on the etcd data directory found above). For example, chown etcd:etcd /var/lib/etcd

To comply we run the following:

➜ root@cluster2-controlplane1:~# chown etcd:etcd /var/lib/etcd
➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd drwx------  3 etcd      etcd      4.0K Sep 11 20:08 etcd

This looks better. We run kube-bench again, and make sure test 1.1.12. is passing.

➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.1.12
[PASS] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)

Done.

Number 3

To continue with number (3), we'll head to the worker node and ensure that the kubelet configuration file has the minimum necessary permissions as recommended:

➜ ssh cluster2-node1
➜ root@cluster2-node1:~# kube-bench run --targets=node ... == Summary == 13 checks PASS 10 checks FAIL 2 checks WARN 0 checks INFO

Also here some passes, fails and warnings. We check the permission level of the kubelet config file:

➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml
777

777 is highly permissive access level and not recommended by the kube-bench guidelines:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep /var/lib/kubelet/config.yaml -B2
4.1.9 Run the following command (using the config file location identified in the Audit step) chmod 644 /var/lib/kubelet/config.yaml

We obey and set the recommended permissions:

➜ root@cluster2-node1:~# chmod 644 /var/lib/kubelet/config.yaml
➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml 644

And check if test 2.2.10 is passing:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep 4.1.9
[PASS] 2.2.10 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)

Number 4

Finally for number (4), let's check whether --client-ca-file argument for the kubelet is set properly according to kube-bench recommendations:

➜ root@cluster2-node1:~# kube-bench run --targets=node | grep client-ca-file
[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated)

This looks passing with 4.2.3. The other ones are about the file that the parameter points to and can be ignored here.

To further investigate we run the following command to locate the kubelet config file, and open it:

➜ root@cluster2-node1:~# ps -ef | grep kubelet
root      5157     1  2 20:28 ?        00:03:22 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2
root     19940 11901  0 22:38 pts/0    00:00:00 grep --color=auto kubelet

➜ root@croot@cluster2-node1:~# vim /var/lib/kubelet/config.yaml
/var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
...

The clientCAFile points to the location of the certificate, which is correct.

Question 6 | Verify Platform Binaries

Task weight: 2%

(can be solved in any kubectl context)

There are four Kubernetes server binaries located at /opt/course/6/binaries. You're provided with the following verified sha512 values for these:

kube-apiserver

f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c

kube-controller-manager

60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

kube-proxy

52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6

kubelet

4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26c

Delete those binaries that don't match with the sha512 values above.

Answer:

We check the directory:

➜ cd /opt/course/6/binaries
➜ ls kube-apiserver  kube-controller-manager  kube-proxy  kubelet

To generate the sha512 sum of a binary we do:

➜ sha512sum kube-apiserver 
f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c  kube-apiserver

Looking good, next:

➜ sha512sum kube-controller-manager
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60  kube-controller-manager

Okay, next:

➜ sha512sum kube-proxy
52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6  kube-proxy

Also good, and finally:

➜ sha512sum kubelet
7b720598e6a3483b45c537b57d759e3e82bc5c53b3274f681792f62e941019cde3d51a7f9b55158abf3810d506146bc0aa7cf97b36f27f341028a54431b335be  kubelet

Catch! Binary kubelet has a different hash!

But did we actually compare everything properly before? Let's have a closer look at kube-controller-manager again:

➜ sha512sum kube-controller-manager > compare
➜ vim compare

Edit to only have the provided hash and the generated one in one line each:

# ./compare
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60  
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

Looks right at a first glance, but if we do:

➜ cat compare | uniq
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60
60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

This shows they are different, by just one character actually.

To complete the task we do:

rm kubelet kube-controller-manager

Question 7 | Open Policy Agent

Task weight: 6%

Use context: kubectl config use-context infra-prod

The Open Policy Agent and Gatekeeper have been installed to, among other things, enforce blacklisting of certain image registries. Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com.

Test it by creating a single Pod using image very-bad-registry.com/image in Namespace default, it shouldn't work.

You can also verify your changes by looking at the existing Deployment untrusted in Namespace default, it uses an image from the new untrusted source. The OPA contraint should throw violation messages for this one.

Answer:

We look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:

➜ k get crd
NAME                                                 CREATED AT
blacklistimages.constraints.gatekeeper.sh            2020-09-14T19:29:31Z
configs.config.gatekeeper.sh                         2020-09-14T19:29:04Z
constraintpodstatuses.status.gatekeeper.sh           2020-09-14T19:29:05Z
constrainttemplatepodstatuses.status.gatekeeper.sh   2020-09-14T19:29:05Z
constrainttemplates.templates.gatekeeper.sh          2020-09-14T19:29:05Z
requiredlabels.constraints.gatekeeper.sh             2020-09-14T19:29:31Z

So we can do:

➜ k get constraint
NAME                                                           AGE
blacklistimages.constraints.gatekeeper.sh/pod-trusted-images   10m
NAME                                                                  AGE requiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels   10m

and then look at the one that is probably about blacklisting images:

k edit blacklistimages pod-trusted-images

# kubectl edit blacklistimages pod-trusted-images
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: BlacklistImages
metadata:
...
spec:
  match:
    kinds:
    - apiGroups:
      - ""
      kinds:
      - Pod

It looks like this constraint simply applies the template to all Pods, no arguments passed. So we edit the template:

<pre><code class="lang-sh">k edit constrainttemplates blacklistimages</code></pre>

<pre><code class="lang-yaml"># kubectl edit constrainttemplates blacklistimages apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: ... spec: crd: spec: names: kind: BlacklistImages targets: - rego: | package k8strustedimages

images { image := input.review.object.spec.containers[_].image not startswith(image, &quot;docker-fake.io/&quot;) not startswith(image, &quot;google-gcr-fake.com/&quot;) not startswith(image, &quot;very-bad-registry.com/&quot;) # ADD THIS LINE }

violation[{&quot;msg&quot;: msg}] { not images msg := &quot;not trusted image!&quot; } target: admission.k8s.gatekeeper.sh</code></pre>

We simply have to add another line. After editing we try to create a Pod of the bad image:

<pre><code class="lang-sh">➜ k run opa-test --image=very-bad-registry.com/image Error from server ([denied by pod-trusted-images] not trusted image!): admission webhook &quot;validation.gatekeeper.sh&quot; denied the request: [denied by pod-trusted-images] not trusted image!</code></pre>

Nice! After some time we can also see that Pods of the existing Deployment "untrusted" will be listed as violators:

<pre><code class="lang-sh">➜ k describe blacklistimages pod-trusted-images ... Total Violations: 2 Violations: Enforcement Action: deny Kind: Namespace Message: you must provide labels: {&quot;security-level&quot;} Name: sidecar-injector Enforcement Action: deny Kind: Pod Message: not trusted image! Name: untrusted-68c4944d48-tfsnb Namespace: default Events: &lt;none&gt;</code></pre>

Great, OPA fights bad registries !

Question 8 | Secure Kubernetes Dashboard

Task weight: 3%

Use context: kubectl config use-context workload-prod

The Kubernetes Dashboard is installed in Namespace kubernetes-dashboard and is configured to:

• Allow users to "skip login"
• Allow insecure access (HTTP without authentication)
• Allow basic authentication
• Allow access from outside the cluster

You are asked to make it more secure by:

• Deny users to "skip login"
• Deny insecure access, enforce HTTPS (self signed certificates are ok for now)
• Add the --auto-generate-certificates argument
• Enforce authentication using a token (with possibility to use RBAC)
• Allow only cluster internal access

Answer:

Head to https://github.com/kubernetes/dashboard/tree/master/docs to find documentation about the dashboard. This link is not on the allowed list of urls during the real exam. This means you should be provided will all information necessary in case of a task like this.

First we have a look in Namespace kubernetes-dashboard:

<pre><code class="lang-sh">➜ k -n kubernetes-dashboard get pod,svc NAME READY STATUS RESTARTS AGE pod/dashboard-metrics-scraper-7b59f7d4df-fbpd9 1/1 Running 0 24m pod/kubernetes-dashboard-6d8cd5dd84-w7wr2 1/1 Running 0 24m

NAME TYPE ... PORT(S) AGE service/dashboard-metrics-scraper ClusterIP ... 8000/TCP 24m service/kubernetes-dashboard NodePort ... 9090:32520/TCP,443:31206/TCP 24m</code></pre>

We can see one running Pod and a NodePort Service exposing it. Let's try to connect to it via a NodePort, we can use IP of any Node:

(your port might be a different)

<pre><code class="lang-sh">➜ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP ... cluster1-controlplane1 Ready master 37m v1.24.1 192.168.100.11 ... cluster1-node1 Ready &lt;none&gt; 36m v1.24.1 192.168.100.12 ... cluster1-node2 Ready &lt;none&gt; 34m v1.24.1 192.168.100.13 ...

➜ curl http://192.168.100.11:32520 &lt;!-- Copyright 2017 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the &quot;License&quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0</code></pre>

The dashboard is not secured because it allows unsecure HTTP access without authentication and is exposed externally. It's is loaded with a few parameter making it insecure, let's fix this.

First we create a backup in case we need to undo something:

<pre><code class="lang-sh">k -n kubernetes-dashboard get deploy kubernetes-dashboard -oyaml &gt; 8deploykubernetes-dashboard.yaml</code></pre>

Then:

<pre><code class="lang-sh">k -n kubernetes-dashboard edit deploy kubernetes-dashboard</code></pre>

The changes to make are :

<pre><code class="lang-sh">template: spec: containers: - args: - --namespace=kubernetes-dashboard - --authentication-mode=token # change or delete, &quot;token&quot; is default - --auto-generate-certificates # add #- --enable-skip-login=true # delete or set to false #- --enable-insecure-login # delete image: kubernetesui/dashboard:v2.0.3 imagePullPolicy: Always name: kubernetes-dashboard</code></pre>

Next, we'll have to deal with the NodePort Service:

<pre><code class="lang-sh">k -n kubernetes-dashboard get svc kubernetes-dashboard -o yaml &gt; 8svckubernetes-dashboard.yaml # backup

k -n kubernetes-dashboard edit svc kubernetes-dashboard</code></pre>

And make the following changes:

<pre><code class="lang-yaml">spec: clusterIP: 10.107.176.19 externalTrafficPolicy: Cluster # delete internalTrafficPolicy: Cluster ports: - name: http nodePort: 32513 # delete port: 9090 protocol: TCP targetPort: 9090 - name: https nodePort: 32441 # delete port: 443 protocol: TCP targetPort: 8443 selector: k8s-app: kubernetes-dashboard sessionAffinity: None type: ClusterIP # change or delete status: loadBalancer: {}</code></pre>

Let's confirm the changes, we can do that even without having a browser:

<pre><code class="lang-sh">➜ k run tmp --image=nginx:1.19.2 --restart=Never --rm -it -- bash If you don&#39;t see a command prompt, try pressing enter. root@tmp:/# curl http://kubernetes-dashboard.kubernetes-dashboard:9090 curl: (7) Failed to connect to kubernetes-dashboard.kubernetes-dashboard port 9090: Connection refused

➜ root@tmp:/# curl https://kubernetes-dashboard.kubernetes-dashboard curl: (60) SSL certificate problem: self signed certificate More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

➜ root@tmp:/# curl https://kubernetes-dashboard.kubernetes-dashboard -k &lt;!-- Copyright 2017 The Kubernetes Authors.</code></pre>

We see that insecure access is disabled and HTTPS works (using a self signed certificate for now). Let's also check the remote access:

(your port might be a different)

<pre><code class="lang-sh">➜ curl http://192.168.100.11:32520 curl: (7) Failed to connect to 192.168.100.11 port 32520: Connection refused

➜ k -n kubernetes-dashboard get svc NAME TYPE CLUSTER-IP ... PORT(S) dashboard-metrics-scraper ClusterIP 10.111.171.247 ... 8000/TCP kubernetes-dashboard ClusterIP 10.100.118.128 ... 9090/TCP,443/TCP</code></pre>

Much better.

Question 9 | AppArmor Profile

Task weight: 3%

Use context: kubectl config use-context workload-prod

Some containers need to run more secure and restricted. There is an existing AppArmor profile located at /opt/course/9/profile for this.

• Install the AppArmor profile on Node cluster1-node1. Connect using ssh cluster1-node1.

• Add label security=apparmor to the Node

• Create a Deployment named apparmor in Namespace default with:

* One replica of image nginx:1.19.2 * NodeSelector for security=apparmor * Single container named c1 with the AppArmor profile enabled

The Pod might not run properly with the profile enabled. Write the logs of the Pod into /opt/course/9/logs so another team can work on getting the application running.

Answer:

https://kubernetes.io/docs/tutorials/clusters/apparmor

Part 1

First we have a look at the provided profile:

<pre><code class="lang-sh">vim /opt/course/9/profile</code></pre>

<pre><code class="lang-yam"># /opt/course/9/profile

#include &lt;tunables/global&gt; profile very-secure flags=(attach_disconnected) { #include &lt;abstractions/base&gt;

file,

# Deny all file writes. deny /** w, }</code></pre>

Very simple profile named very-secure which denies all file writes. Next we copy it onto the Node:

<pre><code class="lang-sh">➜ scp /opt/course/9/profile cluster1-node1:~/ Warning: Permanently added the ECDSA host key for IP address &#39;192.168.100.12&#39; to the list of known hosts. profile 100% 161 329.9KB/s 00:00

➜ ssh cluster1-node1

➜ root@cluster1-node1:~# ls profile</code></pre>

And install it:

<pre><code class="lang-sh">➜ root@cluster1-node1:~# apparmor_parser -q ./profile</code></pre>

Verify it has been installed:

<pre><code class="lang-sh">➜ root@cluster1-node1:~# apparmor_status apparmor module is loaded. 17 profiles are loaded. 17 profiles are in enforce mode. /sbin/dhclient ... man_filter man_groff very-secure 0 profiles are in complain mode. 56 processes have profiles defined. 56 processes are in enforce mode. ... 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.</code></pre>

There we see among many others the very-secure one, which is the name of the profile specified in /opt/course/9/profile.

Part 2

We label the Node:

<pre><code class="lang-sh">k label -h # show examples

k label node cluster1-node1 security=apparmor</code></pre>

Part 3

Now we can go ahead and create the Deployment which uses the profile.

<pre><code class="lang-sh">k create deploy apparmor --image=nginx:1.19.2 $do &gt; 9_deploy.yaml</code></pre>

<pre><code class="lang-yaml">vim 9_deploy.yaml

9_deploy.yaml

apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: apparmor name: apparmor namespace: default spec: replicas: 1 selector: matchLabels: app: apparmor strategy: {} template: metadata: creationTimestamp: null labels: app: apparmor annotations: # add container.apparmor.security.beta.kubernetes.io/c1: localhost/very-secure # add spec: nodeSelector: # add security: apparmor # add containers: - image: nginx:1.19.2 name: c1 # change resources: {}</code></pre>

k -f 9_deploy.yaml create

What the damage?

➜ k get pod -owide | grep apparmor
apparmor-85c65645dc-jbch8     0/1     CrashLoopBackOff  ...   cluster1-node1
➜ k logs apparmor-85c65645dc-w852p /docker-entrypoint.sh: 13: /docker-entrypoint.sh: cannot create /dev/null: Permission denied /docker-entrypoint.sh: No files found in /docker-entrypoint.d/, skipping configuration 2021/09/15 11:51:57 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

This looks alright, the Pod is running on cluster1-node1 because of the nodeSelector. The AppArmor profile simply denies all filesystem writes, but Nginx needs to write into some locations to run, hence the errors.

It looks like our profile is running but we can confirm this as well by inspecting the container:

➜ ssh cluster1-node1
➜ root@cluster1-node1:~# crictl pods | grep apparmor be5c0aecee7c7       4 minutes ago       Ready               apparmor-85c65645dc-jbch8   ...
➜ root@cluster1-node1:~# crictl ps -a | grep be5c0aecee7c7 e4d91cbdf72fb    ...  Exited       c1           6            be5c0aecee7c7
➜ root@cluster1-node1:~# crictl inspect e4d91cbdf72fb | grep -i profile           "apparmor_profile": "localhost/very-secure",         "apparmorProfile": "very-secure",

First we find the Pod by it's name and get the pod-id. Next we use crictl ps -a to also show stopped containers. Then crictl inspect shows that the container is using our AppArmor profile. Notice to be fast between ps and inspect as K8s will restart the Pod periodically when in error state.

To complete the task we write the logs into the required location:

k logs apparmor-85c65645dc-jbch8 > /opt/course/9/logs

Fixing the errors is the job of another team, lucky us.

Question 10 | Container Runtime Sandbox gVisor

Task weight: 4%

Use context: kubectl config use-context workload-prod

Team purple wants to run some of their workloads more secure. Worker node cluster1-node2 has container engine containerd already installed and its configured to support the runsc/gvisor runtime.

Create a RuntimeClass named gvisor with handler runsc.

Create a Pod that uses the RuntimeClass. The Pod should be in Namespace team-purple, named gvisor-test and of image nginx:1.19.2. Make sure the Pod runs on cluster1-node2.

Write the dmesg output of the successfully started Pod into /opt/course/10/gvisor-test-dmesg.

Answer:

We check the nodes and we can see that all are using containerd:

➜ k get node -o wide
NAME                     STATUS   ROLES              ... CONTAINER-RUNTIME
cluster1-controlplane1   Ready    control-plane      ... containerd://1.5.2
cluster1-node1           Ready    <none>             ... containerd://1.5.2
cluster1-node2           Ready    <none>             ... containerd://1.5.2

But just one has containerd configured to work with runsc/gvisor runtime which is cluster1-node2.

(Optionally) we ssh into the worker node and check if containerd+runsc is configured:

➜ ssh cluster1-node2
➜ root@cluster1-node2:~# runsc --version runsc version release-20201130.0 spec: 1.0.1-dev
➜ root@cluster1-node2:~# cat /etc/containerd/config.toml | grep runsc   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]     runtime_type = "io.containerd.runsc.v1"

Now we best head to the k8s docs for RuntimeClasses https://kubernetes.io/docs/concepts/containers/runtime-class, steal an example and create the gvisor one:

vim 10_rtc.yaml
10_rtc.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc

k -f 10_rtc.yaml create

And the required Pod:

k -n team-purple run gvisor-test --image=nginx:1.19.2 $do > 10_pod.yaml

vim 10_pod.yaml
10_pod.yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: gvisor-test
  name: gvisor-test
  namespace: team-purple
spec:
  nodeName: cluster1-node2 # add
  runtimeClassName: gvisor   # add
  containers:
  - image: nginx:1.19.2
    name: gvisor-test
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

k -f 10_pod.yaml create

After creating the pod we should check if its running and if it uses the gvisor sandbox:

➜ k -n team-purple get pod gvisor-test
NAME          READY   STATUS    RESTARTS   AGE
gvisor-test   1/1     Running   0          30s
➜ k -n team-purple exec gvisor-test -- dmesg [    0.000000] Starting gVisor... [    0.417740] Checking naughty and nice process list... [    0.623721] Waiting for children... [    0.902192] Gathering forks... [    1.258087] Committing treasure map to memory... [    1.653149] Generating random numbers by fair dice roll... [    1.918386] Creating cloned children... [    2.137450] Digging up root... [    2.369841] Forking spaghetti code... [    2.840216] Rewriting operating system in Javascript... [    2.956226] Creating bureaucratic processes... [    3.329981] Ready!

Looking good. And as required we finally write the dmesg output into the file:

k -n team-purple exec gvisor-test > /opt/course/10/gvisor-test-dmesg -- dmesg

Question 11 | Secrets in ETCD

Task weight: 7%

Use context: kubectl config use-context workload-prod

There is an existing Secret called database-access in Namespace team-green.

Read the complete Secret content directly from ETCD (using etcdctl) and store it into /opt/course/11/etcd-secret-content. Write the plain and decoded Secret's value of key "pass" into /opt/course/11/database-password.

Answer:

Let's try to get the Secret value directly from ETCD, which will work since it isn't encrypted.

First, we ssh into the master node where ETCD is running in this setup and check if etcdctl is installed and list its options:

➜ ssh cluster1-controlplane1
➜ root@cluster1-controlplane1:~# etcdctl NAME:    etcdctl - A simple command line client for etcd.
WARNING:    Environment variable ETCDCTL_API is not set; defaults to etcdctl v2.    Set environment variable ETCDCTLAPI=3 to use v3 API or ETCDCTLAPI=2 to use v2 API.
USAGE:    etcdctl [global options] command [command options] [arguments...] ...    --cert-file value   identify HTTPS client using this SSL certificate file    --key-file value    identify HTTPS client using this SSL key file    --ca-file value     verify certificates of HTTPS-enabled servers using this CA bundle ...

Among others we see arguments to identify ourselves. The apiserver connects to ETCD, so we can run the following command to get the path of the necessary .crt and .key files:

cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

The output is as follows :

- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379 # optional since we're on same node

With this information we query ETCD for the secret value:

➜ root@cluster1-controlplane1:~# ETCDCTL_API=3 etcdctl \
--cert /etc/kubernetes/pki/apiserver-etcd-client.crt \
--key /etc/kubernetes/pki/apiserver-etcd-client.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/team-green/database-access

ETCD in Kubernetes stores data under /registry/{type}/{namespace}/{name}. This is how we came to look for /registry/secrets/team-green/database-access. There is also an example on a page in the k8s documentation which you could save as a bookmark to access fast during the exam.

The tasks requires us to store the output on our terminal. For this we can simply copy&paste the content into a new file on our terminal:

# /opt/course/11/etcd-secret-content
/registry/secrets/team-green/database-access
k8s
v1Secret
database-access team-green"*$3e0acd78-709d-4f07-bdac-d5193d0f2aa32bB 0kubectl.kubernetes.io/last-applied-configuration{"apiVersion":"v1","data":{"pass":"Y29uZmlkZW50aWFs"},"kind":"Secret","metadata":{"annotations":{},"name":"database-access","namespace":"team-green"}} z kubectl-client-side-applyUpdatevFieldsV1: {"f:data":{".":{},"f:pass":{}},"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:type":{}} pass     confidentialOpaque"

We're also required to store the plain and "decrypted" database password. For this we can copy the base64-encoded value from the ETCD output and run on our terminal:

➜ echo Y29uZmlkZW50aWFs | base64 -d > /opt/course/11/database-password
➜ cat /opt/course/11/database-password confidential

Question 12 | Hack Secrets

Task weight: 8%

Use context: kubectl config use-context restricted@infra-prod

You're asked to investigate a possible permission escape in Namespace restricted. The context authenticates as user restricted which has only limited permissions and shouldn't be able to read Secret values.

Try to find the password-key values of the Secrets secret1, secret2 and secret3 in Namespace restricted. Write the decoded plaintext values into files /opt/course/12/secret1, /opt/course/12/secret2 and /opt/course/12/secret3.

Answer:

First we should explore the boundaries, we can try:

➜ k -n restricted get role,rolebinding,clusterrole,clusterrolebinding
Error from server (Forbidden): roles.rbac.authorization.k8s.io is forbidden: User "restricted" cannot list resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "restricted"
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "restricted" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "restricted"
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User "restricted" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "restricted" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

But no permissions to view RBAC resources. So we try the obvious:

➜ k -n restricted get secret
Error from server (Forbidden): secrets is forbidden: User "restricted" cannot list resource "secrets" in API group "" in the namespace "restricted"
➜ k -n restricted get secret -o yaml apiVersion: v1 items: [] kind: List metadata:   resourceVersion: ""   selfLink: "" Error from server (Forbidden): secrets is forbidden: User "restricted" cannot list resource "secrets" in API group "" in the namespace "restricted"

We're not allowed to get or list any Secrets. What can we see though?

```sh ➜ k -n restricted get all NAME

---

README truncated. View on GitHub