CloudControl ☁️ 🧰

The cloud engineer's toolbox.

Introduction

CloudControl is a Docker based configuration environment containing all the tools required and configured to manage modern cloud infrastructures.

The toolbox comes in different "flavours" depending on what cloud you are working in. Currently supported cloud flavours are:

• AWS (based on amazon/aws-cli) [linux/amd64, linux/arm64]
• Azure (based on mcr.microsoft.com/azure-cli) [linux/amd64, linux/arm64]
• Google Cloud (based on google-cloud-cli) [linux/amd64, linux/arm64]
• Simple (based on alpine) [linux/amd64, linux/arm64]
• Tanzu (based on alpine) [linux/amd64]

• 🐟 Fish Shell
• 📷 AzCopy
• 🪪 CA-Certificates
• 🪪 Certificates ⚠️ Deprecated: Please use the ca-certificates feature instead.
• 📨 cmctl
• ⚙️ Direnv
• ⛵️ Helm
• 🛠 JQ
• 🐾 k9s
• ⌨️ kc Quick Kubernetes Context switch ⚠️ Deprecated: The kc feature will be deprecated in one of the next major CloudControl versions. As an alternative, configure

- USE_krew=yes 
KREW_VERSION=<current version of krew>
 
KREW_PLUGINS=cs

- USE_run=yes
RUN_COMMANDS="alias kc='kubectl cs'"

• 🟦 krew
• 🐚 Kubectlnodeshell
• 🐳 Kubernetes
• 📦 Opentofu
• 📦 Packages
• 📦 Packer
• 👟 Run
• 🔑 sops
• 📜 Stern
• ⎔ Tanzu CLI
• 🌏 Terraform
• 🐗 Terragrunt
• 🕰 Timezone configuration
• 🌊 Velero
• 𝑉 Vim
• 🛠 YQ

Table of contents

• Usage
• Using Kubernetes
• FAQ
• Flavours

• Features

• Development
• Building

Usage

CloudControl can be used best with docker compose. Check out the sample directory in a flavour for a sample compose file and to convenience scripts. It includes a small web server written in Go and Vuejs-client dubbed "CloudControlCenter", which is used as a status screen. It listens to port 8080 inside the container.

Copy the compose file and configure it to your needs. Check below for configuration options per flavour and feature.

Run init.sh. This script basically just runs docker compose up -d and tells you the URL for CloudControlCenter. Open it and wait for CloudControl to finish initializing.

The initialization process will download and configure the additional tools and completes with a message when its done. It will run each time when the stack is recreated.

After the initialization process you can simply run docker compose exec cli /usr/local/bin/cloudcontrol run to jump into the running container and work with the installed features.

If you need to change any of the configuration environment variables, rerun the init script afterwards to apply the changes. Remember, that CloudControl needs to reininitialize for this.

Configuring features

There are two ways to configure a feature and the version it should use. The first way is to use the given USE[feature name]=yes environment variable and specifying the version with [FEATURE NAME]VERSION=[version].

If there are multiple features configured, this can get a bit messy. Another approach is to use the FEATURES environment variable and list the features and optionally the version like this:

FEATURES=kubernetes helm:3.5.1 terraform:1.1.9

This would install the version 3.5.1 of Helm and version 1.1.9 of terraform. (Kubernetes uses the flavour's provided version of kubectl, e.g. using az aks install-cli)

Note: Please see the feature documentation below if a feature supports specifying a version string. All version strings need to be provided in semver format (i.e. 1.2.3), the feature installers will take care about prefixes for download URLs, if required.

Using Kubernetes (Preview)

CloudControl is targeted to run on a local machine. It requires the following features to work:

• host path volumes
• host based networking

The sample directories of each flavour provide an example Kubernetes configuration based on a deployment and a service. They were preliminary tested on Rancher desktop.

Modify them to your local requirements and then run

kubectl apply -f k8s.yaml

to apply them.

This will create a new namespace for your project and a deployment and a service in that. Check kubectl get -n [project] pod to watch the progress until a cli pod has been created.

Use kubectl get -n [project] svc cli to see the bound ports for the cli service and use your browser to connect to the CloudControlCenter instance.

After the initialization is done, use kubectl -n [project] exec -it deployment/cli -- /usr/local/bin/cloudcontrol run to enter CloudControl.

Warning: This implementation is currently a preview feature and hasn't been tested thoroughly. It highly depends on the proper support for host based volumes and networking of the Kubernetes distribution. Please refer to the documentation and support of your Kubernetes distribution if something isn't working.

FAQ

What does an error message like "no matching manifest for linux/arm64/v8 in the manifest list entries" mean?

Apparently you're using CloudControl on a system for which no specific image exist. Some cloud providers have not provided base images for all architectures (e.g. the Apple ARM-based processors) yet. See the list of flavours above for the available platforms per flavour.

As a workaround this, you can use the platform parameter for docker compose or the --platform parameter for docker run to specify a compatible architecture (e.g. linux/amd64 on Apple ARM-based machines).

How can I add an informational text for users of CloudControl?

If you want to display a custom login message when users enter the container, set environment variable MOTD to that message. If you want to display the default login message as well, also set the environment variable MOTDDISPLAYDEFAULT to yes.

How can I forward ports to the host?

If you'd like to forward traffic into a cluster using kubectl port-forward you can do the following:

• Add a ports key to the cli-service in your docker-compose.yaml file to forward a free port on your host to a defined

Inside CloudControl*, check the IP of the container:

bash-5.0$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:42:AC:15:00:02
          inet addr:172.21.0.2  Bcast:172.21.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:53813 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20900 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:75260363 (71.7 MiB)  TX bytes:2691219 (2.5 MiB)

• Use the IP address used by the container as the bind address for port-forward to forward traffic

kubectl port-forward --address 172.21.0.2 svc/my-service 8081:8080

• Check out, which host port docker bound to the private port you set up (e.g. 8081)

docker compose port cli 8081

• Connect to localhost:[host port] on your host

How to set up command aliases

If you'd like to set up aliases to save some typing, you can use the run feature. Run your container with these environment variables:

• USE_run=yes: Set up the run feature
• RUN_COMMANDS=alias firstalias=command;alias secondalias=command: Set up some aliases

How can I share my SSH-keys with the CloudControl container

First, mount your .ssh directory into the container at /home/cloudcontrol/.ssh.

Also, to not enter your passphrase every time you use the key, you should mount the ssh agent socket into the container and set the environment variable SSHAUTHSOCK to that path. CloudControl will automatically fix the permissions of that file so the CloudControl user can use it.

Here are snippets for your docker-compose.yaml file for convenience:

(...) volumes: - "[Path to .ssh directory]:/home/cloudcontrol/.ssh" # for Linux: - "${SSHAUTHSOCK}:/ssh-agent" # for macOS: - "/run/host-services/ssh-auth.sock:/ssh-agent" environment: - "SSHAUTHSOCK=/ssh-agent"

How to identify Terraform state locks by other CloudControl-users

Because of how CloudControl is designed it uses a defined user named "cloudcontrol", so Terraform state lock messages look like this:

Error: Error locking state: Error acquiring the state lock: storage: service returned error: StatusCode=409, ErrorCode=LeaseAlreadyPresent, ErrorMessage=There is already a lease present.

It's hard to identify from that who the other CloudControl user is, that may have opened a lock. The system user can't be changed, but it's possible to set a better hostname than the one Docker autogenerated.

See this docker-compose.yaml snippet on how to set a better hostname:

services: cli: image: "dodevops/cloudcontrol-azure:latest" hostname: "[TODO yourname]" volumes: (...)

If you set hostname in that snippet to "alice", the state lock will look like this now:

Error: Error locking state: Error acquiring the state lock: storage: service returned error: StatusCode=409, ErrorCode=LeaseAlreadyPresent, ErrorMessage=There is already a lease present.

I get an error "repomd.xml signature could not be verified for kubernetes" when using Kubernetes in the AWS flavour

CloudControl uses the official guide to install kubectl on an RPM-based system. However, Google seems to regularly have problems with its key-signing in the used repository, so we added a workaround to this problem. If you add the environment variable AWSSKIPGPG=1 to your docker-compose.yaml, it will ignore an invalid GPG key during the installation.

Please note though, that this affects the security of the system and should not be used constantly.

When I try to start CloudControl, it keeps exiting with "./ccc: exit status 1". What can I do?

Use the docker logs command with the failed container to see the complete log output. You can enhance the log by using the "DEBUG[feature]" options or add the environment variable "DEBUGFLAVOUR" to turn on the debug log for the flavour installation.

If you are really stuck, you can convince the container to keep running by setting "CONTINUEONERROR=yes" as an environment variable in the docker-compose.yaml file. Then you can debug with the running container.

Flavours

aws

Can be used to connect to infrastructure in the AWS cloud. Also see the AWS CLI documentation for more configuration options. If you have activated MFA, set AWSMFAARN to the ARN of your MFA device so CloudControl will ask you for your code. To start a new session in the CloudControl context, run createSession <token> afterwards

Configuration

• Environment AWSACCESSKEY_ID: Specifies an AWS access key associated with an IAM user or role
• Environment AWSSECRETACCESS_KEY: Specifies the secret key associated with the access key. This is essentially the password for the access key
• Environment AWSDEFAULTREGION: Specifies the AWS Region to send the request to
• Environment AWSMFAARN: ARN of the MFA device to use to log in

azure

Can be used to connect to infrastructure in the Azure cloud. Because we're using a container, a device login will happen, requiring the user to go to a website, enter a code and login.

The azure login tokens usually expire after some time. You can run the azure-relogin script (located in ~/bin, thus available without path) to re-execute the same login commands as the initialization process does.

Configuration

• Environment ARMSUBSCRIPTIONID: The Azure subscription to use in this container
• Environment ARMTENANTID: The Azure tenant id to log into (optional)
• Environment AZ_SUBSCRIPTION: The Azure subscription to use in this container (deprecated)
• Environment AZ_TENANTID: The Azure tenant id to log into (optional, deprecated)
• Environment AZUSEARMSPI: Uses the environment variables ARMCLIENTID and ARMCLIENT_SECRET for service principal auth [false]

gcloud

Includes workflows and tools to connect to Google Cloud.

Authentication requires the following:

• Enable Cloud Resource Manager API
• Create a service account, that has access to the

• Create a key and download it as a JSON file
• Mount a directory that contains the JSON file into the CloudControl container and set GCLOUD_KEYPATH accordingly

Configuration

• Environment GOOGLE_PROJECT: The id of the Google Cloud project to connect to
• Environment GOOGLE_CREDENTIALS: Path inside CloudControl that holds the service account JSON file. Will use

simple

Can be used to connect to infrastructure outside of a specific cloud provider.

tanzu

Includes workflows and tools to connect to a Tanzu cluster.

Note about logins when using the "kubernetes" feature

The kubernetes login tokens usually expire after a few hours already. You can run the k8s-relogin script (located in ~/bin, thus available without path) to re-execute the same login commands as the initialization process does.

Features

Fish Shell

Installs and configures the Fish Shell with configured Spacefish theme

Configuration

• USE_fish: Enable this feature (or use the FEATURES variable instead)
• DEBUG_fish: Debug this feature

AzCopy

Installs AzCopy

Configuration

• USE_azcopy: Enable this feature (or use the FEATURES variable instead)
• DEBUG_azcopy: Debug this feature

CA-Certificates

Adds specified trusted certificate authorities into the container

Configuration

• USE_ca-certificates: Enable this feature (or use the FEATURES variable instead)
• DEBUG_ca-certificates: Debug this feature
• Add a volume mount to the volumes: section of docker compose like this:

- USE_krew=yes
KREW_VERSION=<current version of krew>
KREW_PLUGINS=cs

- USE_run=yes
RUN_COMMANDS="alias kc='kubectl cs'"

flowchart TD
    A[Every Push] --> B[Update documentation]
    D[Every PR] --> E[Check commits]
    D --> F[Run Testsuite]
    G[Push to Main] --> H[Generate Changelog and Release]
    G --> C
    I[Push to Develop] --> C[Build images]
    click B "https://github.com/dodevops/cloudcontrol/blob/develop/.github/workflows/docs.yml" "Docs workflow"
    click C "https://github.com/dodevops/cloudcontrol/blob/develop/.github/workflows/image.yml" "Image workflow"
    click E "https://github.com/dodevops/cloudcontrol/blob/develop/.github/workflows/check_commits.yml" "Check workflow"
    click F "https://github.com/dodevops/cloudcontrol/blob/develop/.github/workflows/test.yml" "Test workflow"
    click H "https://github.com/dodevops/cloudcontrol/blob/develop/.github/workflows/release.yml" "Release workflow"