DEPRECATED

Secretary

Secretary helps solve the problem of secrets distribution and authorization in highly dynamic container and VM environments. NaCL and AWS Key Management Service (KMS) are supported crypto backends and can be mixed freely.

System Components

• secretary executable embedded into service Docker images and with access to

• secretary daemon running on master nodes behind a load balancer and with

• config repo containing configuration, public keys and encrypted secrets.

Design

In Mesos clusters it may not be desirable to have all slave nodes hold master keys or access KMS directly. A container would instead call secretary daemon which authenticates its signature and performs the decryption in a central place. The secretary daemon queries Marathon to retrieve a containers public keys and determine what secrets it may access.

Encryption is done at configuration time through public keys or by calling KMS. This enables delegation of secrets management to non-admin users and help keep configuration, secrets and software versions together throughout the delivery pipeline.

NaCL Crypto

Secretary uses 4 distinct key pairs for encrypting secrets and authenticating service instances.

• master key is used to encrypt the secrets stored in the config repo and is

• config key pair is used to sign encrypted secrets and control who can create

• deploy key pair is used to control what service can access what secrets, and

Access to the Marathon REST API should be restricted to avoid reading out the deploy private keys, and not to mention prevent anyone from starting containers with --privileged --volume=/:/host-root.

• The optional service key pair is used to authenticate Docker images or slave

Amazon AWS KMS

KMS coupled with IAM roles and CloudTrail provides access control and audit trails at the instance level. AWS EC2 instances could then use secretary to decrypt secrets embedded into user-data or VM images.

Compared to Centralized Systems?

• Encryption of secrets and modifications to the config repo can safely be

• It's often desirable to tightly couple deployment of configuration and secrets

This helps avoid mismatches between what parameters and secrets a specific software version expects, and what's actually present in the central secret/config management system.

Initial Secret Problem?

Secretary mitigates this problem by storing encrypted secrets in the config repo and keeping them encrypted all the way into the runtime environment. Secrets are only ever decrypted inside the container at startup and stored in environment variables visible only to the service.

Secrets can only be accessed by a process that holds both the deploy and service private keys. The deploy key is generated for each single deployment and is available only to specific containers. While the service key is available on slave nodes or embedded into a single application image.

What is needed to get the secrets?

In the runtime env:

• Encrypted secret from runtime config
• Deploy private key from runtime config
• Service private key from Docker image or slave node
• Network access to secretary daemon

• Encrypted secret from config repo
• Master private key from master nodes

Getting Started

Store master-public-key and config private/public key in the config repo together with other environment config and encrypted secrets. This enables users with access to the config repo to encrypt secrets and store them in the config.

Generate a new deploy key for each deployment and insert it into the env element of the Marathon app config. Lighter will perform this step automatically given this config example

someenv/globals.yml - stored in the Lighter config repo

secretary:   url: 'https://secretary-daemon-loadbalancer:5070'   master:     publickey: 'someenv/keys/master-public-key.pem'

someenv/myservice.yml - stored in the Lighter config repo

maven:   groupid: "com.example"   artifactid: "myservice"   classifier: "marathon"   resolve: [1.0.0,2.0.0) override:   env:     DATABASE_USERNAME: "myservice"     DATABASE_PASSWORD: "ENC[NACL,NVnSkhxA010D2yOWKRFog0jpUvHQzmkmKKHmqAbHAnz8oGbPEFkDfyKHQHGO7w==]"     DATABASE_URL: "jdbc:mysql://hostname:3306/schema"

Service Key In Docker Image

At build time generate a new service key using e.g. secretary genkeys service and embed the service-private-key into the Docker image. Ensure it's chmod 0600 root-only readable and that a new key is created for each build/release.

The service-public-key needs to be available in the Marathon app env as $SERVICEPUBLICKEY so that secretary daemon can find it when querying Marathon. A solution could be deploying a template JSON app config to a Maven repository and use Lighter to pull it down at deployment time. For example

myservice-1.0.0-marathon.json - deployed to Maven

{   "id": "/myproduct/mysubsystem/myservice"   "env" {     "SERVICEPUBLICKEY": "rEmz7Rt6tUnlC4TKYeNzePYg+p1ePAw4BAtfJAY4zzs="   } }

Service Key In VM Image

someenv/globals.yml - stored in the Lighter config repo

variables:   secretary.service.publickey: "WvDT+V2fB5ZKkbAmHaFh2XqDXC/veVsl1FKSE/HzxC0="

someenv/myservice.yml

override:   env:     SERVICEPUBLICKEY: "%{secretary.service.publickey}"     SERVICEPRIVATEKEY: /service/keys/service-private-key.pem   container:     volumes:      - containerPath: "/service/keys"        hostPath: "/etc/secretary/service-keys"        mode: "RO"

Runtime Config

{
    "id": "/myproduct/mysubsystem/myservice"
    ...
    "env" {
        "SECRETARY_URL": "https://secretary-daemon-loadbalancer:5070",
        "MASTERPUBLICKEY": "MX+S1xWkxfKlZUvzaEhBLkIVWEkwIrEaD9uKXVC5IGE=",
        "DEPLOYPUBLICKEY": "0k+v11LV3SOr+XiFJ/ug0KcPPhwkXnVirmO65nAd1LI=",
        "DEPLOYPRIVATEKEY": "rEmz7Rt6tUnlC4TKYeNzePYg+p1ePAw4BAtfJAY4zzs=",
        "SERVICEPUBLICKEY": "/1fbWGMTaR+lLQJnEsmxdfwWybKOpPQpyWB3FpNmOF4=",
        "SERVICEPRIVATEKEY": "/service/keys/service-private-key.pem"
        "DATABASE_USERNAME": "myservice",
        "DATABASE_PASSWORD": "ENC[NACL,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA=]",
        "DATABASE_URL": "jdbc:mysql://hostname:3306/schema"
        "BUCKET_TOKEN": "ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]"
    }
    ...
}

Container Startup Sequence

Dockerfile <pre><code class="lang-"># Install secretary ENV SECRETARY_VERSION x.y.z RUN curl -fsSLo /usr/bin/secretary &quot;https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-uname -s-uname -m" && \ chmod +x /usr/bin/secretary

• client asks the secretary daemon for the DATABASE_PASSWORD secret to be decrypted. This

• daemon retrieves SERVICEPUBLICKEY and DEPLOYPUBLICKEY from Marathon and uses it to

• client outputs a sh script export DATABASE_PASSWORD='secret' fragment that is sourced into the

BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" SECRETARY="$BASEDIR/target/secretary-uname -s-uname -m-${SECRETARY_VERSION}"

if [ ! -x "$SECRETARY" ]; then mkdir -p $(dirname "$SECRETARY") curl -sSfLo "$SECRETARY" https://github.com/meltwater/secretary/releases/download/${SECRETARY_VERSION}/secretary-uname -s-uname -m chmod +x "$SECRETARY" fi

The daemon has an HTTP health check endpoint at /v1/status that will respond with HTTP 200 OK if all is well. This could be used to point a load balancers health check mechanism at.

In order to enable end to end encryption, you can supply the ssl certificate through environment variable: TLSKEYFILE and TLSCERTFILE or with the following options: <pre><code class="lang-">secretary daemon --tls-key-file &lt;path to key file&gt; --tls-cert-file &lt;path to cert file&gt;</code></pre>

In the examples replace the SECRETARYVERSION with a version from the releases page. You also need to replace the e59c5534e4e6fb3c2ad0d3c075d9e2fa664889b9 sha1sum with one that is calculated from the exact version you intend to use. This can be done like

#cloud-config

coreos:
  units:
  - name: myservice.service
    command: start
    content: |
      [Unit]
      After=docker.service decrypt.service
      Requires=docker.service decrypt.service
[Install]       WantedBy=multi-user.target
[Service]       EnvironmentFile=/etc/environment.encrypted       Environment=IMAGE=myservice:latest NAME=myservice
# Allow docker pull to take some time       TimeoutStartSec=600
# Restart on failures       KillMode=none       Restart=always       RestartSec=15
# Start Docker container       ExecStartPre=-/usr/bin/docker kill ${NAME}       ExecStartPre=-/usr/bin/docker rm ${NAME}       ExecStartPre=-/bin/sh -c 'if ! docker images | tr -s " " : | grep "^${IMAGE}:"; then docker pull "${IMAGE}"; fi'             ExecStart=/usr/bin/docker run --name ${NAME} \         -e "DATABASEPASSWORD=${DATABASEPASSWORD}" \         -e "APIKEY=${APIKEY}" \         ${IMAGE}
write_files:   - path: "/etc/environment.encrypted"     permissions: "0600"     owner: "root"     content: |       DATABASE_PASSWORD=ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]       API_KEY=ENC[KMS,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA==]

#cloud-config

coreos:
  units:
  - name: myservice.service
    command: start
    content: |
      [Unit]
      After=docker.service decrypt.service
      Requires=docker.service decrypt.service
[Install]       WantedBy=multi-user.target
[Service]       EnvironmentFile=/etc/environment.encrypted       Environment=IMAGE=myservice:latest NAME=myservice SECRETARY_VERSION=x.y.z
# Allow docker pull to take some time       TimeoutStartSec=600
# Restart on failures       KillMode=none       Restart=always       RestartSec=15
# Download and verify signature of secretary binary       ExecStartPre=/bin/sh -c '\         if [ ! -f /tmp/secretary ]; then \           curl -sSLo /tmp/secretary https://github.com/meltwater/secretary/releases/download/${SECRETARYVERSION}/secretary-Linux-x8664 && \           chmod +x /tmp/secretary; \         fi'       ExecStartPre=/bin/sh -c 'echo e59c5534e4e6fb3c2ad0d3c075d9e2fa664889b9 /tmp/secretary | sha1sum -c -'
# Start Docker container       ExecStartPre=-/usr/bin/docker kill ${NAME}       ExecStartPre=-/usr/bin/docker rm ${NAME}       ExecStartPre=-/bin/sh -c 'if ! docker images | tr -s " " : | grep "^${IMAGE}:"; then docker pull "${IMAGE}"; fi'             ExecStart=/bin/sh -c 'eval $(/tmp/secretary decrypt -e) && docker run --name myservice \         -e "DATABASEPASSWORD=${DATABASEPASSWORD}" \         -e "APIKEY=${APIKEY}" \         myservice:latest'
write_files:   - path: "/etc/environment.encrypted"     permissions: "0600"     owner: "root"     content: |       DATABASE_PASSWORD=ENC[KMS,RP+BAwEBCmttc1BheWxvYWQB/4IAAQMBEEVuY3J5cHRlZERhdGFLZXkBCgABBU5vbmNlA==]       API_KEY=ENC[KMS,SLXf+O9iG48uyojT0Zg30Q8/uRV8DizuDWMWtgL5PmTU54jxp5cTGrYeLpd86rA==]