Selkies - Stateful Workload Operator

Selkies is a platform built on GKE to orchestrate per-user stateful workloads.

Quick start

Assumptions

• You are a member of a Google Cloud organization.

* This is required for setup/scripts/createoauthclient.sh to use gcloud alpha iap oauth-brand commands, because these implicity operate on organization-internal brands. For more information, see this guide.

• You are granted the Owner role in a project in that organization.
• You have gcloud installed in your environment.

Steps

The steps below will create the infrastructure for the app launcher. You should deploy to a new project.

• Clone the source repository:

git clone -b master https://github.com/selkies-project/selkies.git
    cd selkies

• Configure gcloud (replace XXX & us-west1 with your project ID & preferred region):

export PROJECT_ID=XXX
    export REGION=us-west1
    gcloud config set project ${PROJECT_ID?}
    gcloud config set compute/region ${REGION?}

• Enable the required GCP project services:

gcloud services enable \
        --project ${PROJECT_ID?} \
        cloudresourcemanager.googleapis.com \
        compute.googleapis.com \
        container.googleapis.com \
        cloudbuild.googleapis.com \
        servicemanagement.googleapis.com \
        serviceusage.googleapis.com \
        stackdriver.googleapis.com \
        secretmanager.googleapis.com \
        iap.googleapis.com

• Grant the cloud build service account permissions on your project:

PROJECT_NUMBER=$(
      gcloud projects describe ${PROJECT_ID?} \
        --format='value(projectNumber)'
    ) && \
      CLOUDBUILDSA="${PROJECTNUMBER?}@cloudbuild.gserviceaccount.com" && \
      gcloud projects add-iam-policy-binding ${PROJECT_ID?} \
        --member serviceAccount:${CLOUDBUILD_SA?} \
        --role roles/owner && \
      gcloud projects add-iam-policy-binding ${PROJECT_ID?} \
        --member serviceAccount:${CLOUDBUILD_SA?} \
        --role roles/iam.serviceAccountTokenCreator

• Deploy with Cloud Build:

ACCOUNT=$(gcloud config get-value account) && \
      gcloud builds submit \
        --project=${PROJECT_ID?} \
        --substitutions=USER=${ACCOUNT?},REGION=${REGION?}

• Deploy sample app:

(cd examples/jupyter-notebook/ && \
      gcloud builds submit \
        --project=${PROJECT_ID?} \
        --substitutions=_REGION=${REGION?})

• Connect to the App Launcher web interface at the URL output below:

echo "https://broker.endpoints.${PROJECT_ID?}.cloud.goog/"

Troubleshooting

• If the initial cloud build fails with the message

Step #2 - "create-oauth-client": ERROR: (gcloud.alpha.iap.oauth-brands.list) INVALID_ARGUMENT: Request contains an invalid argument., it is most likely due to running as a user that is not a member of the Cloud Identity Organization. See the assumption described above.

• If the initial cloud build fails with the message

Step #2 - "create-oauth-client": ERROR: (gcloud.alpha.iap.oauth-clients.create) FAILED_PRECONDITION: Precondition check failed., it is most likely due to reusing a project that already had its OAuth consent screen set to "External", which cannot be changed via gcloud. Click the "MAKE INTERNAL" button here in your project.

• If a wget step fails, retry the same command.

Some third-party artifact URLs are flaky (due to globally-rate-limited hosts).

• If your region only has 500 GB of Persistent Disk SSD quota, run the following,

but keep in mind the number of apps and image pull performance will be affected.

cat - > selkies-min-ssd.auto.tfvars <<EOF
    defaultpooldisksizegb = 100
    turnpooldisksizegb = 100
    gpucospooldisksize_gb = 100
    tier1pooldisksizegb = 100
    EOF

gcloud secrets create broker-tfvars-selkies-min-ssd \
        --replication-policy=automatic \
        --data-file selkies-min-ssd.auto.tfvars

• If the load balancer never comes online and you receive 500 errors

after the deployment has completed for at least 30 minutes, the autoneg controller annotation may need to be reset:

gcloud container clusters get-credentials broker-${REGION?}

./setup/scripts/fix_autoneg.sh