Penelope is a tool for performing BigQuery and Cloud Storage Backups on the Google Cloud Platform
Penelope - GCP Backup Solution
Introduction
Penelope is a tool, which allows you to back up data stored in GCP automatically. You can create backups from BigQuery datasets and tables as well as from Cloud Storage buckets within Google Cloud Storage. For authentication against GCP services Penelope uses Google service accounts for performing backups and it assumes that it is behind an authentication provider like Google Identity Aware Proxy.
Penelope consists of three main components:
- A Docker image for a server written in GO providing an API with different methods to create, start, etc. backups
- A web frontend allowing users to easily create and manage backup jobs
- A PostgreSQL database storing different pieces of information about backup jobs
Requirements
- Go >= 1.23
- PostgreSQL >= 15
- Google Service Account
Getting Started
This repository provides a starter kit to set up Penelope on your own. Penelope uses providers for different purposes, for example penelope needs a credential to connect with the configured database (see environment variables). You can use Penelopes basic secret provider, which uses a specific environment variable to provide a secret credential, or you can define a more advance provider, which for example fetches the credentials during runtime. Penelope needs four specific providers:
SecretProvider - containing the method GetSecret*, which provides the database password for given user. SinkGCPProjectProvider - containing the method GetSinkGCPProjectID*, which provides for a given GCP project id a specific cloud storage backup sink. TargetPrincipalForProjectProvider - contains the method GetTargetPrincipalForProject*, which provides a target service account to be impersonated for a given project. PrincipalProvider - contains the method GetPrincipalForEmail*, which provides the users principal (containing the user and role bindings) for a given email address.
Migration
Penelope uses a PostgreSQL database to store the backup state. You can find the migrations under the folder resources/migrations/. You can use Flyway to run the migrations against your own PostgreSQL database.
Configuration
Penelope uses environment variables for customization. Therefore, you can configure penelope to a certain degree by setting specific environment variables (e.g. configure database connection). There are optional and required settings. If you not provide required settings, penelope will not run.
| Name | Required | Description | |-------------------------------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------| | GCPPROJECTID | required | Set the GCP project. | | DEFAULTPROVIDERBUCKET | required | Set the bucket for all providers | | DEFAULTBACKUPSINKPROVIDERFORPROJECTFILE_PATH | required | Set the path to the .yaml file which contains the target backup project for SinkGCPProjectProvider. | | DEFAULTUSERPRINCIPALPROVIDERFILE_PATH | required | Set the path to the .yaml file which contains the user principal for PrincipalProvider. | | DEFAULTGCPSOURCEPROJECTPROVIDERFILEPATH | required | Set the path to the .yaml file which contains the user principal for PrincSourceGCPProjectProvideripalProvider. | | DEFAULTPROVIDERCACHE_TTL | required | Set time to life (TTL) for data stored in cache by defualt providers | | DEFAULTPROVIDERIMPERSONATEGOOGLESERVICE_ACCOUNT | required | Set default impersonated google service account for TargetPrincipalForProjectProvider. | | DEV_MODE | required | Set Penelope to run locally in dev mode any skipping user authentification. | | APPJWTAUDIENCE | required | Set the expected audience value of the jwt token. | | COMPANY_DOMAINS | required | Set the company domains for validating user email. Value can be a comma separated list. | | DEFAULTBUCKETSTORAGE_CLASS | required | Set the default storage class for backup sinks. | | POSTGRES_SOCKET | required | Set socket address to PostgreSQL server. | | POSTGRES_HOST | required | Set host address to PostgreSQL server. If PostgreSQL socket is specified, setting this is optional. | | POSTGRES_PORT | required | Set port of PostgreSQL server default to 5432. | | POSTGRES_DB | required | Set name of PostgreSQL database. | | POSTGRES_USER | required | Set username to connect with PostgreSQL database. | | POSTGRES_PASSWORD | required | Set password for user to connect with PostgreSQL database. | | TOKENHEADERKEY | required | Set the key for token header. | | PENELOPE_PORT | optional | Set port for localhost when running Penelope local. | | PENELOPE_TRACING | optional | Set true to export tracing metrics to Stackdriver. Default is true. | | PENELOPETRACINGMETRICS_PREFIX | optional | Set prefix for tracing metrics when activated. Default is penelope-server. | | PENELOPEUSEDEFAULTHTTPCLIENT | optional | Switch to use default http request for testing by setting true. Default is false. | | CORSALLOWEDMETHODS | optional | Set the allowed methods for CORS with a comma separated list. For example, POST, PATCH, GET | | CORSALLOWEDORIGIN | optional | Set the allowed origins for defined cors methods. | | CORSALLOWEDHEADERS | optional | Set the allowed request headers. | | TASKSVALIDATIONHTTPHEADERNAME | optional | Adds request validation to tasks triggers. Specifies the expected request head for validation. | | TASKSVALIDATIONHTTPHEADERVALUE | optional | Expected value for request validation. | | TASKSVALIDATIONALLOWEDIPADDRESSES | optional | Adds ip address validation to tasks triggers. Multiple comma separated ip addresses can be specified. | | UNIFORMBUCKETLEVELACCESS | optional | Set uniform bucket level access for created backups (see more) |
Deploy Basic Setup
This step-by-step guide will walk you through how to set up Penelope in your own Google App Engine instance. Let us start with the database migration.
1. Step: Migration with Flyway
In the following you will learn, how you can use Flyway for migration. However, feel free to use any other tool which fits best for your use case. The migration files are in the folder resource/migrations as already mentioned above.
script
flyway migrate -url=jdbc:postgresql://<HOST>:<PORT>/<DB> -user=<USER> -password=<PW> -locations=filesystem:./resources/migrationsBecause we are going to deploy Penelope to App Engine, it maybe useful to take CloudSQL into consideration. You can use Cloud SQL Proxy to connect with your instance via a secure connection. In order to find out more about the proxy client see the About the Cloud SQL Proxy documentation.
2. Step: Configuration of App Engine
You are going to need a app.yaml file to deploy and configure your App Engine service. In this file you specify the go runtime version, url handlers and all environment variables to configure Penelope. This repository provides a default configuration template for your own App Engine. Replace the brackets and feel free to change the values, but be carefully with the handlers.
# app.yaml
runtime: go125
service: default
handlers:
- url: /
static_files: static/ui/index.html
upload: static/ui/index.html
# ...
env_variables: GCPPROJECTID: <GCPPROJECTID> PENELOPEPORT: <PENELOPEPORT> POSTGRESSOCKET: /cloudsql/<GCPPROJECT>:<REGION>:<DB_INSTANCE>/.s.PGSQL.5432 POSTGRESUSER: <POSTGRESUSER> POSTGRESDB: <POSTGRESDB> POSTGRESPASSWORD: <POSTGRESPASSWORD> # ...
3. Step: Penelope Deployment
Now that you have specified the configuration for Penelope, you are able to deploy the local application and configuration settings with Cloud SDK. For more details on how to install or manage your GCP resources and applications see Google Cloud SDK Documentation. Since we are going to deploy the application to app engine, we will use gcloud app deploy for deployment.
script
gcloud app deploy app.yaml4. Step: Configuration of Cron-Jobs
Congratulations. If you configured your application correctly, you successfully deployed Penelope to App Engine. But you're not done yet. There are still tasks, which need to be triggered. These Penelope tasks are responsible for making backups, cleanups of expired sinks and so on. This repository provides a basic cron job configuration as well for all tasks. There are no changes required.
# cron.yaml
cron:
- description: "prepare backup jobs"
url: /api/tasks/preparebackupjobs
schedule: every 60 minutes from 00:00 to 23:00
- description: "schedule new jobs"
url: /api/tasks/runnewjobs
schedule: every 10 minutes from 00:05 to 23:55
# ...5. Step: Cron-Jobs Scheduling
Deploying the cron.yaml configuration file to App Engine is straight forward. You just need to run the following command and you are finished.
script
gcloud app deploy cron.yamlProviders
This section is specifically tell you about the special Penelope providers. As mentioned before, there are four providers which provide Penelope with information like where to store the backup, which role bindings has the user and so on. This repository contains default providers. However, you are able to implement your own providers. In the following, you will find out how each default provider works and how you can implement your own provider. To use your own Penelope defined providers use AppStartArguments and pass it to the run function of the github.com/ottogroup/penelope/cmd package.
package main
import ( "github.com/ottogroup/penelope/cmd" )
func main() { // Create all your providers here ...
appStartArguments := app.AppStartArguments{ PrincipalProvider: principalProvider, SinkGCPProjectProvider: sinkGCPProjectProvider, TargetPrincipalForProjectProvider: targetPrincipalForProjectProvider, SecretProvider: secretProvider, }
cmd.Run(appStartArguments) }
The Secret Provider
Let's have a look at the first provider. The secret provider, specified by the SecretProvider interface, provides Penelope with the database password. This provider defines only one method. It receives a context.Context and string argument and returns a string and error type. You can probably guess the meaning of each argument. However, we will go through each parameter to be clear. The first expected argument is a context, which is created for each (http) request. This is golang specific. If you want to find out more about the Context type, you can read the Package Context documentation. The next argument contains the database user name. All you have to do is to return the password for this user. If you are not able to return the database password, you can return an error value.
package secret
import "context"
type SecretProvider interface { GetSecret(ctxIn context.Context, user string) (string, error) }
Default
The default provider is actually pretty straight forward. It basically doesn't care about the user argument. It just returns the value you have specified in the POSTGRES_PASSWORD environment variable. If this default provider is not advance enough for your need, then feel free to implement your own secret provider.
Backup Provider
The tasks of the backup sink provider is to provide Penelope with a GCP project where the backup should be stored. This provider is defined by the SinkGCPProjectProvider interface. The first argument is the same for all provider methods, which is again context. The next argument is the source GCP project id. It is the project of the source data, which should be backup on a target project. The task of this interface is to return the target project for the received source project.
package provider
import "context"
type SinkGCPProjectProvider interface { GetSinkGCPProjectID(ctxIn context.Context, sourceGCPProjectID string) (string, error) }
Default
The default provide is a bit more complex this time. You will not only have to define the environment variables DEFAULTPROVIDERBUCKET and DEFAULTBACKUPSINKPROVIDERFORPROJECTFILE_PATH, you also have to store a .yaml file in the specified bucket. The content of the file should look like this.
- project: project-one
backup: project-one-backup
- project: project-two
backup: project-two-backupFor each project you define a backup project (actually not that complex, huh?). But what happens, if a source project is not listed in the file? Then the default implementation returns an error. You think there are other solutions? Maybe you would like to create a backup projects on-the-fly or just use the source project as the target project. Then feel free to implement your own SinkGCPProjectProvider.
Target Principal Provider
This provider can be more difficult to comprehend than the previous providers. Behind the scenes, Penelope uses impersonation to create all the backup sinks and so on in GCP. And what does it impersonate to do all these tasks? Service accounts, which are special google account to represent non-human user like applications. To determine which service account should be impersonated by Penelope, the TargetPrincipalForProjectProvider interface is required. It returns the service account for a target project.
package impersonate
import "context"
type TargetPrincipalForProjectProvider interface { GetTargetPrincipalForProject(ctxIn context.Context, projectID string) (string, error) }
Default
The default is again pretty straight forward. You only have to define one single google service account, which should be impersonated. This is done by setting the DEFAULTPROVIDERIMPERSONATEGOOGLESERVICE_ACCOUNT environment variable.
Principal Provider
This section explains the concept of a user principal and the role of the PrincipalProvider interface.
User Principal:
- A
Principaldata type represents a user's identity and access rights within the system. - It contains two components:
email: A string representing the user's email address (unique identifier).
* role_bindings: A list of role bindings, which define a user's role within each project.
Role Binding:
- A role binding associates a project ID with a user's role for that specific project.
- Possible roles are:
None: User has no access to the project.
* Viewer: User can view project data but cannot modify it.
* Owner: User has full access to the project, including editing and backup privileges.
PrincipalProvider Interface:
- The
PrincipalProvider interface defines a single method:
: This method takes a user's email address and returns their
corresponding Principal data type.
Importance of PrincipalProvider:
- The
PrincipalProvider plays a crucial role in access control.
By retrieving a user's principal data, the system can determine their roles for specific projects.
This information is critical for authorizing actions:
* Only Owner users can perform backups.
* Users without the appropriate role (e.g., None or Viewer) cannot edit project data.
In summary:
- The Principal data type stores user identity and project access levels.
- The PrincipalProvider interface provides access to this information for authorization purposes.
<pre><code class="lang-go">package provider
import ( "context" "github.com/ottogroup/penelope/pkg/http/auth/model" )
type PrincipalProvider interface { GetPrincipalForEmail(ctxIn context.Context, email string) (*model.Principal, error) }</code></pre>
The data type
Principal is shown in following source code, which contains additionally all relevant information. You can see it consist of a User and a list of ProjectRoleBindings. Furthermore, you can see that User only consists of the email address. The ProjectRoleBinding contains the role for each project.
<pre><code class="lang-go">package model
type Role string
type Principal struct { User User RoleBindings []ProjectRoleBinding }
type User struct { Email string }
var ( None Role = "none" Viewer Role = "viewer" Owner Role = "owner" )
type ProjectRoleBinding struct { Role Role Project string }</code></pre>
Default
Now let's have a look at the default implementation. The default is very similar to the
SinkGCPProjectProvider. It also needs the path to a .yaml file. Therefore DEFAULTUSERPRINCIPALPROVIDERFILE_PATH needs to be set. The content can look like this.
<pre><code class="lang-yaml">- user: email: 'first-user@example.de' role_bindings: - role: owner project: 'project-one' - role: viewer project: 'project-two'
- user:
email: 'second-user@example.de' role_bindings: - role: viewer project: 'project-one' - role: viewer project: 'project-two'</code></pre>
Source Project Provider
source project orincipal is used to retrieve additioanl information about the source project. The
SourceGCPProjectProvider represents the interface for this provider. It contains only one method, which returns the SourceGCPProject for a given project id.
<pre><code class="lang-go">package provider
import ( "context" )
type SourceGCPProjectProvider interface { GetSourceGCPProject(ctxIn context.Context, gcpProjectID string) (SourceGCPProject, error) }</code></pre>
Default
Now let's have a look at the default implementation. The default is very similar to the
SinkGCPProjectProvider. It also needs the path to a .yaml file. Therefore DEFAULTBACKUPSINKPROVIDERFORPROJECTFILE_PATH needs to be set. The content can look like this.
<pre><code class="lang-yaml">- project: local-account availability_class: A1 data_owner: john.doe</code></pre>
Internal Data Model and Backup Mechanics
Penelope tracks backup configuration specified by the user as well as the backups current success state in the
backups table. Each backup definition lets Penelope schedule, execute and track (aka orchestrate) jobs in GCP. Depending on the backup source a job is either implemented as
- a StorageTransferJob, if the backup source is CloudStorage
- a BigQueryExtractJob, if the backup source is BigQuery
Penelope keeps track of jobs in the jobs table. A fk relation to the corresponding backup indentifies which backup
definition lead to a certain job.
Both db entities -
backups and jobs - have a status field representing the current state they are in. The status field different Penelope tasks operate on it and trigger a state chang in the model. State changes of backups are well-defined in the processor.go.
Since a StorageTransferJob is handled by GCP, it runs asynchronously. Penelope checks job statuses in regular intervals when the task
CheckJobsStatus is invoked. It monitors StorageTransferJobs and corresponding TransferOperations. The following diagram shows how the job status is assessed.
<pre><code class="lang-mermaid">flowchart LR Start["GetJobs(Scheduled || Pending)"] --> A[ListOperations] A --> B{Any Runs?} B --> |Yes| C{Any Run not done?} B --> |No| SPen[Pending] C --> |No| SDone[FinishedOk] C --> |Yes| D{Any of them failed?} D --> |Yes| SFail[FinishedError] D--> |No| SPen</code></pre>
Role and rights concept
<pre><code class="lang-mermaid">flowchart LR subgraph GCPsourceProject BigQuery GCS(Cloud Storage) end
subgraph GCPsinkProject SinkBucket end subgraph GCPruntimeProject Backup(Backup) Runner--"impersonate"-->Backup Runner--"runs PenelopeApplication in"-->AppEngine end STS--"member of"-->GCPsinkProject Backup--"PenelopeBackupManager"-->GCPsinkProject Backup--"PenelopeDataExporter"-->GCPsourceProject STS(Storage Transfer Service SA)--"
storage.legacyBucketWriter storage.legacyBucketReader"-->SinkBucket STS--"storagetransfer.user"-->GCPsinkProject STS--"storage.legacyBucketReader storage.objectViewer"-->GCPsourceProject</code></pre>
Service accounts
Runner
There should be one custom service account <pre><code class="lang-runner"></code></pre> that is used to run the Penelope application. This service account should have the following roles in the project it runs:
- CloudSQL Client (
roles/cloudsql.client)
* to be able to connect to the database running in the same project as Penelope application
- Cloud Trace Agent
* to be able to connect to write traces to Google Monitoring
Backup
The GCP Projects that are created for the pure purpose to store only backup data will be accessed by the <pre><code class="lang-backup"></code></pre> service accounts.
permission for runner service account
The <pre><code class="lang-runner"></code></pre> service account should to be able to impersonate the<pre><code class="lang-backup"></code></pre> service account with the role: Service Account Token Creator (
roles/roles/iam.serviceAccountTokenCreator)
permission in data source (PenelopeDataExporter)
The <pre><code class="lang-backup"></code></pre> service account should have the following roles in the source project:
- to be able to evaluate data store costs during backup creation
* Monitoring Viewer (roles/monitoring.viewer)
- to be able to access project metadata
* resourcemanager.projects.get
* resourcemanager.projects.list
- to be able to list and get GCS bucket metadata
* storage.buckets.get
* storage.buckets.list
- to be able to make GCS Mirroring
* Private Logs Viewer (roles/logging.privateLogViewer)
- to be able to for GCS Mirroring strategy to check logging.list quota usage
* serviceusage.services.list
- to be able to list and get BigQuery datasets/table/views metadata
* bigquery.datasets.get
* bigquery.tables.get
* bigquery.tables.list
- to be able to list and export from BigQuery
* bigquery.tables.createSnapshot
* bigquery.tables.export
* bigquery.tables.getData
* bigquery.tables.replicateData
permission in data sink (PenelopeBackupManager)
The <pre><code class="lang-backup"></code></pre> service account should have the following roles in the target (backup only) project:
- to be able to manage GCS buckets and its objects
* storage.buckets.create
* storage.buckets.delete
* storage.buckets.enableObjectRetention
* storage.buckets.get
* storage.buckets.getIamPolicy
* storage.buckets.list
* storage.buckets.setIamPolicy
* storage.buckets.update
* storage.objects.create
* storage.objects.delete
* storage.objects.get
* storage.objects.list
* storage.objects.update
- to be able to trigger export jobs in BigQuery from source project(s)
* BigQuery Job User (roles/bigquery.jobUser)
- to be able to create&update Storage Transfer jobs
* Storage Transfer User (roles/storagetransfer.user)
- to be able to clean up backups that transit to status
BackupDeleted
* storagetransfer.jobs.delete
* storagetransfer.jobs.get
* storagetransfer.jobs.list
* bigquery.jobs.delete
* bigquery.jobs.get
* bigquery.jobs.list
- to be able to check compliance permissions
* iam.denypolicies.list
* iam.denypolicies.get
Storage Transfer Service Account
For the backup that are made for the bucket in the source project the service account - created by the Storage Transfer Service, is used. The service agent's email uses the format <pre><code class="lang-project">-SINKPROJECTNUMBER@storage-transfer-service.iam.gserviceaccount.com</code></pre>.
permission in data source
Google's managed service account need following permission in the source project:
- to be able to export GCS date from source
* on project level permissions that are part of following GCP roles
* Storage Object Viewer (<pre><code class="lang-roles">/storage.objectViewer</code></pre>)
* Storage Legacy Bucket Reader (<pre><code class="lang-roles">/storage.legacyBucketReader</code></pre>)
* NOTE: this role can be set only on the bucket level you need to define custom role
permission in data sink
Google's managed service account need following permission in the target (backup only) project:
- on project level a role wit the following IAM permissions:
* Storage Transfer User (roles/storagetransfer.user`)
- on sink bucket level:
/storage.legacyBucketReader/storage.legacyBucketWriterBackup settings
There are two types of backups supported by Penelope:
- Cloud Storage
- BigQuery
Backup Settings for Time To Life/Move (TTL/TTM)
There are three settings that affect stored backup data:
Mirror TTL
- Deletes objects in the backup bucket that are older than the configured TTL.
- added to meet regulatory requirements for data retention limitations in some industries
Snapshot TTL
- Deletes objects in the backup bucket and tables/partition metadata that are older than the configured TTL.
- Added to meet regulatory requirements for data retention limitations in some industries and for oneshot backups.
Archive Transition (Archive TTM)
- Changes the storage class of objects in the backup bucket to Archive after the configured TTM.
- Added to reduce storage costs for long-term backups.
Cloud Storage
Source of the backup is a Cloud Storage Bucket. Backups are performed using GCP's Storage Transfer Service. Data are stored in a GCS bucket.
| Feature | Oneshot | Snapshot | Mirroring | |-------------------------|-----------------------------------------|------------------------------------------|------------------------------------------| | Backup Frequency | once | every X hours, min=1 | every hour | | Retention Period | configurable, default=0 (forever) | configurable, default=0 (forever) | configurable, default=0 (forever) | | Updated files in source | N/A | Files are replaced | Files are replaced | | Deleted files in source | N/A | Files are kept in backup Bucket | Files are kept in backup Bucket | | Trashcan support | no | no | yes | | Object versioning | N/A, default to GCP organization policy | N/A, default to GCP organization policy | N/A, default to GCP organization policy |
Trashcan for Cloud Storage
This feature is used when the backup strategy is set to "Mirroring".
In the target bucket a "trashcan" folder is created. When a data object is deleted in the source bucket, then the object is moved to the "trashcan" folder in the backup bucket. When in the source project same file with the same path is re-created then existing file in the trashcan is replaced. The object remains in the trashcan for a fours weeks - after the retention period the object is permanently deleted from the trashcan.
Cloud Storage limitations
Penelope is deployed in Google App Engine Standard Environment. After years of testing we found out that the following limitations exists:
- up to 100 backups can be handled concurrently every 5 minutes
- up to 15,000 objects deleted objets in source can be moved to trashcan per minute
BigQuery
Source of the backup is a BigQuery Dataset or specific Table or specific **Table partition **. Backups are performed using BigQuery's Extract Jobs. Data are stored in a GCS bucket in AVRO format.
| Feature | Oneshot | Snapshot | Mirroring | |-----------------------------------|-----------------------------------------|------------------------------------------------------------|--------------------------------------------------------------| | Backup Frequency | once | every X hours, min=1 | every hour | | Retention Period | configurable, default=0 (forever) | configurable, default=0 (forever) | configurable, default=0 (forever) | | Updated table/partition in source | N/A | Full copy of table/partition is always made to a new path. | Only changed table/partition will be copied to backup bucket | | Deleted table/partition | N/A | Existing copy are kept. | Existing copy is kept for four weeks. | | Trashcan support | no | no | no | | Object versioning | N/A, default to GCP organization policy | N/A, default to GCP organization policy | N/A, default to GCP organization policy |
BigQuery limitations
Penelope is deployed in Google App Engine Standard Environment. After years of testing we found out that the following limitations exists:
- up to 1800 new BigQuery export jobs (for each Table/Partition) can be started every 5 minutes
- up to 100,000 new BigQuery export jobs can be handled per project per day - this is hard quota from Google Cloud Platform
- up to 15,000 objects deleted objets in source can be deleted from trashcan per minute
