a Concourse task for building OCI images
oci-build task
A Concourse task for building OCI images. Currently uses buildkit for building.
A stretch goal of this is to support running without privileged: true, though it currently still requires it.
imageresource
* params
* inputs
* outputs
* caches
* run
usage
The task implementation is available as an image on Docker Hub at concourse/oci-build-task. (This image is built from Dockerfile using the oci-build task itself.)
This task implementation started as a spike to explore patterns around reusable tasks to hopefully lead to a proper RFC. Until that RFC is written and implemented, configuration is still done by way of providing your own task config as follows:
image_resource
First, your task needs to point to the oci-build-task image:
image_resource:
type: registry-image
source:
repository: concourse/oci-build-task
params
Any of the following optional parameters may be specified. These are all exposed as environment variables to the task, therefore only string values are allowed. This is a pain point with re-usable tasks that will ideally be resolved by prototypes.
CONTEXT(default.): the path to the directory to provide as the context
DOCKERFILE(default$CONTEXT/Dockerfile): the path to theDockerfile
BUILDKIT_SSHyour ssh key location that is mounted in yourDockerfile. This is
For Example. In your Dockerfile, you can mount a key as
RUN --mount=type=ssh,id=githubsshkey pip install -U -r ./hats/requirements-test.txt
Then in your Concourse YAML configuration:
params: BUILDKITSSH: githubssh_key=<PATH-TO-YOUR-KEY>
Read more about ssh mount here.
BUILDARG: params prefixed with BUILDARG will be provided as build args. For example BUILDARGfoo=bar, will set the foo build arg as bar.
BUILDARGSFILE(default empty): path to a file containing build args. By
foo=bar, one per
line. Empty lines are skipped. If the file ends in yml or yaml it will
be parsed as a YAML file. The YAML file can only contain string keys and
values.
Example simple file contents:
EMAIL=me@example.com
HOWMANYTHINGS=1
DO_THING=false
Example YAML file contents:
EMAIL: me@example.com
HOWMANYTHINGS: "1"
DO_THING: "false"
MULTILINEARG: |
thing1
thing2
BUILDKITSECRET: files with extra secrets which are made available via --mount=type=secret,id=.... See New Docker Build secret information for more information on build secrets.
For example, running with BUILDKITSECRETconfig=my-repo/config will allow you to do the following...
RUN --mount=type=secret,id=config cat /run/secrets/config
BUILDKITSECRETTEXT: literal text of extra secrets to be made available via the same mechanism described for $BUILDKITSECRET* above. The difference is that this is easier to use with credential managers:
BUILDKITSECRETTEXTmysecret=(( mysecret )) puts the content that (( mysecret )) expands to in /run/secrets/mysecret.
IMAGEARG: params prefixed with IMAGEARG* point to image tarballs (i.e. docker save format) or path to images in OCI layout format, to preload so that they do not have to be fetched during the build. An image reference will be provided as the given build arg name. For example, IMAGEARGbaseimage=ubuntu/image.tar will set baseimage to a local image reference for using ubuntu/image.tar.
This must be accepted as an argument for use; for example:
ARG base_image
FROM ${base_image}
IMAGE_PLATFORM: Specify the target platform(s) to build the image for. For
IMAGE_PLATFORM=linux/arm64,linux/amd64 will build the image for the
Linux OS and architectures arm64 and amd64. By default, images will be
built for the current worker's platform that the task is running on. If
multiple platforms are specified, OUTPUT_OCI will be set to true
automatically, resulting in the output being a directory instead of a tarball.
LABEL: params prefixed with LABEL will be set as image labels. For example LABEL_foo=bar, will set the foo label to bar.
LABELS_FILE(default empty): path to a file containing labels in
foo=bar, one per line. Empty lines are skipped.
TARGET(default empty): a target build stage to build, as named with the
FROM … AS <NAME> syntax in your Dockerfile.
TARGET_FILE(default empty): path to a file containing the name of the
ADDITIONAL_TARGETS(default empty): a comma-separated (,) list of
REGISTRY_MIRRORS(default empty): a comma-separated (,) list of registry
docker.io. If you need to specify authentication details
then consider using BUILDKITEXTRACONFIG instead.
UNPACK_ROOTFS(defaultfalse): unpack the image as Concourse's image
rootfs/, metadata.json) for use with the image task step
option.
OUTPUT_OCI(defaultfalse): outputs an OCI image, allowing for multi-arch
IMAGEPLATFORM to multiple
platforms.
The image output will be a directory (image/image) in OCI Image
Layout format when this flag is set to true.
BUILDKITADDHOSTS(default empty): extra host definitions forbuildkit
,) list of key-value pairs (using syntax hostname=ip-address), each
defining an IP address for resolving some custom hostname.
BUILDKITEXTRACONFIG(default empty): a string written verbatim to builkit's
inputs
There are no required inputs - your task should just list each artifact it needs as an input. Typically this is in close correlation with $CONTEXT:
params:
CONTEXT: my-image
inputs:
- name: my-image
Should your build be dependent on multiple inputs, you may want to leave CONTEXT as its default (.) and set an explicit path to the DOCKERFILE:
params:
DOCKERFILE: my-repo/Dockerfile
inputs:
- name: my-repo
- name: some-dependency
It might also make sense to place one input under another, like so:
params:
CONTEXT: my-repo
inputs:
- name: my-repo
- name: some-dependency
path: my-repo/some-dependency
Or, to fully rely on the default behavior and use path to wire up the context accordingly, you could set your primary context as path: . and set up any additional inputs underneath:
inputs:
- name: my-repo
path: .
- name: some-dependency
outputs
A single output named image may be configured:
outputs:
- name: image
Use [output_mapping] to map this output to a different name in your build plan.
This approach should be used if you're building multiple images in parallel so that
they can have distinct names.
[outputmapping]: https://concourse-ci.org/jobs.html#schema.step.task-step.outputmapping
The output will contain the following files:
image.tar: the OCI image tarball. This tarball can be uploaded to a registry
OUTPUT_OCI is false, which is the default.
image/: a directory containing the OCI image(s) in OCI Image Layout format.
OUTPUT_OCI is true.
digest: the digest of the OCI config. This file can be used to tag the
docker load, like so:
docker load -i image/image.tar
docker tag $(cat image/digest) my-name
If $UNPACK_ROOTFS is configured, the following additional entries will be created:
rootfs/: the unpacked contents of the image's filesystem.
metadata.json: a JSON file containing the image's env and user
This is a Concourse-specific format to support using the newly built image for a subsequent task by pointing the task step's image option to the output, like so:
plan:
- task: build-image
params:
UNPACK_ROOTFS: true
output_mapping: {image: my-built-image}
- task: use-image
image: my-built-image
(The output_mapping here is just for clarity; alternatively you could just set image: image.)
Note: at some point Concourse will likely standardize on OCI instead.
caches
Caching can be enabled by caching the cache path on the task:
caches:
- path: cache
This only caches the build layers that Buildkit makes and will only be hit if
the same worker is used between one build and the next.
NOTE: the contents of --mount=type=cache directories are not cached, see https://github.com/concourse/oci-build-task/issues/87
run
Your task should run the build executable:
run:
path: build
migrating from the docker-image resource
The docker-image resource was previously used for building and pushing a Docker image to a registry in one fell swoop.
The oci-build task, in contrast, only supports building images - it does not support pushing or even tagging the image. It can be used to build an image and use it for a subsequent task image without pushing it to a registry, by configuring $UNPACK_ROOTFS.
In order to push the newly built image, you can use a resource like the registry-image resource like so:
resources:
- name: my-image-src
type: git
source:
uri: https://github.com/...
- name: my-image
type: registry-image
source:
repository: my-user/my-repo
jobs:
- name: build-and-push
plan: # fetch repository source (containing Dockerfile) - get: my-image-src
# build using oci-build task # # note: this task config could be pushed into my-image-src and loaded using # file: instead - task: build privileged: true config: platform: linux
image_resource: type: registry-image source: repository: concourse/oci-build-task
inputs: - name: my-image-src path: .
outputs: - name: image
run: path: build
# push using registry-image resource - put: my-image params: {image: image/image.tar}
example
This repo contains an example.yml, which builds the image for the task itself:
fly -t dev execute -c example.yml -i context=. -o image=. -p
docker load -i image.tar
That -p at the end is not a typo; it runs the task with elevated privileges.
Providing Custom CA Certificates
Assuming your custom CA cert is passed in as an input to the oci-build task, you can use a task config like this to load your custom CA certificate:
platform: linux
inputs:
- name: certs
path: /var/certs #Absolute path only works on Concourse >=7.5.0 #..other inputs
outputs:
- name: image
params: BUILDKITEXTRACONFIG: | [registry."my-registry.com"] ca=["/var/certs/my-ca.pem"]
run: path: build