convey

Convey is a pipeline for running containers and protecting the host as much as possible.

It does this by creating a volume and making it available to each item in the pipeline.

License

GPLv3+

Concepts

Convey is build around a concept that I call a build container. A build container is a container that has all of the dependencies to turn an input into something else. This could be source code or even just data processing. For example the convey/go-build image is a simple wrapper around the golang image that just builds a project instead of installing it.

Convey is built around this concept to allow you to build/process anything on any platform while at the same time protecting the host from a malicious config. This protection is provided by not exposing dangerous options like volume mounts or port forwards while running containers. This is accomplished by creating a volume a data volume at the start of the build plan and volume mounting it into containers as they are run. This allows you to pass state between modular tasks that can be reused between multiple plans.

Once built, convey can build itself entirely. This is done by using an import task to copy the source code into the workspace. Once that stage is complete, it runs another stage that builds for linux-x86_64, windows-x86_64, and darwin-x86_64 in parallel. The final stage will take those compiled binaries and export them back to the hosts file system.

Of course you don't have to export back to the host file system. Since these are just containers, you could instead upload them to your staging environment, packagecloud.io, bintray.com, whatever. Of you could export to the host and create another plan that will import the artifacts and then publish them.

Usage

usage: convey [<flags>] [<plan>...]

container runner

Flags:
      --help                   Show context-sensitive help (also try --help-long and --help-man).
      --version                Show application version.
      --color                  Enable colorized output
  -l, --config-loader=convey   Select the configuration loader
  -f, --config=CONFIG          The config file name to use
  -c, --cpu-shares=CPU-SHARES  The amount of cpu shares to give to a run task
  -e, --env=ENV ...            Set environment variables
  -S, --force-sequential       Don't run anything concurrently
  -g, --graphviz               Output a graphviz diagram of the config file
  -P, --list-plans             List the plans that are available
  -M, --list-meta-plans        List the meta plans that are available
  -L, --list-tasks             List the supported tasks
  -m, --memory=MEMORY          The ammount of memor to give the run task
      --ssh-agent              Enabled ssh-agent support for tasks
      --timeout=15m            The maximum amount of time a task can run in seconds. 0 to disable
  -v, --verbose                Be more verbose

Args:
  [<plan>]  The plan or list of plans to run in specified order

Examples

There are a handful of examples in the examples/ directory.

always.yml

This example shows how the always stage attribute works and how it can be used to run a stage to report a build status, run clean up, or whatever.

always-simple.yml

A much easier to follow version of having stages always run.

bitbucket.yml

This example shows how to use the convey/bitbucket-upload image to upload files to a Bitbucket repository's downloads section.

default-plan-alias.yml

A simple plan that aliases the default plan.

environment.yml

This example shows how environment substitution works in your config file.

meta-plans.yml

This example shows how you can use meta-plans to group multiple plans together.

override.yml

This example shows how environment substitution overriding works and uses a -override.yml file as well.

script.yml

This example shows how to use the script attribute of a run task.

ssh-identities.yml

This example shows how you can specify an ssh identity to automatically turn on ssh-agent forwarding.

traditional.yml

This example shows your traditional clone, test, build, deploy, and report for a project.

Configuration

Configuration is done via a file named convey.yaml. This file defines the tasks as well as the plans. There are few top level-items: options, environment, meta-plans, plans, and tasks.

Options

The options section let's you fine tune a few things about your configuration.

default-plan

default-plan is the name of the plan that should be ran by default. When not set, this defaults to default.

ssh-identities

You can specify a list of SSH key fingerprints that are required for your run. This is done by using the ssh-identities attribute which is a list of SSH key fingerprints with an option checksum prefix. You can also specify * to allow any key that's been added to the ssh-agent.

ssh-identities are also supported in your override file. However, any ssh-identities specified in your convey.yml will be overridden by the values in the override file; they will not be merged.

NOTE If running on docker4mac, there is an (issue)[https://github.com/docker/for-mac/issues/483] with xhyve (the hypervisor that the Docker daemon runs in) where you can't volume mount the ssh-agent socket. You can use this (workaround)[https://hub.docker.com/r/uber/ssh-agent-forward/] to expose the ssh-agent via an alternative method.

Examples

options:
  ssh-identities:
    - ivZaDYamb5xdguIUUVT7DXvXwvE9JLsOkOkR3XAfzeI
    - SHA256:Efrocgd+rvwjDAnHt2jZAcwDqeka0s8Vv7N3m08cVnA

When using * you have to quote it because * treated specially in yaml.

options:
  ssh-identities:
    - '*'

Tasks

The tasks section defines each task. By defining them separately from the plan the tasks can be reused in multiple plans.

There are many tasks types that you can use by setting the type attribute. However if the type attribute is not set, it default to run.

Build task

A build task will build a docker image.

Attributes

Example

build-image:
  type: build
  dockerfile: Dockerfile
  tag: myimage:latest

Clean task

A clean task will remove files from the host relative and limited to the working directory of convey.

Attributes

Example

clean:
  type: clean
  files:
    - convey-amd64
    - "**/*.pyc"
    - "**/__pycache__"

Export tasks

An export task copies files from the volume to the host.

Attributes

Example

export:
  type: export
  files:
    - logo.png
    - binary:binary-linux-x86_64

Import tasks

An import task copies files from the host to the volume. It has one required attribute named sources. This is a list of files relative to the directory that convey was run from that will be copied into the volume.

Attributes

Example

import:
  type: import
  files:
    - Dockerfile
    - src:source

Login task

A login task will run docker login to login to a registry.

Attributes

Example

registry-login:
  type: login
  username: superuser1
  password: abc123

Logout task

A logout task will log you out from a Docker registry.

Attributes

Example

registry-logout:
  type: logout
  server: registry.my.domain:5000

Pull tasks

A pull task pulls an image.

Attributes

At least one image must be supplied by either the image or images attribute. If both are supplied, image is inserted to the front of images.

Example

pull-alpine:
  type: pull
  image: gliderlabs/alpine:edge

Push tasks

A push task pushes an image.

Attributes

At least one image must be supplied by either the image or images attribute. If both are supplied, image is inserted to the front of images.

Example

push-image:
  type: push
  images:
    - registry.my.domain:5000/newimage:master-latest
    - registry.my.domain:5000/newimage:master-deadbeef

Remove tasks

A remove task removes an image.

Attributes

At least one image must be supplied by either the image or images attribute. If both are supplied, image is inserted to the front of images.

Example

remove-image:
  type: remove
  images: 
    - registry.my.domain:5000/newimage:master-latest
    - registry.my.domain:5000/newimage:master-deadbeef

Run tasks

A run task runs an image.

Attributes

Example

A basic example where the image knows everything to do.

build-golang:
  type: run
  image: golang:onbuild

A basic example using a standard image to do something else

download-file:
  type: run
  image: debian:jessie-slim
  script:
    - wget https://somedomain.tld/file1
    - wget https://somedomain.tld/file1.sha256
    - sha256sum -c file1.sha256

Tag Tasks

A tag task will tag an existing image.

Attributes

Example

tag-development:
  type: tag
  source: registry.my.domain:5000/newimage:latest
  destination: registry.my.domain:5000/newimage:development

Environment

The environment section defines a list of environment variables to set for the run.

It should be specified in a NAME or NAME=VALUE format. If no value is provided, the value of the variable from the host will be provided if it is available.

Example

environment:
  - BUILD=development
  - SECRET_TOKEN

Plans

The plans sections defines your execution plans which tie everything together by organizing tasks into stages into the order that they will be executed.

Each plan is defined by a name in the plans section.

Example

plans:
  plan1:
    ...
  plan2:
    ...

Plan

Plans themselves are pretty straight forward. They group stages with an optional environment.

Attributes

Stages

Stages group tasks together into a logical unit. They can be run sequentially or concurrently.

Attributes

Meta Plans

Meta plans are used to group plans together and run them in serial. This is also possible via the command line by specifying multiple plans to run.

There is currently no support for nesting meta plans.

Example

meta-plans:
  world:
    plans:
      - plan1
      - plan2
      - plan3

Meta Plan

Meta plans contain a single attribute which is the list of plans to run.

Attributes

Config Loaders

Convey supports multiple different types of configs that it can load. Currently supported loaders are convey and bitbucket.\

Convey

The convey config is the one described above and is the default.

Bitbucket

The bitbucket config is for bitbucket-pipelines which will allow you to run your pipeline locally. Convey does a best attempt at emulating the pipelines environment, but your mileage may vary. If you're having problems, please open an issue.

To use the bitbucket config loader, just run convey -l bitbucket. If there is a bitbucket-pipelines.yml file in the working directory, convey will happily run it.

bitbucket pipelines does not have plans in the same sense that convey does, so if you do not specify one, convey will use the same heuristics that bitbucket-pipelines does to run the correct pipeline from the config.

Also, since the bitbucket-pipelines.yml file is converted into the convey config format, --graphviz, --list-plans, and --list-tasks all work like normal as well.