Aller au contenu

Concourse - Getting Started#

Install the CLI#

The Concourse CLI command is named fly.

To install the fly command:

toolbox install fly

The toolbox does not expose the binary itself. A wrapper is in charged to select the right version of fly and a second script will a some features to the fly command.

Login on Concourse with the CLI#

When you're logging to Concourse, you will have to choose two things: the cluster you want to use and the team.

There are two kinds of concourse instances:

  • ci.<env>.caascad.com => for our clients
  • ci-infra.<env>.caascad.com => for us

This part uses the command fly switch to connect to Concourse. This command has been added in the wrapper and it's not part of the official fly binary. If you want to use the client's CI, add the -c flag to the command.

$ fly switch -z infra-stg -n deploy [-c]
Asking for changing zone...
Current target is infra-stg.infra using team deploy

This command configure your ~/.flyrc and setup the ~/.fly-current-target file. The next time you enter a fly command that need authentication, Concourse will ask you to log in.

For example:

$ fly login
could not find a valid token.
logging in to team 'deploy'

navigate to the following URL in your browser:

https://ci-infra.infra-stg.caascad.com/login?fly_port=40615

You simply need to click on the URL and then log in with Keycloak. At the end of the process, the command should return the result of the command.

Use the login command to connect to concourse. You will have to provide these parameters:

  • -t <target> set a friendly name for your configuration, you can see this as an alias. Examples of targets : infra-stg or ocb-test01. First examples below use target infra-stg.
  • --team-name <team> concourse is deployed with team configured. Your team can be: deploy, build, test, cron.
  • --concourse-url <url> which is the URL I mentioned earlier.

Example (using infra-stg as the <target>):

$ # Change deploy in the line below and set your team
$ # (probably one of deploy, build, test or cron)
$ fly -t infra-stg login --team-name build --concourse-url https://ci-infra.infra-stg.caascad.com
logging in to team 'build'
navigate to the following URL in your browser:
https://ci-infra.infra-stg.caascad.com/login?fly_port=45743
or enter token manually:

You simply need to click on the URL and then log in with Keycloak. At the end of the process, the command should return the result of the command.

Once you are connected, you can check your teams and roles with the subcommand userinfo:

$ fly userinfo
username   team/role
yourname   build/member,build/viewer,cron/member,cron/viewer,deploy/member,deploy/viewer,main/owner,test/member,test/viewer

Introduction to pipelines#

Resources#

Concourse use "resources" to represent external inputs and outputs of jobs. Example of resources: git repository, docker images, a S3 bucket...

Each resources has a type and Concourse knows how to deal with a limited set of basic resource types. You can add third party resources types or even develop your own.

Check out this website for a list of official and third party resource types: Resource Types

Jobs#

A pipeline is compound of one or several jobs. The documentation define jobs as:

Jobs determine the actions of your pipeline. They determine how resources progress through it, and how the pipeline is visualized.

Jobs are described using a plan. The plan explains which resources are needed and what to do with it. You can have a plan that fetch a git repository (input resource), build the Dockerfile located in this repository and finally, push the image into a registry (output resource).

In a pipeline, all jobs are independent. You can link your resources by passing resources between them.

For example, a pipeline can have the following three jobs:

  • Test: fetch the repository, execute the go testing suite
  • Build: fetch the repository, build the Dockerfile
  • Deploy: fetch the repository, get the name of the docker image, execute a helm command with the name of the new image

As you can see, the resource "repository" is used in all three jobs. If you want to trigger the jobs sequentially, you have to "pass" the repository resource through the three jobs using the keyword "passed" (documentation).

Pipelines#

Pipelines defines the resources and jobs that will work together. Pipelines are defined in YAML and are set using the command fly set-pipeline. The pipeline belongs to a TEAM.

Variables can be used to configure pipelines. Your pipeline use the variables by substituting the following pattern ((myvariable)). There is no templating language in Concourse, just variable substitution (like your favorite Kubernetes files).

The variables can be defined using two ways: - The variable is defined during the execution of the set-pipeline command and the substitution is done by the fly command. The value of this variables will be visible by anybody that have a read access to the Concourse API. - The variable is not defined when the set-pipeline is executed, the variable will not be substituted by fly but by Concourse itself just before the pipeline is executed by the worker. There is different back-end for storing these secret and we have chose Vault for Caascad.

Info

  • The maximum number of active build tasks is set for concourse workers, so when this value is reached the other tasks will be in state waiting
  • Tasks have default memory limited, so task exceeded the fixed value will be killed by kernel.
  • If a task needs more memory than the default set you have to add container_limits in the pipeline itself.
  • To check these values in zone: kubectl --namespace concourse-infra get deployments.apps -o yaml | grep -E -A 1 'TASKS_PER_WORKER|TASK_MEMORY_LIMIT'

Naming convention#

Warning

Do not use the same pipeline name on the same repository for both infra-stg and infra-prd. Gitlab uses it as an ID and needs it to be unique. Else, the display of the pipeline status in merge requests will be broken.

The name of these Concourse identifiers will have some limits in the names:

  • Team names
  • Pipeline names
  • Job names
  • Step names
  • Var source names

The names in our pipelines should be limited to theses characters:

  • Unicode letters in lower case.
  • Decimal numbers.
  • Hyphens (-), as the canonical word separator.
  • Periods (.), in order to support domain names and version numbers.

The name should start with a lower case letter.

The regex to validate names is : [a-z][a-z0-9.-]*

How to use secrets?#

You shouldn't store passwords, keys or any secrets values in your pipeline. The variable you pass when you do a set-pipeline command are not a good place neither because these variables are stored in the pipeline and anybody will be able to see those as long as they can access Concourse. The solution is to store your secrets in the secret back-end of Concourse.

Write secrets#

Adding secrets is done by going on Vault web interface or using the vault command.

First, you have to find the right path in Vault. Most of your secrets would go here:

  • secret/concourse-infra/<pipeline-team>/<secret-name>
  • secret/concourse-infra/<pipeline-team>/<pipeline-name>/<secret-name>

The first case would set the secret for every pipeline in the team. The second case define the secret only for the pipeline named <pipeline-name>.

With your web browser go to https://vault.<zone>.caascad.com.

Choose OIDC in Method.

Click on Sign in with OIDC provider. This will open a popup window to keycloak. Make sure to allow popup windows in your browser. If the popup window was blocked the first time, allow it and restart the process.

Once logged in you can navigate in the secret/ directory and add secrets in the correct location. Beware, in Vault UI, the secret path should not start by "secret/" and should follow this pattern: concourse-infra/<team>/<secret-name>.

First login in vault with:

$ export VAULT_ADDR=https://vault.<zone>.caascad.com
$ vault login -method oidc

Then to write some secret:

$ vault write secret/concourse-infra/infra/my-secret foo=bar
Success! Data written to: secret/concourse-infra/infra/my-secret

To check the contents of a secret:

$ vault read secret/concourse-infra/infra/my-secret
Key                 Value
---                 -----
refresh_interval    768h
foo                 bar

Read secrets#

Concourse is integrated with Vault and you won't have to use any external client to read secrets from your pipelines. You just have to use the same syntax as the variables.

Example: my pipeline belongs to the team monitoring and I want to use a secret named "demo-team". A Vault secret contains one or several values referenced by a key. This secret can be viewed as a JSON object:

{
    "bar": "Hello",
    "foo": "World"
}

To read this secret in my pipeline, I can simply do:

echo ((demo-team.foo)) ((demo-team.bar))

Concourse have a feature that allow you to call your secret like this ((demo-team)) without specifying a key name. You can do this by creating a secret with only one entry. The key name of this entry should be "value". You cannot have a secret with different entries and a key with the name "value". It would cause this error:

failed to interpolate task config: Finding variable 'demo-team.value':
    cannot access field 'value' of non-map value ('string') from var: demo-team.value

Shared secrets#

Secrets shown in the previous section are scoped to a team. With Concourse, we can create shared secret that would be available in all pipelines in all teams. These secrets can only be written by a concourse admin.

Use case: we can share by default credentials to access to Kubernetes, push images in to docker registry or pull git repositories.

These secrets are setup in Vault under the path concourse-infra/global/*. You can access it the same way as mentioned before.

List of infra zone global secrets :

Secret name Keys Description
git-corp url
username
password
token		 Allow you to connect to Git corp.
https://git.corp.caascad.com
global-docker-registry
Deprecated		 url
username
password		 Secret for pushing docker images in the "infra" registry.
docker-registry.infra-stg.caascad.com/caascad
global-docker-registry-dev
Deprecated		 url
username
password		 Secret for pushing docker images in the "infra" registry.
docker-registry.infra-stg.caascad.com/caascad-dev
docker-registry-dev
docker-registry-external
docker-registry-internal		 url
username
password
dockerconfig		 Secret for pushing docker images in our global docker registry
docker-registry.caascad.com.
The suffixes(dev, external, internal) allow you to push/pull
images in one of the organization.
docker-registry.ocb-corp.caascad.com/external
dockerconfig: contains the full docker/config.json
that allows you to push images easily with Concourse.
dockerhub username
password
dockerconfig		 Secret to access the dockerhub without restrictions.
dockerconfig: contains the full docker/config.json
that allows you to push images easily with Concourse.
terraform role_id
secret_id		 The role_id/secret_id are used to login into Vault
and fetch secrets to configure the cloud provider backends.
kubernetes-$ZONE_NAME
Examples:
kubernetes-infra-stg
kubernetes-test01
kubernetes-charlie		 url
token
kubeconfig		 There is one secret for each following zones:
- current infra zone
- all zones cloud managed by this infra zone
- all its associated client zones
The connection to the kubernetes cluster is done using Rancher.

List of cloud zone global secrets for Caascad:

Secret name Keys Description
kubernetes-$ZONE_NAME url
token		 There is one secret for each following zones:
- current cloud zone
- all its associated client zones.

List of cloud zone global secrets for Client:

Secret name Keys Description
caascad-kubernetes-$ZONE_NAME url
token		 There is one secret for each client zones associated with
this cloud zone.

Interpolation in secret name#

Concourse allows interpolation inside the name of the secret. For example, you have two secrets named kubernetes-ocb-corp and kubernetes-ocb-test01 in Vault. You also have two pipelines, one to deploy on ocb-corp and another to deploy on ocb-test01 but you want to use the same template for the two pipelines. With Concourse, you can set a variable in the secret name. Here is an example on how to do this to configure a Kubernetes resource:

- name: kubernetes-cluster
  type: kubernetes
  source:
    server: ((kubernetes-((zone)).server))
    token: ((kubernetes-((zone)).token))
    certificate_authority: ((kubernetes-((zone)).cacert))

The variable zone can be set when the pipeline is saved in Concourse:

$ fly set-pipeline -p demo -c my-template-pipeline.yaml -v zone=ocb-corp

Documentation: Vault credential manager

Images for tasks and resources#

We have imported, in our docker registry, some images of premade tasks and resources.

NOTE: It is important to use the images of the internal registry.

Image Repo/docs
docker-registry.caascad.com/internal/concourse-sonarqube-resource GitHub
docker-registry.caascad.com/internal/gitlab-merge-request-resource GitHub
docker-registry.caascad.com/internal/keyval-resource GitHub
docker-registry.caascad.com/internal/oci-build-task GitHub

For using this images, simply configure the source section of your task or resource like this:

source:
  repository: "((docker-registry-internal.url))/gitlab-merge-request-resource"
  username: "((docker-registry-internal.username))"
  password: "((docker-registry-internal.password))"
  tag: v0.1.0

All these images are built using a pipeline, code source in this repo.

More info#

Articles and projects#

Writing pipelines

Create custom resource

Example and real world pipelines#

From Caascad team#

Community#