Aller au contenu

Concourse - Pipeline Q&A#

How to configure the Git resource for git.corp.caascad.com?#

We have created users ci-infra-stg and ci-infra-prd on Gitlab for the project caascad (the user is inherited in all projets in this group). You can add these users on your personal projects in Gitlab if you need it.

To use git.corp.caascad.com from your pipeline, configure the git resource like this:

resources:
  - name: terraform-envs
    type: git
    icon: git
    source:
      uri: "((git-corp.url))/caascad/repo/example"
      username: "((git-corp.username))"
      password: "((git-corp.password))"

Documentation of this resource: git-resource.

How to do a if statement?#

You can't but you can implement it using the same tips as for the loop in Concourse.

How to do a loop in a pipeline?#

You can't but here is some tips to allow you to implement loops with Concourse.

Tip n°1: put everything in the same task and loop in your task.

Depending on your case, this can be the right solution or a very bad one. This is also a solution when you have to do a if statement which doesn't exists in Concourse.

Tip n°2: use a template pipeline and instanciate the same template with different variables.

With the set_pipeline, you can re-use the same template several times:

- set_pipeline: concourse-oci-build-task
  file: concourse-images-def/template-build-git.yml
  vars:
    name: oci-build-task
    git_repo: https://github.com/vito/oci-build-task.git

- set_pipeline: concourse-gitlab-merge-request-resource
  file: concourse-images-def/template-build-git.yml
  vars:
    name: gitlab-merge-request-resource
    git_repo: https://github.com/samcontesse/gitlab-merge-request-resource.git

Tip n°3: use Cue (or another external tool but cue is better)

If you need to do more advanced pipelines, you will need a better tool to allow you to generate pipelines. We recommand to use Cue since most of our pipeline generation use it.

With Cue, we can generate pipelines from caascad-zones easily. You can see an example with our postgres-backup pipeline.

How to increase the memory limit on a task?#

By default, we have configured a memory limit for all tasks ran by a worker. You can find this limit by using the following command:

kubectl --namespace concourse-infra get deployments.apps -o yaml | grep -A 1 'TASK_MEMORY_LIMIT'

You can configure the memory limit in your pipeline by adding the container_limits.memory attribute:

- task: some-task
  config:
    container_limits:
      memory: 1073741824 # = 1GB

How to use a private docker image in my task?#

- task: hello-world-task
  config:
    platform: linux
    image_resource:
      type: registry-image
      source:
        repository: "((docker-registry-dev.url))/mga-test-concourse"
        username: "((docker-registry-dev.username))"
        password: "((docker-registry-dev.password))"
        tag: latest
    run:
      path: ash
      args:
        - -c
        - echo "Hello world"

How to use the docker resource with a private image?#

For images from the caascad docker registry:

resources:
  - name: my-image
    type: registry-image
    icon: docker
    source:
      repository: "((docker-registry-dev.url))/mga-test-concourse"
      username: "((docker-registry-dev.username))"
      password: "((docker-registry-dev.password))"
      tag: latest

For images from the docker hub:

resources:
  - name: my-image
    type: registry-image
    icon: docker
    source:
      repository: "((docker-registry-internal.url))/alpine"
      username: "((docker-registry-internal.username))"
      password: "((docker-registry-internal.password))"
      tag: 3.13.1

How to build a Dockerfile and push the image?#

Use the task vito/oci-build-task to build your Dockerfile and the push it using the builtin resource registry-image.

resources:
  - name: my-image
    type: registry-image
    icon: docker
    source:
      repository: "((docker-registry-dev.url))/alpine"
      username: "((docker-registry-dev.username))"
      password: "((docker-registry-dev.password))"
      tag: 3.13.1

jobs:
  - name: build
    serial: true
    plan:
      # Fetch your git repo (you need to add if in the resources section)
      - get: git

      # Create a file with the credentials used to authenticate to the docker registry
      # This step is mandatory only if you need to fetch an image from a private registry
      - task: generate-docker-creds
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: "((docker-registry-dev.url))/alpine"
              username: "((docker-registry-dev.username))"
              password: "((docker-registry-dev.password))"
              tag: 3.13.1
          outputs:
            - name: docker_creds
          run:
            path: ash
            args:
              - -c
              # Use the secret based on the base(s) image(s) of your dockerfile.
              # If you need to use the Docker Hub, use `docker-registry-dev.dockerconfig`.
              - echo '((docker-registry-dev.dockerconfig))' > docker_creds/config.json

      # Build the docker image using the credentials we previously forged
      - task: build-docker-image
        privileged: true
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: "((docker-registry-internal.url))/oci-build-task"
              username: "((docker-registry-internal.username))"
              password: "((docker-registry-internal.password))"
              tag: v0.7.0
          caches:
            - path: cache
          inputs:
            - name: git
            - name: docker_creds
          outputs:
            - name: image
          params:
            CONTEXT: git
            DOCKER_CONFIG: docker_creds
          run:
            path: build

      - put: image
        params:
          image: image/image.tar

How to run a job periodically?#

Use the cron-resource. Documentation.

How to use the semver resource?#

The semver resource can be used if you want to manage the versioning of your project with Concourse. This resource need to use git or S3 to store the version number.

How to use:

  • Create a branch in your repository (branch = version in this sample)
  • Initiate a file called version in this branch, containing the following string of chars: 0.0.0 (this will be your start version)
  • Add the version resource in your pipeline (notice that the private-key is used to push the modification in the version branch, in the file version)
resources:
  - name: version
    type: semver
    icon: tag
    source:
      driver: git
      uri: "((git-corp.url))/caascad/my-project.git"
      username: "((git-corp.username))"
      password: "((git-corp.password))"
      branch: version
      file: version

You can create a versioning management, with 3 jobs, using the version resource, with a different bump parameter. This will add 3 buttons on the Web UI so you can choose what new version you want to create.

jobs:
  - name: create-version-major
    serial: true
    plan:
      - put: version
        params: {bump: major}
  - name: create-version-minor
    serial: true
    plan:
      - put: version
        params: {bump: minor}
  - name: create-version-patch
    serial: true
    plan:
      - put: version
        params: {bump: patch}