Pipelines for the GitLab project

Pipelines for https://gitlab.com/gitlab-org/gitlab and https://gitlab.com/gitlab-org/gitlab-foss (as well as the dev instance's mirrors) are configured in the usual .gitlab-ci.yml which itself includes files under .gitlab/ci/ for easier maintenance.

We're striving to dogfood GitLab CI/CD features and best-practices as much as possible.

Stages

The current stages are:

Default image

The default image is currently registry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.5-golang-1.12-git-2.24-lfs-2.9-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33.

It includes Ruby 2.6.5, Go 1.12, Git 2.24, Git LFS 2.9, Chrome 73, Node 12, Yarn 1.16, PostgreSQL 9.6, and Graphics Magick 1.3.33.

The images used in our pipelines are configured in the gitlab-org/gitlab-build-images project, which is push-mirrored to https://dev.gitlab.org/gitlab/gitlab-build-images for redundancy.

The current version of the build images can be found in the "Used by GitLab section".

Default variables

In addition to the predefined variables, each pipeline includes default variables defined in https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml.

Common job definitions

Most of the jobs extend from a few CI definitions that are scoped to a single configuration parameter.

These common definitions are:

Changes detection

If a job extends from .default-only (and most of the jobs should), it can restrict the cases where it should be created based on the changes from a commit or MR by extending from the following CI definitions:

See https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml for the list of exact patterns.

Rules conditions and changes patterns

We're making use of the rules keyword but we're currently duplicating the if conditions and changes patterns lists since they cannot be shared across included files as we do with extends.

If you update an if condition or changes patterns list, make sure to mass-update those across all the CI config files (i.e. .gitlab/ci/*.yml).

Canonical commits only

This condition limits jobs creation to commits under the gitlab-org/ top-level group on GitLab.com only. This is similar to the .only:variables-canonical-dot-com CI definition:

.if-canonical-gitlab: &if-canonical-gitlab
  if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/'

Canonical merge requests only

Same as the "Canonical commits only" condition above but further limits jobs creation to merge requests only (i.e. this won't run for master, stable or auto-deploy branches). This is similar to the .only:variables-canonical-dot-com + .except:refs-master-tags-stable-deploy CI definitions:

.if-canonical-gitlab-merge-request: &if-canonical-gitlab-merge-request
  if: '$CI_SERVER_HOST == "gitlab.com" && $CI_PROJECT_NAMESPACE =~ /^gitlab-org($|\/)/ && $CI_MERGE_REQUEST_IID'

Code changes patterns

Similar patterns as for .only:changes-code:

.code-patterns: &code-patterns
  - ...

QA changes patterns

Similar patterns as for .only:changes-qa:

.qa-patterns: &qa-patterns
  - ...

Code and QA changes patterns

Similar patterns as for .only:changes-code-qa:

.code-qa-patterns: &code-qa-patterns
  - ...

Directed acyclic graph

We're using the needs: keyword to execute jobs out of order for the following jobs:

graph RL;
  A[setup-test-env];
  B["gitlab:assets:compile pull-push-cache<br/>(master only)"];
  C[gitlab:assets:compile pull-cache];
  D["cache gems<br/>(master and tags only)"];
  E[review-build-cng];
  F[build-qa-image];
  G[review-deploy];
  G2["schedule:review-deploy<br/>(master only)"];
  H[karma];
  I[jest];
  J["compile-assets pull-push-cache<br/>(master only)"];
  K[compile-assets pull-cache];
  L[webpack-dev-server];
  M[coverage];
  N[pages];
  O[static-analysis];
  Q[package-and-qa];
  S["RSpec<br/>(e.g. rspec unit pg9)"]
  T[retrieve-tests-metadata];

subgraph "`prepare` stage"
    A
    B
    C
    F
    K
    J
    T
    end

subgraph "`test` stage"
    D --> |needs| A;
    H -.-> |needs and depends on| A;
    H -.-> |needs and depends on| K;
    I -.-> |needs and depends on| A;
    I -.-> |needs and depends on| K;
    L -.-> |needs and depends on| A;
    L -.-> |needs and depends on| K;
    O -.-> |needs and depends on| A;
    O -.-> |needs and depends on| K;
    S -.-> |needs and depends on| A;
    S -.-> |needs and depends on| K;
    S -.-> |needs and depends on| T;
    downtime_check --> |needs and depends on| A;
    db:* --> |needs| A;
    gitlab:setup --> |needs| A;
    downtime_check --> |needs and depends on| A;
    graphql-docs-verify --> |needs| A;
    end

subgraph "`review-prepare` stage"
    E --> |needs| C;
    X["schedule:review-build-cng<br/>(master schedule only)"] --> |needs| C;
    end

subgraph "`review` stage"
    G
    G2
    end

subgraph "`qa` stage"
    Q --> |needs| C;
    Q --> |needs| F;
    review-qa-smoke -.-> |needs and depends on| G;
    review-qa-all -.-> |needs and depends on| G;
    review-performance -.-> |needs and depends on| G;
    X2["schedule:review-performance<br/>(master only)"] -.-> |needs and depends on| G2;
    dast -.-> |needs and depends on| G;
    end

subgraph "`post-test` stage"
    M
    end

subgraph "`pages` stage"
    N -.-> |depends on| C;
    N -.-> |depends on| H;
    N -.-> |depends on| M;
    end

Test jobs

Consult GitLab tests in the Continuous Integration (CI) context for more information.

Review app jobs

Consult the Review Apps dedicated page for more information.

Return to Development documentation