Configuring a Monitoring node for Scaling and High Availability

Introduced in GitLab 12.0.

You can configure a Prometheus node to monitor GitLab.

Standalone Monitoring node using GitLab Omnibus

The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana. The monitoring node is not highly available. See Scaling and High Availability for an overview of GitLab scaling and high availability options.

The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus:

  1. SSH into the Monitoring node.
  2. Download/install the Omnibus GitLab package you want using steps 1 and 2 from the GitLab downloads page.

  3. Do not complete any other steps on the download page.

  4. Make sure to collect CONSUL_SERVER_NODES, which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z

  5. Edit /etc/gitlab/gitlab.rb and add the contents:

```ruby external_url 'http://gitlab.example.com'

# Enable Prometheus prometheus['enable'] = true prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false

# Enable Login form grafana['disable_login_form'] = false

# Enable Grafana grafana['enable'] = true grafana['admin_password'] = 'toomanysecrets'

# Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true

# Replace placeholders # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z # with the addresses of the Consul server nodes consul['configuration'] = { retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), }

# Disable all other services gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false gitlab_workhorse['enable'] = false nginx['enable'] = true postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false redis_exporter['enable'] = false sidekiq['enable'] = false unicorn['enable'] = false node_exporter['enable'] = false gitlab_exporter['enable'] = false ```

  1. Run sudo gitlab-ctl reconfigure to compile the configuration.

Migrating to Service Discovery

Once monitoring using Service Discovery is enabled with consul['monitoring_service_discovery'] = true, ensure that prometheus['scrape_configs'] is not set in /etc/gitlab/gitlab.rb. Setting both consul['monitoring_service_discovery'] = true and prometheus['scrape_configs'] in /etc/gitlab/gitlab.rb will result in errors.