Working with Prometheus Metrics

Adding to the library

We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made to the common_metrics.yml file.

Query identifier

The requirement for adding a new metric is to make each query to have an unique identifier which is used to update the metric later when changed:

- group: Response metrics (NGINX Ingress)
  metrics:
  - title: "Throughput"
    y_label: "Requests / Sec"
    queries:
    - id: response_metrics_nginx_ingress_throughput_status_code
      query_range: 'sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)'
      unit: req / sec
      label: Status Code

Update existing metrics

After you add or change an existing common metric, you must re-run the import script that will query and update all existing metrics.

Or, you can create a database migration:

NOTE: Note: If a query metric (which is identified by id:) is removed it will not be removed from database by default. You might want to add additional database migration that makes a decision what to do with removed one. For example: you might be interested in migrating all dependent data to a different metric.

class ImportCommonMetrics < ActiveRecord::Migration[4.2]
  include Gitlab::Database::MigrationHelpers

  DOWNTIME = false

  def up
    ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute
  end

  def down
    # no-op
  end
end

GitLab Prometheus metrics

GitLab provides Prometheus metrics to monitor itself.

Adding a new metric

This section describes how to add new metrics for self-monitoring (example).

  1. Select the type of metric:

  2. Gitlab::Metrics.counter

  3. Gitlab::Metrics.gauge
  4. Gitlab::Metrics.histogram

  5. Gitlab::Metrics.summary

  6. Select the appropriate name for your metric. Refer to the guidelines for Prometheus metric names.

  7. Update the list of GitLab Prometheus metrics.
  8. Trigger the relevant page/code that will record the new metric.
  9. Check that the new metric appears at /-/metrics.