Hub/Prometheus

Technology

Prometheus

Collect Prometheus metrics using OpenTelemetry collector

Overview

Collect Prometheus Metrics with OpenTelemetry Collector

The OpenTelemetry Collector can be configured to scrape Prometheus metrics, effectively acting as a drop-in replacement for the Prometheus server. This integration allows for a seamless transition from Prometheus to OpenTelemetry, enabling users to leverage both systems' strengths.

Key Components

  • Prometheus Receiver: This component in the OpenTelemetry Collector is designed to scrape metrics from Prometheus targets. It supports the full set of Prometheus scraping configurations, including service discovery and relabeling.
  • Target Allocator: An optional component introduced by the OpenTelemetry Operator to simplify Prometheus receiver configuration. It ensures even distribution of Prometheus targets among multiple Collector instances and facilitates the discovery of Prometheus Custom Resources.

Advantages

  • Flexibility: Allows gradual migration from Prometheus to OpenTelemetry.
  • Scalability: With the Target Allocator, it's possible to distribute scraping load across multiple Collector instances.
  • Interoperability: Enables the use of Prometheus metrics within the OpenTelemetry ecosystem and Dash0.

Setup

Overview

Many Kubernetes services and pods expose metrics through a Prometheus endpoint that you can scrape. We recommend to scrape and forward these metrics via an OpenTelemetry Collector.

Prometheus metrics endpoint

To make your services scrapeable, add these annotations to your pod or service definitions

plaintext
123
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
prometheus.io/port: '8888'

Collector Configuration

The following code snippet shows an OpenTelemetry Collector configuration file.

yaml
123456789101112131516171820212224252627293031323335363739404143444547484951525455575859606263656667686970727374757677
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "kubernetes-pods"
          kubernetes_sd_configs:
            - role: pod
          scrape_interval: 30s
          scrape_timeout: 5s
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: false
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: true
              action: keep
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: false
              action: drop
            - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_prometheus_io_port]
              regex: (.+);(.+)
              replacement: $1:$2
              target_label: __address__
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              regex: ()
              replacement: /metrics
              target_label: __meta_kubernetes_pod_annotation_prometheus_io_path
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: pod_name
            - source_labels: [__meta_kubernetes_pod_container_name]
              action: replace
              target_label: container_name
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - action: labeldrop
              regex: "container_id|image_id|controller_revision_hash|pod_template_generation|pod_template_hash"
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "process_virtual.*|process_max_fds|process_start_time.*|http_.*|go_gc.*"
              action: drop
processors:
  batch:
exporters:
  otlp/dash0:
    endpoint: "{{endpoint_otlp_grpc}}"
    headers:
      Authorization: "Bearer {{token}}"
      "Dash0-Dataset": "{{dataset}}"
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp/dash0]

Collector Deployment

Learn how to deploy the collector within our OpenTelemetry collector integration documentation.

References