Dash0 Operator for Kubernetes

The Dash0 operator installs an OpenTelemetry collector into your cluster that sends data to your Dash0 ingress endpoint, with authentication already configured out of the box. Additionally, it will enable gathering OpenTelemetry data from applications deployed to the cluster for a selection of supported runtimes, plus automatic log collection and metrics.

Supported Runtimes

Supported runtimes for automatic workload instrumentation:

• Java 8+
• Node.js 16+
• .NET

Other features like metrics and log collection are independent of the runtime of workloads.

Prerequisites

• Kubernetes 1.25.16 or later.
• Helm >= 3.x, please refer to Helm's documentation for more information on installing Helm.

To use the operator, you will need provide two configuration values:

• endpointCopy: The URL of the Dash0 ingress endpoint backend to which telemetry data will be sent. This property is mandatory when installing the operator. This is the OTLP/gRPC endpoint of your Dash0 organization. The correct OTLP/gRPC endpoint can be copied fom https://app.dash0.com -> organization settings -> "Endpoints" -> "OTLP/gRPC". Note that the correct endpoint value will always start with ingress.Copy and end in dash0.com:4317Copy. Including a protocol prefix (e.g. https://Copy) is optional.
• Either tokenCopy or secretRefCopy: Exactly one of these two properties needs to be provided when installing the operator.
  • tokenCopy: This is the Dash0 authorization token of your organization. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". The prefix Bearer Copy must not be included in the value. Note that when you provide a token, it will be rendered verbatim into a Kubernetes ConfigMap object. Anyone with API access to the Kubernetes cluster will be able to read the value. Use a secret reference and a Kubernetes secret if you want to avoid that.
  • secretRefCopy: A reference to an existing Kubernetes secret in the Dash0 operator's namespace. The secret needs to contain the Dash0 authorization token. See below for details on how exactly the secret should be created and configured.

Installation

Before installing the operator, add the Dash0 operator's Helm repository as follows:

console
Copy
12
helm repo add dash0-operator https://dash0hq.github.io/dash0-operator
helm repo update dash0-operator

Now you can install the operator into your cluster via Helm with the following command:

console
Copy
12345678910
helm install \
  --namespace dash0-system \
  --create-namespace \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.apiEndpoint=REPLACE THIS WITH YOUR DASH0 API ENDPOINT \
  --set operator.dash0Export.token=REPLACE THIS WITH YOUR DASH0 AUTH TOKEN \
  --set operator.clusterName=REPLACE THIS WITH YOUR THE NAME OF THE CLUSTER (OPTIONAL) \
  dash0-operator \
  dash0-operator/dash0-operator

Instead of providing the auth token directly, you can also use a secret reference:

console
Copy
1234567891011
helm install \
  --namespace dash0-system \
  --create-namespace \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.apiEndpoint=REPLACE THIS WITH YOUR DASH0 API ENDPOINT \
  --set operator.dash0Export.secretRef.name=REPLACE THIS WITH THE NAME OF AN EXISTING KUBERNETES SECRET \
  --set operator.dash0Export.secretRef.key=REPLACE THIS WITH THE PROPERTY KEY IN THAT SECRET \
  --set operator.clusterName=REPLACE THIS WITH YOUR THE NAME OF THE CLUSTER (OPTIONAL) \
  dash0-operator \
  dash0-operator/dash0-operator

See the section Using a Kubernetes Secret for the Dash0 Authorization Token for more information on using a Kubernetes secrets with the Dash0 operator.

You can consult the chart's values.yaml file for a complete list of available configuration settings.

See the section Notes on Creating the Operator Configuration Resource Via Helm for more information on providing Dash0 export settings via Helm and how it affects manual changes to the operator configuration resource.

Last but not least, you can also install the operator without providing a Dash0 backend configuration:

console
Copy
12345
helm install \
  --namespace dash0-system \
  --create-namespace \
  dash0-operator \
  dash0-operator/dash0-operator

However, you will need to create a Dash0 operator configuration resource later that provides the backend connection settings. That is, providing --set operator.dash0Export.enabled=trueCopy and the other backend-related settings when running helm installCopy is simply a shortcut to deploy the Dash0 operator configuration resource automatically at startup.

On its own, the operator will only collect Kubernetes metrics. To actually have the operator properly monitor your workloads, two more things need to be set up:

1. a Dash0 backend connection has to be configured (unless you did that already with the Helm values operator.dash0Export.*Copy), and
2. monitoring namespaces and their workloads to collect logs, traces and metrics has to be enabled per namespace.

Both steps are described in the following sections.

Configuration

Configuring the Dash0 Backend Connection

You can skip this step if you provided --set operator.dash0Export.enabled=trueCopy together with the endpoint and either a token or a secret reference when running helm installCopy. In that case, proceed to the next section, Enable Dash0 Monitoring For a Namespace.

Otherwise, configure the backend connection now by creating a file dash0-operator-configuration.yamlCopy with the following content:

yaml
Copy
1234567891011121314151617
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      # Replace this value with the actual OTLP/gRPC endpoint of your Dash0 organization.
      endpoint: ingress... # TODO needs to be replaced with the actual value, see below

      authorization:
        # Provide the Dash0 authorization token as a string via the token property:
        token: auth_... # TODO needs to be replaced with the actual value, see below

      apiEndpoint: https://api.....dash0.com # TODO needs to be replaced with the actual value, see below

  clusterName: my-kubernetes-cluster # optional, see below

Here is a list of configuration options for this resource:

• spec.export.dash0.endpointCopy: The URL of the Dash0 ingress endpoint to which telemetry data will be sent. This property is mandatory. Replace the value in the example above with the OTLP/gRPC endpoint of your Dash0 organization. The correct OTLP/gRPC endpoint can be copied fom https://app.dash0.com -> organization settings -> "Endpoints" -> "OTLP/gRPC". Note that the correct endpoint value will always start with ingress.Copy and end in dash0.com:4317Copy. Including a protocol prefix (e.g. https://Copy) is optional.
• spec.export.dash0.authorization.tokenCopy or spec.export.dash0.authorization.secretRefCopy: Exactly one of these two properties needs to be provided. Providing both will cause a validation error when installing the Dash0Monitoring resource.
  • spec.export.dash0.authorization.tokenCopy: Replace the value in the example above with the Dash0 authorization token of your organization. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". The prefix Bearer Copy must not be included in the value. Note that the value will be rendered verbatim into a Kubernetes ConfigMap object. Anyone with API access to the Kubernetes cluster will be able to read the value. Use the secretRefCopy property and a Kubernetes secret if you want to avoid that.
  • spec.export.dash0.authorization.secretRefCopy: A reference to an existing Kubernetes secret in the Dash0 operator's namespace. See the section Using a Kubernetes Secret for the Dash0 Authorization Token for an example file that uses a secretRefCopy. The secret needs to contain the Dash0 authorization token. See below for details on how exactly the secret should be created and configured. Note that by default, Kubernetes secrets are stored unencrypted, and anyone with API access to the Kubernetes cluster will be able to read the value. Additional steps are required to make sure secret values are encrypted. See https://kubernetes.io/docs/concepts/configuration/secret/ for more information on Kubernetes secrets.
• spec.export.dash0.apiEndpointCopy: The base URL of the Dash0 API to talk to. This is not where telemetry will be sent, but it is used for managing dashboards, check rules, synthetic checks and views via the operator. This property is optional. The value needs to be the API endpoint of your Dash0 organization. The correct API endpoint can be copied fom https://app.dash0.com -> organization settings -> "Endpoints" -> "API". The correct endpoint value will always start with "https://api." and end in ".dash0.com". If this property is omitted, managing dashboards, check rules, synthetic checks and views via the operator will not work.
• spec.selfMonitoring.enabledCopy: An opt-out for self-monitoring for the operator. If enabled, the operator will collect self-monitoring telemetry and send it to the configured Dash0 backend. This setting is optional, it defaults to true.
• spec.kubernetesInfrastructureMetricsCollection.enabledCopy: If enabled, the operator will collect Kubernetes infrastructure metrics. This setting is optional, it defaults to true; unless telemetryCollection.enabledCopy is set to falseCopy, then kubernetesInfrastructureMetricsCollection.enabledCopy defaults to falseCopy as well. It is a validation error to set telemetryCollection.enabled=falseCopy and kubernetesInfrastructureMetricsCollection.enabled=trueCopy at the same time.
• spec.collectPodLabelsAndAnnotations.enabledCopy: If enabled, the operator will collect all Kubernetes pod labels and annotations and convert them to resource attributes for all spans, log records and metrics. The resulting resource attributes are prefixed with k8s.pod.label.Copy or k8s.pod.annotation.Copy respectively. This setting is optional, it defaults to true; unless telemetryCollection.enabledCopy is set to falseCopy, then collectPodLabelsAndAnnotations.enabledCopy defaults to falseCopy as well. It is a validation error to set telemetryCollection.enabled=falseCopy and collectPodLabelsAndAnnotations.enabled=trueCopy at the same time.
• spec.clusterNameCopy: If set, the value will be added as the resource attribute k8s.cluster.nameCopy to all telemetry. This setting is optional. By default, k8s.cluster.nameCopy will not be added to telemetry.
• spec.telemetryCollection.enabledCopy: An opt-out switch for all telemetry collection, and to avoid having the operator deploy OpenTelemetry collectors in the cluster. This setting is optional, it defaults to true (that is, by default, OpenTelemetry collectors will be deployed and telemetry will be collected). If telemetry collection is disabled via this switch, the operator will not collect any telemetry, in particular it will not deploy any OpenTelemetry collectors in the cluster. This is useful if you want to use the operator for infrastructure-as-code (e.g. to synchronize dashboards & check rules), but do not want it to deploy the OpenTelemetry collector. Note that setting this to false does not disable the operator's self-monitoring telemetry, use the setting spec.selfMonitoring.enabledCopy to disable self-monitoring if required (self-monitoring does not require an OpenTelemetry collector). Also note that this setting is not exposed via Helm, i.e. if you want to set this to false you need to deploy the operator configuration resource manually, i.e. omit the Helm value operator.dash0Export.enabledCopy or set it to false, then deploy an operator configuration resource via kubectl apply -fCopy or similar.

After providing the required values (at least endpointCopy and authorizationCopy), save the file and apply the resource to the Kubernetes cluster you want to monitor:

console
Copy
1
kubectl apply -f dash0-operator-configuration.yaml

The Dash0 operator configuration resource is cluster-scoped, so a specific namespace should not be provided when applying it.

Note: All configuration options available in the operator configuration resource can also be configured when letting the Helm chart auto-create this resource, as explained in the section Installation. You can consult the chart's values.yaml file for a complete list of available configuration settings.

Notes on Creating the Operator Configuration Resource Via Helm

Providing the backend connection settings to the operator via Helm parameters is a convenience mechanism to get monitoring started right away when installing the operator. Setting operator.dash0Export.enabledCopy to trueCopy and providing other necessary operator.dash0Export.*Copy values like operator.dash0Export.endpointCopy will instruct the operator manager to create an operator configuration resource with the provided values at startup. This automatically created operator configuration resource will have the name dash0-operator-configuration-auto-resourceCopy.

If an operator configuration resource with any other name already exists in the cluster (e.g. a manually created operator configuration resource), the operator will treat this as an error and refuse to overwrite the existing operator configuration resource with the values provided via Helm.

If an operator configuration resource with the name dash0-operator-configuration-auto-resourceCopy already exists in the cluster (e.g. a previous startup of the operator manager has created the resource), the operator manager will update/overwrite this resource with the values provided via Helm.

Manual changes to the dash0-operator-configuration-auto-resourceCopy are permissible for quickly experimenting with configuration changes, without an operator restart, but you need to be aware that they will be overwritten with the settings provided via Helm the next time the operator manager pod is restarted. Possible reasons for a restart of the operator manager pod include upgrading to a new operator version, running helm upgrade ... dash0-operator dash0-operator/dash0-operatorCopy, or Kubernetes moving the operator manager pod to a different node.

For this reason, when using this feature, it is recommended to treat the Helm values as the source of truth for the operator configuration. Any changes you want to be permanent should be applied via Helm and the operator.dash0Export.*Copy settings.

If you would rather retain manual control over the operator configuration resource, you should omit any operator.dash0Export.*Copy Helm values and create and manage the operator configuration resource manually (that is, via kubectl, ArgoCD etc.).

Enable Dash0 Monitoring For a Namespace

Note: By default, when enabling Dash0 monitoring for a namespace, all workloads in this namespace will be restarted to apply the Dash0 instrumentation. If you want to avoid this, set the instrumentWorkloadsCopy property in the monitoring resource spec to created-and-updatedCopy. See below for more information on the instrumentWorkloadsCopy modes.

For each namespace that you want to monitor with Dash0, enable monitoring by installing a Dash0 monitoring resource into that namespace:

Create a file dash0-monitoring.yamlCopy with the following content:

yaml
Copy
1234
apiVersion: operator.dash0.com/v1beta1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource

Save the file and apply the resource to the namespace you want to monitor. For example, if you want to monitor workloads in the namespace my-nodejs-applicationsCopy, use the following command:

console
Copy
1
kubectl apply --namespace my-nodejs-applications -f dash0-monitoring.yaml

If you want to monitor the defaultCopy namespace with Dash0, use the following command:

console
Copy
1
kubectl apply -f dash0-monitoring.yaml

Note: Even when no monitoring resources has been installed and no namespace is being monitored by Dash0, the Dash0 operator's collector will collect Kubernetes infrastructure metrics that are not namespace scoped, like node-related metrics. The only prerequisite for this is an operator configuration with export settings.

Additional Configuration Per Namespace

The Dash0 monitoring resource supports additional configuration settings:

• spec.instrumentWorkloads.modeCopy: A namespace-wide configuration for the workload instrumentation strategy for the target namespace. There are three possible settings: allCopy, created-and-updatedCopy and noneCopy. By default, the setting allCopy is assumed; unless there is an operator configuration resource with telemetryCollection.enabled=falseCopy, then the setting noneCopy is assumed by default. Note that spec.instrumentWorkloads.modeCopy is the path for this setting starting with version v1beta1Copy of the Dash0 monitoring resource; when using v1alpha1Copy, the path is spec.instrumentWorkloadsCopy.

  • allCopy: If set to allCopy, the operator will:

    • instrument existing workloads in the target namespace (i.e. workloads already running in the namespace) when the Dash0 monitoring resource is deployed,
    • instrument existing workloads or update the instrumentation of already instrumented workloads in the target namespace when the Dash0 operator is first started or updated to a newer version,,
    • instrument new workloads in the target namespace when they are deployed, and
    • instrument changed workloads in the target namespace when changes are applied to them. Note that the first two actions (instrumenting existing workloads) will result in restarting the pods of the affected workloads. Use created-and-updatedCopy if you want to avoid pod restarts.

  • created-and-updatedCopy: If set to created-and-updatedCopy, the operator will not instrument existing workloads in the target namespace. Instead, it will only:

    • instrument new workloads in the target namespace when they are deployed, and
    • instrument changed workloads in the target namespace when changes are applied to them. This setting is useful if you want to avoid pod restarts as a side effect of deploying the Dash0 monitoring resource or restarting the Dash0 operator.

  • noneCopy: You can opt out of instrumenting workloads entirely by setting this option to noneCopy. With spec.instrumentWorkloads: noneCopy, workloads in the target namespace will never be instrumented to emit telemetry.

  If this setting is omitted, the value allCopy is assumed and new/updated as well as existing Kubernetes workloads will be intrumented by the operator to emit telemetry, as described above. There is one exception to this rule: If there is an operator configuration resource with telemetryCollection.enabled=falseCopy, then the default setting is noneCopy instead of allCopy, and no workloads will be instrumented by the Dash0 operator.

  More fine-grained per-workload control over instrumentation is available by setting the label dash0.com/enable=falseCopy on individual workloads, see Disabling Auto-Instrumentation for Specific Workloads.

  The behavior when changing this setting for an existing Dash0 monitoring resource is as follows:

  • When this setting is updated to spec.instrumentWorkloads=allCopy (and it had a different value before): All existing uninstrumented workloads will be instrumented. Their pods will be restarted to apply the instrumentation.
  • When this setting is updated to spec.instrumentWorkloads=noneCopy (and it had a different value before): The instrumentation will be removed from all instrumented workloads. Their pods will be restarted to remove the instrumentation. (After this change, the operator will no longer instrument any workloads nor will it restart any pods.)
  • Updating this value to spec.instrumentWorkloads=created-and-updatedCopy has no immediate effect; existing uninstrumented workloads will not be instrumented, existing instrumented workloads will not be uninstrumented. Newly deployed or updated workloads will be instrumented from the point of the configuration change onwards as described above.

  Automatic workload instrumentation will automatically add tracing to your workloads. You can read more about what exactly this feature entails in the section Automatic Workload Instrumentation.

• spec.instrumentWorkloads.labelSelectorCopy: A custom Kubernetes label selector for controlling the workload instrumentation on the level of individual workloads, see Using a Custom Label Selector to Control Auto-Instrumentation.

• spec.instrumentWorkloads.traceContext.propagatorsCopy: When set, the operator will add the environment variable OTEL_PROPAGATORSCopy to all instrumented workloads in the target namespace. This environment variable determines which trace context propagation headers an OTel SDK uses. Setting this can be useful if the workloads in this namespace interact with services do not use the W3C trace context standard header traceparentCopy for trace context propagation, but for example AWS X-Ray (X-Amzn-Trace-IdCopy). The value of the setting will be validated, it needs to be a comma-separated list of valid propagators. See https://opentelemetry.io/docs/languages/sdk-configuration/general/#otel_propagators for more information.

  When the option is no set, the operator will not set the environment variable OTEL_PROPAGATORSCopy. If the option is set in the monitoring resource at some point and then later removed again, the operator will remove the environment variable from instrumented workloads if and only if the value of the environment variable matches the previously used setting in the monitoring resource. This is done to prevent accidentally removing an OTEL_PROPAGATORSCopy environment variable that has been set manually and not by the operator. (For that purpose, the previous setting is stored in the monitoring resource's status.)

• spec.logCollection.enabledCopy: A namespace-wide opt-out for collecting pod logs via the filelogCopy receiver. If enabled, the operator will configure its OpenTelemetry collector to watch the log output of all pods in the namespace and send the resulting log records to Dash0. This setting is optional, it defaults to true; unless there is an operator configuration resource with telemetryCollection.enabled=falseCopy, then log collection is off by default. It is a validation error to set telemetryCollection.enabled=falseCopy in the operator configuration resource and logCollection.enabled=trueCopy in any monitoring resource at the same time.

• spec.prometheusScraping.enabledCopy: A namespace-wide opt-out for Prometheus scraping for the target namespace. If enabled, the operator will configure its OpenTelemetry collector to scrape metrics from pods in the namespace of this Dash0Monitoring resource according to their prometheus.io/scrape annotations via the OpenTelemetry Prometheus receiver. This setting is optional, it defaults to true; unless there is an operator configuration resource with telemetryCollection.enabled=falseCopy, then Prometheus scraping is off by default. It is a validation error to set telemetryCollection.enabled=falseCopy in the operator configuration resource and prometheusScraping.enabled=trueCopy in any monitoring resource at the same time. Note that the collection of OpenTelemetry-native metrics is not affected by setting prometheusScraping.enabled=falseCopy for a namespace.

• spec.filterCopy: An optional custom filter configuration to drop some of the collected telemetry before sending it to the configured telemetry backend. Filters for a specific telemetry object type (e.g. spans) are lists of OTTL expressions. If at least one of the one conditions of a list evaluates to true, the object will be dropped. (That is, conditions are implicitly connected by a logical OR.) The configuration structure is identical to the configuration of the OpenTelemetry collector's filter processor. One difference to the filter processor is that the filter rules configured in a Dash0 monitoring resource will only be applied to the telemetry collected in the namespace the monitoring resource is installed in. Telemetry from other namespaces is not affected. Existing configurations for the filter processor can be copied and pasted without syntactical changes.

  • spec.filter.traces.spanCopy: A list of OTTL conditions for filtering spans. All spans where at least one condition evaluates to true will be dropped. (That is, conditions are implicitly connected by a logical OR.)
  • spec.filter.traces.spaneventCopy: A list of OTTL conditions for filtering span events. All span events where at least one condition evaluates to true will be dropped. If all span events for a span are dropped, the span will be left intact.
  • spec.filter.metrics.metricCopy: A list of OTTL conditions for filtering metrics. All metrics where at least one condition evaluates to true will be dropped.
  • spec.filter.metrics.datapointCopy: A list of OTTL conditions for filtering individual data points of metrics. All data points where at least one condition evaluates to true will be dropped. If all datapoints for a metric are dropped, the metric will also be dropped.
  • spec.filter.logs.log_recordCopy: A list of OTTL conditions for filtering log records. All log records where at least one condition evaluates to true will be dropped. This setting is optional, by default, no filters are applied. It is a validation error to set telemetryCollection.enabled=falseCopy in the operator configuration resource and set filters in any monitoring resource at the same time.

• spec.transformCopy: An optional custom transformation configuration that will be applied to the collected telemetry before sending it to the configured telemetry backend. Transformations for a specific telemetry signal (e.g. traces, metrics, logs) are lists of OTTL statements. All telemetry for the respective signal will be routed through all transformation statements. The statements are executed in the order they are listed. The configuration structure is identical to the configuration of the OpenTelemetry collector's transform processor. Both the basic configuration style, and the advanced configuration style of the transform processor are supported. One difference to the transform processor is that the transform rules configured in a Dash0 monitoring resource will only be applied to the telemetry collected in the namespace the monitoring resource is installed in. Telemetry from other namespaces is not affected. If both spec.filterCopy and spec.transformCopy are configured, the filtering for a given signal (traces, metrics, logs) will be executed before the transform processor. (That is, you cannot assume that transformations have already been applied when writing filter rules.) Existing configurations for the transform processor can be copied and pasted without syntactical changes.

  • spec.transform.trace_statementsCopy: A list of OTTL statements (or a list of groups in the advanced config style) for transforming trace telemetry.
  • spec.transform.metric_statementsCopy: A list of OTTL statements (or a list of groups in the advanced config style) for filtering metric telemetry.
  • spec.transform.log_statementsCopy: A list of OTTL statements (or a list of groups in the advanced config style) for filtering log telemetry. This setting is optional, by default, no transformations are applied. It is a validation error to set telemetryCollection.enabled=falseCopy in the operator configuration resource and set transforms in any monitoring resource at the same time.

• spec.synchronizePersesDashboardsCopy: A namespace-wide opt-out for synchronizing Perses dashboard resources found in the target namespace. If enabled, the operator will watch Perses dashboard resources in this namespace and create corresponding dashboards in Dash0 via the Dash0 API. More fine-grained per-resource control over synchronization is available by setting the label dash0.com/enable=falseCopy on individual Perses dashboard resources. See Managing Dash0 Dashboards for details. This setting is optional, it defaults to true.

• spec.synchronizePrometheusRulesCopy: A namespace-wide opt-out for synchronizing Prometheus rule resources found in the target namespace. If enabled, the operator will watch Prometheus rule resources in this namespace and create corresponding check rules in Dash0 via the Dash0 API. More fine-grained per-resource control over synchronization is available by setting the label dash0.com/enable=falseCopy on individual Prometheus rule resources. See Managing Dash0 Check Rules for details. This setting is optional, it defaults to true.

Here is comprehensive example for a monitoring resource which

• sets the instrumentation mode to created-and-updatedCopy,
• disables Prometheus scraping,
• sets a couple of filters for all five telemetry object types,
• applies transformations to limit the length of span attributes, datapoint attributes, and log attributes (with the metric transform using the advanced transform config style),
• disables Perses dashboard synchronization, and
• disables Prometheus rule synchronization.

yaml
Copy
1234567891011121314151617181920212223242526272829303132333435363738394041424344
apiVersion: operator.dash0.com/v1beta1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource
spec:
  instrumentWorkloads: created-and-updated

  prometheusScraping:
    enabled: false

  filter:
    traces:
      span:
      - 'attributes["http.route"] == "/ready"'
      - 'attributes["http.route"] == "/metrics"'
      spanevent:
      - 'attributes["grpc"] == true'
      - 'IsMatch(name, ".*grpc.*")'
    metrics:
      metric:
      - 'name == "k8s.replicaset.available"'
      - 'name == "k8s.replicaset.desired"'
      datapoint:
      - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
      - 'resource.attributes["service.name"] == "my_service_name"'
    logs:
      log_records:
      - 'IsMatch(body, ".*password.*")'
      - 'severity_number < SEVERITY_NUMBER_WARN'

  transform:
    trace_statements:
    - 'truncate_all(span.attributes, 1024)'
    metric_statements:
      - conditions:
        - 'metric.type == METRIC_DATA_TYPE_SUM'
        statements:
        - 'truncate_all(datapoint.attributes, 1024)'
    log_statements:
    - 'truncate_all(log.attributes, 1024)'

  synchronizePersesDashboards: false

  synchronizePrometheusRules: false

Using a Kubernetes Secret for the Dash0 Authorization Token

If you want to provide the Dash0 authorization token via a Kubernetes secret instead of providing the token as a string, create the secret in the namespace where the Dash0 operator is installed. If you followed the guide above, the name of that namespace is dash0-systemCopy. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". You can freely choose the name of the secret and the key of the token within the secret.

Create the secret by using the following command:

console
Copy
1234
kubectl create secret generic \
  dash0-authorization-secret \
  --namespace dash0-system \
  --from-literal=token=auth_...your-token-here...

With this example command, you would create a secret with the name dash0-authorization-secretCopy in the namespace dash0-systemCopy. If you installed (or plan to install) the operator into a different namespace, replace the --namespaceCopy parameter accordingly.

The name of the secret as well as the key of the token value within the secret must be provided when referencing the secret during helm installCopy, or in the YAML file for the Dash0 operator configuration resource (in the secretRefCopy property).

For creating the operator configuration resource with helm installCopy, the command would look like this, assuming the secret has been created as shown above:

console
Copy
123456789
helm install \
  --namespace dash0-system \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.apiEndpoint=REPLACE THIS WITH YOUR DASH0 API ENDPOINT \
  --set operator.dash0Export.secretRef.name=dash0-authorization-secret \
  --set operator.dash0Export.secretRef.key=token \
  dash0-operator \
  dash0-operator/dash0-operator

The Helm chart will validate whether the referenced secret exists and contains the required key. You can disable this check by setting --set operator.dash0Export.disableSecretValidation=trueCopy. This can be useful if you want to render the manifests via the Helm chart without accessing a live Kubernetes cluster.

If you do not want to install the operator configuration resource via helm installCopy but instead deploy it manually, and use a secret reference for the auth token, the following example YAML file would work with the secret created above:

yaml
Copy
123456789101112131415
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # TODO REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT

      authorization:
        secretRef:
          name: dash0-authorization-secret
          key: token

      apiEndpoint: https://api... # optional, see above

When deploying the operator configuration resource via kubectlCopy, the following defaults apply:

• If the nameCopy property is omitted, the name dash0-authorization-secretCopy will be assumed.
• If the keyCopy property is omitted, the key tokenCopy will be assumed.

With these defaults in mind, the secretRefCopy could have also been written as follows:

yaml
Copy
12345678910111213
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # TODO needs to be replaced with the actual value, see above

      authorization:
        secretRef: {}

      apiEndpoint: https://api... # optional, see above

Note: There are no defaults when using --set operator.dash0Export.secretRef.nameCopy and --set operator.dash0Export.secretRef.keyCopy with helm installCopy, so for that approach the values must always be provided explicitly.

Note that by default, Kubernetes secrets are stored unencrypted, and anyone with API access to the Kubernetes cluster will be able to read the value. Additional steps are required to make sure secret values are encrypted, if that is desired. See https://kubernetes.io/docs/concepts/configuration/secret/ for more information on Kubernetes secrets.

Dash0 Dataset Configuration

Use the spec.export.dash0.datasetCopy property to configure the dataset that should be used for the telemetry data. By default, data will be sent to the dataset defaultCopy. Here is an example for a configuration that uses a different Dash0 dataset:

yaml
Copy
123456789101112131415
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # see above

      dataset: my-custom-dataset # This optional setting determines the Dash0 dataset to which telemetry will be sent.

      authorization: # see above
        ...

      apiEndpoint: https://api... # optional, see above

Configure Metrics Collection

By default, the operator collects metrics as follows:

• The operator collects node, pod, container, and volume metrics from the API server on kubelets via the Kubelet Stats Receiver, cluster-level metrics from the Kubernetes API server via the Kubernetes Cluster Receiver, and system metrics from the underlying nodes via the Host Metrics Receiver. Collecting these metrics can be disabled per cluster by setting kubernetesInfrastructureMetricsCollection.enabled: falseCopy in the Dash0 operator configuration resource (or setting the value operator.kubernetesInfrastructureMetricsCollectionEnabledCopy to falseCopy when deploying the operator configuration resource via the Helm chart).
• Namespace-scoped metrics (e.g. metrics related to a workload running in a specific namespace) will only be collected if the namespace is monitored, that is, there is a Dash0 monitoring resource in that namespace.
• The Dash0 operator scrapes Prometheus endpoints on pods annotated with the prometheus.io/*Copy annotations in monitored namespaces, as described in the section Scraping Prometheus endpoints. This can be disabled per namespace by explicitly setting prometheusScraping.enabled: falseCopy in the Dash0 monitoring resource.
• Metrics which are not namespace-scoped (for example node metrics like k8s.node.*Copy or host metrics like system.cpu.utilizationCopy) will always be collected, unless metrics collection is disabled globally for the cluster (kubernetesInfrastructureMetricsCollection.enabled: falseCopy, see above). An operator configuration resource with export settings has to be present in the cluster, otherwise no metrics collection takes place.

Disabling or enabling individual metrics via configuration is not supported.

Resource Attributes for Prometheus Scraping

When the operator scrapes Prometheus endpoints on pods, it does not have access to all the same metadata that is available to the OpenTelemetry SDK in an instrumented application. For that reason, resource attributes including the service name might be different. The operator makes an effort to derive reasonable resource attributes.

The service name is derived as follows:

1. If the scraped service provides the target_infoCopy metric with a service_nameCopy attribute, that service name will be used. See https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#resource-attributes
2. If no service name was found via (1.), but the pod has the app.kubernetes.io/nameCopy label, the value of that label will be used as the service name. If the service name is derived from this pod label, the following pod labels (if present) will be mapped to resource attributes as well:
   • app.kubernetes.io/versionCopy to service.versionCopy
   • app.kubernetes.io/part_ofCopy to service.namespaceCopy
3. If no service name was found via (1.) or (2.), no service name is set for the Prometheus metrics. If there is other telemetry (tracing, logs, OpenTelemetry metrics) for the same pod in Dash0, and these other signals carry a service name, the Prometheus metrics for this pod will be associated with that service name as well. This is actually the recommendation for handling Prometheus metrics for most users: If you do not have specific reasons to set the service name for Prometheus metrics, the best option is usually to not use the target_infoCopy metric or the app.kubernetes.io/nameCopy pod label, but let the Dash0 backend aggregate all telemetry into one service (to see everything in one place), with the service name taken from other signals than Prometheus metrics.

Note that in contrast to Resource Attributes for Workloads via Labels and Annotations, Prometheus scraping can only see pod labels, not workload level (deployment, daemonset, ...) labels.

Providing a Filelog Offset Volume

The operator's collector uses the filelog receiver to read pod log files for monitored workloads. When the collector is restarted (which can happen for various reasons, for example to apply configuration changes), it is important that the filelog receiver can continue reading the log files from where it left off. If the filelog receiver started to read all log files from the beginning again after a restart, log records would be duplicated, that is, they would appear multiple times in Dash0. For that purpose, the filelog receiver stores the log file offsets in persistent storage. By default, the offsets are stored in a config map in the operator's namespace. For small- to medium-sized clusters, this is usually sufficient, and it requires no additional configuration by users. For larger clusters or clusters with many short-lived pods, we recommend providing a persistent volume for storing offsets.

Any persistent volume that is accessible from the collector pods can be used for this purpose.

Here is an example with a hostPathCopy volume (see also https://kubernetes.io/docs/concepts/storage/volumes/#hostpath for considerations around using hostPathCopy volumes):

yaml
Copy
1234567
operator:
  collectors:
   filelogOffsetSyncStorageVolume:
     name: offset-storage
     hostPath:
       path: /data/dash0-operator/offset-storage
       type: DirectoryOrCreate

The directory in the hostPath volume will automatically be created with the correct permissions so that the OpenTelemetry collector container can write to it.

Here is another example based on persistent volume claims. (This assumes that a PersistentVolumeClaim named offset-storage-claim existsCopy.) See also https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims and https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming.

yaml
Copy
123456
operator:
  collectors:
    filelogOffsetSyncStorageVolume:
      name: offset-storage
      persistentVolumeClaim:
        claimName: offset-storage-claim

Important: Since this volume is needed by a Daemonset run by the Dash0 operator, the PersistentVolumeClaim needs to be set with the ReadWriteManyCopy access mode.

Using a volume instead of the default config map approach is also helpful if you have webhooks in your cluster which process every config map update. One known example is the Kyverno admission controller. When operating Kyverno in the same cluster as the Dash0 operator, it is highly recommended to either use a volume for filelog offsets, or to exclude ConfigMaps (or all resource types) in the Dash0 operator's namespace from Kyverno's processing. Leaving Kyverno processing in place and using the config map filelog offset storage can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause Kyverno to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the Kyverno admission controller.

Using cert-manager

When installing the Helm chart, it generates TLS certificates on the fly for all components that need certificates (the operator's webhook service and its metrics service). This also happens when updating the operator, e.g. via helm upgradeCopy. This is the default behavior, and it works out of the box without the need to manage certificates with a third-party solution.

As an alternative, you can use cert-managerCopy to manage the certificates. To let cert-managerCopy handle TLS certificates, provide the following additional settings when applying the Helm chart:

yaml
Copy
1234567891011121314151617181920212223242526272829303132333435
operator:

  # Settings for using cert-manager instead of auto-generating TLS certificates.
  certManager:

    # This disables the usage of automatically generated certificates by the Helm chart.
    # If this is set to true, the assumption is that certificates are managed by
    # cert-manager (see https://cert-manager.io/), and the other settings shown in this
    # snippet (certManager.secretName, certManager.certManagerAnnotations) become
    # required settings. It is recommended to set webhookService.name as well.
    useCertManager: true

    # The name of the secret used by cert-manager for the certificate.
    # The provided name must match the `secretName` in the `cert-manager.io/v1.Certificate`
    # resource's spec (see below).
    # Note: This secret is created and managed by cert-manager, you do not need to create
    # it manually.
    secretName: dash0-operator-certificate-secret

    # A map of additional annotations that are added to all Kubernetes resources that need
    # the certificate.
    # Usually this will be a single `cert-manager.io/inject-ca-from` annotation with the
    # namespace and name of the Certificate resource (see below), but other annotations
    # are possible, see https://cert-manager.io/docs/reference/annotations/.
    certManagerAnnotations:
      cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"

  webhookService:
    # A name override for the webhook service, defaults to dash0-operator-webhook-service.
    # The name of the webhook service must match the DNS names provided in the
    # cert-manager's Certificate resource.
    # For that reason it is recommended to set this value when using cert-manager, as it
    # guarantees that the names match, even if the Dash0 operator helm chart would change
    # the default name of the webhook service in a future release.
    name: dash0-operator-webhook-service-name

You will also need to provide cert-managerCopy resources, for example a CertificateCopy and an IssuerCopy. Explaining all configuration options of cert-managerCopy is out of scope for this documentation. Please refer to the cert-manager documentation for details on how to install and configure cert-managerCopy in your cluster.

The following annotated example is a minimal configuration that matches the Dash0 operator configuration snippet shown above. Both configuration snippets assume that the Dash0 operator is installed in the default dash0-systemCopy namespace, and the Certificate and Issuer resource are also deployed into that namespace.

yaml
Copy
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:

  # NOTE: This value, together with the namespace this resource is deployed into, must
  # match the value of the cert-manager.io/inject-ca-from annotation provided in the
  # certManager.certManagerAnnotations setting, i.e. this matches the following annotation:
  #   cert-manager.io/inject-ca-from: "dash0-system/dash0-operator-serving-certificate"
  name: dash0-operator-serving-certificate

  labels:
    app.kubernetes.io/name: certificate
    app.kubernetes.io/instance: serving-cert
    app.kubernetes.io/component: certificate

spec:

  # NOTE: Provide all DNS names that are used to access the webhook service.
  # The service names usually follow the patterns <service-name>.<namespace>.svc and
  # <service-name>.<namespace>.svc.cluster.local.
  # If you use a custom name for the Dash0 operator's webhook service (recommended), the
  # first part of the DNS names must match that name.
  # If you use the default name for the webhook service, the first part of the DNS names
  # must match the default name (dash0-operator-webhook-service).
  # If the Dash0 operator is installed into a different namespace, you need to change the
  # ".dash0-system." part of the DNS names accordingly.
  dnsNames:
  - dash0-operator-webhook-service-name.dash0-system.svc
  - dash0-operator-webhook-service-name.dash0-system.svc.cluster.local

  issuerRef:
    kind: Issuer

    # NOTE: Must match the name of the Issuer resource provided below.
    name: dash0-operator-selfsigned-issuer

  # NOTE: The secret name must match the operator.certManager.secretName setting above
  # provided to the Dash0 operator Helm chart. This secret is created and managed by
  # cert-manager, you do not need to create it manually.
  secretName: dash0-operator-certificate-secret

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:

  # NOTE: This issuer's name must match the issuerRef.name in the Certificate resource above.
  name: dash0-operator-selfsigned-issuer

  labels:
    app.kubernetes.io/name: certificate
    app.kubernetes.io/instance: serving-cert
    app.kubernetes.io/component: certificate
spec:

  # There are many options for configuring an Issuer, this example uses a simple self-signed
  # issuer. Make sure to configure the issuer according to your requirements.
  selfSigned: {}

Controlling On Which Nodes the Operator's Collector Pods Are Scheduled

Allow Scheduling on Tainted Nodes

The operator uses a Kubernetes daemonset to deploy the OpenTelemetry collector on each node; to collect telemetry from that node and workloads running on that node. If you use taints on certain nodes, Kubernetes will not schedule any pods there, preventing the daemonset collector pods to be present on these nodes. You can allow the daemonset collector pods to be scheduled there by configuring tolerations matching your taints for the collector pods. Tolerations can be configured as follows:

operator:
  collectors:
    daemonSetTolerations:
    - key: key1
      operator: Equal
      value: value1
      effect: NoSchedule
    - key: key2
      operator: Exists
      effect: NoSchedule
Copy

Changing this setting while the operator is already running requires a helm upgradeCopy/helm upgrade --reuse-valuesCopy or similar to take effect.

Note: The tolerations will be added to the daemonset collector pods, but not to the deployment collector pod.

Preventing Operator Scheduling on Specific Nodes

All the pods deployed by the operator have a default node anti-affinity for the dash0.com/enable=falseCopy node label. That is, if you add the dash0.com/enable=falseCopy label to a node, none of the pods owned by the operator will be scheduled on that node.

IMPORTANT: This includes the daemonset that the operator will set up to receive telemetry from the pods, which might leads to situations in which instrumented pods cannot send telemetry because the local node does not have a daemonset collector pod. In other words, if you want to monitor workloads with the Dash0 operator and use the dash0.com/enable=falseCopy node anti-affinity, make sure that the workloads you want to monitor have the same anti-affinity:

yaml
Copy
12345678910
# Add this to your workloads
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "dash0.com/enable"
            operator: "NotIn"
            values: ["false"]

Disabling Auto-Instrumentation for Specific Workloads

In namespaces that are Dash0-monitoring enabled, all workloads are automatically instrumented for tracing and to improve OpenTelemetry resource attributes. This process will modify the Pod spec, e.g. by adding environment variables, Kubernetes labels and an init container. The modifications are described in detail in the section Automatic Workload Instrumentation.

You can disable these workload modifications for specific workloads by setting the label dash0.com/enable: "false"Copy in the top level metadata section of the workload specification.

Note: The actual label selector for enabling or disabling workload modification can be customized in the Dash0 monitoring resource. The label dash0.com/enable: "false"Copy can be used when no custom label selector has been configured in the Dash0 monitoring resource, see Using a Custom Label Selector to Control Auto-Instrumentation.

Here is an example for a deployment with this label:

yaml
Copy
1234567891011121314151617181920
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: my-deployment-app
    dash0.com/enable: "false"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-deployment-app
  template:
    metadata:
      labels:
        app: my-deployment-app
    spec:
      containers:
        - name: my-deployment-app
          image: "some-image:latest"

The label can also be applied by using kubectlCopy:

kubectl label --namespace $YOUR_NAMESPACE --overwrite deployment $YOUR_DEPLOYMENT_NAME dash0.com/enable=false
Copy

Note that setting dash0.com/enable: "false"Copy will not prevent log collection for pods of workloads with that label, in namespaces that have Dash0 log collection enabled. Also, log collection is enabled by default for all monitored namespaces, unless spec.logCollection.enabledCopy has been set to falseCopy explicitly in the respective Dash0 monitoring resource. Controlling log collection for individual workloads via Kubernetes labels is not supported. To disable log collection for specific workloads in namespaces where log collection is enabled, you can add a filter rule to the monitoring resource. Here is an example:

yaml
Copy
123456789
apiVersion: operator.dash0.com/v1beta1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource
spec:
  filter:
    logs:
      log_records:
      - 'IsMatch(resource.attributes["k8s.pod.name"], "my-workload-name-*")'

Using a Custom Label Selector to Control Auto-Instrumentation

By providing a custom Kubernetes label selector in spec.instrumentWorkloads.labelSelectorCopy in a Dash0 monitoring resource, you can control which workloads in this namespace will be instrumented by the Dash0 operator.

• Workloads which match this label selector will be instrumented, subject to the value of spec.instrumentWorkloads.modeCopy.
• Workloads which do not match this label selector will never be instrumented, regardless of the value of spec.instrumentWorkloads.modeCopy.
• The setting spec.instrumentWorkloads.labelSelectorCopy setting is ignored if spec.instrumentWorkloads.mode=noneCopy.

If not set explicitly, this label selector assumes the value "dash0.com/enable!=false"Copy by default. That is, when no expicit label selector is provided via spec.instrumentWorkloads.labelSelectorCopy, workloads which

• do not have the label dash0.com/enableCopy at all, or
• have the label dash0.com/enableCopy with a value other than "false"Copy

will be instrumented, as explained in the previous section.

It is recommended to leave this setting unset (i.e. leave the default "dash0.com/enable!=false"Copy in place), unless you have a specific use case that requires a different label selector.

One such use case is implementing an opt-in model for workload instrumentation instead of the usual opt-out model. That is, instead of instrumenting all workloads in a namespace by default and only disabling instrumentation for a few specific workloads, you want to deliberately turn on auto-instrumentation for a few specific workloads and leave all others uninstrumented. Use a label selector with equals (=Copy) instead of not-equals (!=Copy) to achieve this, for example auto-instrument-this-workload-with-dash0="true"Copy.

Note that opting out of auto-instrumentation and workload modification via a label/label selector will not prevent log collection for pods in namespaces that have Dash0 log collection enabled, see previous section for details.

Specifying Additional Resource Attributes via Labels and Annotations

Note: The labels and annotations listed in this section can be specified at the pod level, or at the workload level (i.e., the cronjob, deployment, daemonset, job, replicaset, or statefulset). Pod labels and annotations take precedence over workload labels and annotations.

The following standard Kubernetes labels are mapped to resource attributes as follows:

• the label app.kubernetes.io/nameCopy is mapped to service.nameCopy
• if app.kubernetes.io/nameCopy is set, and the label app.kubernetes.io/versionCopy is also set, it is mapped to service.versionCopy
• if app.kubernetes.io/nameCopy is set, and the label app.kubernetes.io/part-ofCopy is also set, it is mapped to service.namespaceCopy

The operator will not combine pod labels with workload labels for this mapping. The labels app.kubernetes.io/versionCopy and app.kubernetes.io/part-ofCopy are only read from the pod labels if app.kubernetes.io/nameCopy is present on the pod. Similarly, the labels app.kubernetes.io/versionCopy and app.kubernetes.io/part-ofCopy are only read from the workload labels if app.kubernetes.io/nameCopy is present on the workload. Workload labels are not considered at all if app.kubernetes.io/nameCopy is present on the pod. This ensures that resource attributes are not partially based on pod and partially on workload labels, giving an inconsistent result.

Note: The OTEL_SERVICE_NAMECopy environment variable and service.*Copy key-value pairs specified in the OTEL_RESOURCE_ATTRIBUTESCopy environment variable have precedence over attributes derived from the app.kubernetes.io/*Copy labels.

Additionally, any annotation in the form of resource.opentelemetry.io/<key>: <value>Copy will be mapped to the resource attribute <key>=<value>Copy. For example, the following will result in the my.attribute=my-valueCopy resource attribute:

yaml
Copy
12345
apiVersion: v1
kind: Pod
metadata:
  annotations:
    resource.opentelemetry.io/my.attribute: my-value

As with the app.kubernetes.io/*Copy labels, the resource.opentelemetry.io/*Copy annotations can be set on the pod as well as on the workload. In contrast to the app.kubernetes.io/*Copy labels, mixing workload level and pod annotations is allowed, that is, you can set resource.opentelemetry.io/attribute-oneCopy on the workload and resource.opentelemetry.io/attribute-twoCopy on the pod, and both will be used. In case the same keyCopy is listed both on the workload and on the pod, the pod annotation takes precedence.

Key-value pairs with a specific keyCopy set via the OTEL_RESOURCE_ATTRIBUTESCopy environment variable will override the value derived from a resource.opentelemetry.io/<key>: <value>Copy annotation. Resource attributes set via the resource.opentelemetry.io/<key>: <value>Copy annotations will override the resource attributes value set via app.kubernetes.io/*Copy labels: for example, resource.opentelemetry.io/service.nameCopy has precendence over app.kubernetes.io/nameCopy.

Sending Data to the OpenTelemetry Collectors Managed by the Dash0 operator

Besides automatic workload instrumentation (which will make sure that the instrumented workloads send telemetry to the OpenTelemetry collectors managed by the operator), you can also send telemetry data from workloads that are not instrumented by the operator.

To do so, you need to add an OpenTelemetry SDK to your workload.

If the workload is in a namespace that is monitored by Dash0, the OpenTelemetry SDK will automatically be configured to send telemetry to the OpenTelemetry collectors managed by the Dash0 operator. This is because the operator automatically sets OTEL_EXPORTER_OTLP_ENDPOINTCopy and OTEL_EXPORTER_OTLP_PROTOCOLCopy to the correct values when applying the automatic workload instrumentation.

If the workload is in a namespace that is not monitored by Dash0 (or if spec.instrumentWorkloads.modeCopy is set to noneCopy in the respective Dash0 monitoring resource, or if the workload has opted out of auto-instrumentations via a label, you need to set the environment variable OTEL_EXPORTER_OTLP_ENDPOINTCopy (and optionally also OTEL_EXPORTER_OTLP_PROTOCOLCopy) yourself.

The DaemonSet OpenTelemetry collector managed by the Dash0 operator listens on host port 40318 for HTTP traffic and 40317 for gRPC traffic (unless the Helm chart has been deployed with operator.collectors.disableHostPorts=trueCopy, which disables the host ports for the collector pods). Additionally, there is also a service for the DaemonSet collector, which listens on the standard ports, that is 4318 for HTTP and 4317 for gRPC.

The preferred way of sending OTLP from your workoad to the Dash0-managed collector is to use node-local traffic via the host port. To do so, add the following environment variables to your workload:

yaml
Copy
123456789
env:
  - name: K8S_NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(K8S_NODE_IP):40318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"

• Note that listing the definition for K8S_NODE_IPCopy before OTEL_EXPORTER_OTLP_ENDPOINTCopy is crucial.
• Adding OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufCopy is optional when the OpenTelemetry SDK in question uses that protocol as the default.
• For gRCP, use OTEL_EXPORTER_OTLP_ENDPOINT=http://$(K8S_NODE_IP):40317Copy together with OTEL_EXPORTER_OTLP_PROTOCOL=grpcCopy instead.

To use the service endpoint instead of the host port, you need to know

• the Helm release name of the Dash0 operator (for example dash0-operatorCopy), and
• the namespace where the Dash0 operator is installed (for example dash0-systemCopy).

With that information at hand, add the following to your workload:

yaml
Copy
12345
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://${helm-release-name}-opentelemetry-collector-service.${namespace-of-the-dash0-operator}.svc.cluster.local:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"

or, for gRPC:

yaml
Copy
12345
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://${helm-release-name}-opentelemetry-collector-service.${namespace-of-the-dash0-operator}.svc.cluster.local:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"

In both cases, ${helm-release-name}Copy and ${namespace-of-the-dash0-operator}Copy needs to be replaced with the actual values.

If the workload is in a namespace that is monitored by Dash0 and workload instrumentation is enabled, Dash0 will automatically add Kubernetes-related OpenTelemetry resource attributes to your telemetry, even if the runtime in question is not yet supported by Dash0's auto-instrumentation. (There is currently one caveat: The resource attribute auto-detection relies on the process or runtime in question to use the Posix function getenvCopy to read environment variables, which is true for almost all runtimes. Two notable exceptions are Python and some statically compiled binaries.)

Exporting Data to Other Observability Backends

Instead of spec.export.dash0Copy in the Dash0 operator configuration resource, you can also provide spec.export.httpCopy or spec.export.grpcCopy to export telemetry data to arbitrary OTLP-compatible backends, or to another local OpenTelemetry collector.

Here is an example for HTTP:

yaml
Copy
123456789101112
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    http:
      endpoint: ... # provide the OTLP HTTP endpoint of your observability backend here
      headers: # you can optionally provide additional headers, for example for authorization
        - name: X-My-Header
          value: my-value
      encoding: json # optional, can be "json" or "proto", defaults to "proto"

Here is an example for gRPC:

yaml
Copy
1234567891011
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    grpc:
      endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here
      headers: # you can optionally provide additional headers, for example for authorization
        - name: X-My-Header
          value: my-value

You can combine up to three exporters (i.e. Dash0 plus gRPC plus HTTP) to send data to multiple backends. This allows sending the same data to two or three targets simultaneously. At least one exporter has to be defined. More than three exporters cannot be defined. Listing two or more exporters of the same type (i.e. providing spec.export.grpcCopy twice) is not supported.

Here is an example that combines three exporters:

yaml
Copy
1234567891011121314
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # TODO needs to be replaced with the actual value, see above
      authorization:
        token: auth_... # TODO needs to be replaced with the actual value, see above
    http:
      endpoint: ... # provide the OTLP HTTP endpoint of your observability backend here
    grpc:
      endpoint: ... # provide the OTLP gRPC endpoint of your observability backend here

Exporting Telemetry to Different Backends Per Namespace

Exporting telemetry to different backends per namespace is not yet implemented. This feature will be available in a future release. The export settings are available on the per-namespace resource (the Dash0 monitoring resource) to prepare for this feature.

Important: For that reason, having different export settings on the Dash0 monitoring resources in your cluster is currently strongly discouraged -- the export settings of one Dash0 monitoring resource would overwrite the settings of the other Dash0 monitoring resources, leading to non-deterministic behavior. For now, we recommend to only set the exportCopy attribute on the cluster's Dash0 operator configuration resource.

This restriction will be lifted once exporting telemetry to different backends per namespace is implemented.

Disable Self-Monitoring

By default, self-monitoring is enabled for the Dash0 operator as soon as you deploy a Das0 operator configuration resource with an export. That means, the operator will send self-monitoring telemetry to the configured Dash0 backend. Disabling self-monitoring is available as a setting on the Dash0 operator configuration resource. Dash0 does not recommend to disable the operator's self-monitoring.

Here is an example with self-monitoring disabled:

yaml
Copy
123456789
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration-resource
spec:
  selfMonitoring:
    enabled: false
  export:
    # ... see above for details on the export settings

Disable Dash0 Monitoring For a Namespace

If you want to stop monitoring a namespace with Dash0, remove the Dash0 monitoring resource from that namespace. For example, if you want to stop monitoring workloads in the namespace my-nodejs-applicationsCopy, use the following command:

console
Copy
1
kubectl delete --namespace my-nodejs-applications Dash0Monitoring dash0-monitoring-resource

or, alternatively, by using the dash0-monitoring.yamlCopy file created earlier:

console
Copy
1
kubectl delete --namespace my-nodejs-applications -f dash0-monitoring.yaml

Upgrading

To upgrade the Dash0 operator to a newer version, run the following commands:

console
Copy
12
helm repo update dash0-operator
helm upgrade --namespace dash0-system dash0-operator dash0-operator/dash0-operator

CRD Version Upgrades

Occasionally, the custom resource definitions (CRDs) used by the Dash0 operator (Dash0OperatorConfiguration, Dash0Monitoring) will be updated to new versions. Whenever possible, this will happen in a way that requires no manual intervention by users. This section contains details about CRD version updates and version migrations.

Operator Version 0.71.0: operator.dash0.com/v1alpha1/Dash0Monitoring -> operator.dash0.com/v1beta1/Dash0Monitoring

With operator version 0.71.0, the Dash0 operator's Dash0MonitoringCopy custom resource definition (CRD) is upgraded from version v1alpha1Copy to v1beta1Copy. The operator handles both versions correctly, that is version v1alpha1Copy to v1beta1Copy are both fully supported. Here is what you need to know about this version update for Dash0MonitoringCopy:

• If you have existing Dash0MonitoringCopy resources in version v1alpha1Copy in your cluster, they will be automatically converted on the fly, for example when the Dash0 operator reads the v1alpha1Copy resource version. At some point Kubernetes might also convert the resource permanently and store it in version v1beta1Copy.
• After the upgrade to version 0.71.0, you can still deploy new Dash0MonitoringCopy resource in version v1alpha1Copy (for example via kubectl applyCopy). They will be automatically converted and stored as v1beta1Copy resources by Kubernetes.
• If you want to migrate a Dash0MonitoringCopy template (e.g. a yaml file) from version v1alpha1Copy to v1beta1Copy, follow these steps:
  • If the template specifies the workload instrumentation mode via spec.instrumentWorkloadsCopy, replace that with spec.instrumentWorkloads.modeCopy. That is:

    yaml
    Copy
    12
      spec:
        instrumentWorkloads: created-and-updated

    becomes

    yaml
    Copy
    123
    spec:
      instrumentWorkloads:
        mode: created-and-updated

    If the template does not specify the workload instrumentation mode explicitly (that is, it relies on using the default instrumentation mode), no change is necessary here.
  • If the template contains the attribute spec.prometheusScrapingEnabledCopy, replace that with spec.prometheusScraping.enabledCopy. That is:

    yaml
    Copy
    12
      spec:
        prometheusScrapingEnabled: true

    becomes

    yaml
    Copy
    123
    spec:
      prometheusScraping:
        enabled: true

    The attribute spec.prometheusScraping.enabledCopy is also already valid for v1alpha1Copy, so this particular change can be applied independently of the CRD version change. If the template does not specify whether prometheusScraping is enabled or not (that is, it relies on using the default value), no change is necessary here.
• We recommend to update your templates from v1alpha1Copy to v1beta1Copy at some point. However, there are currently no plans to remove support for version v1alpha1Copy.
• If you want to use the new trace context propagators option that has been added in version 0.71.0, you need to use version v1beta1Copy of the Dash0MonitoringCopy resource. This includes updating your Yaml templates to that version, as described above.
• After upgrading to operator version 0.71.0 or later, you can no longer easily downgrade to a version before 0.71.0. In particular, this downgrade would require to manually delete all Dash0MonitoringCopy resources in the cluster. The reason is that the Dash0Monitoring resource are now stored as version v1beta1Copy by Kubernetes and there is no automatic downward conversion from v1beta1Copy back to v1alpha1Copy.
• The only supported version for Dash0OperatorConfigurationCopy is still v1alpha1Copy, that is, trying to use operator.dash0.com/v1beta1/Dash0OperatorConfigurationCopy will not work.

Uninstallation

To remove the Dash0 operator from your cluster, run the following command:

helm uninstall dash0-operator --namespace dash0-system
Copy

Depending on the command you used to install the operator, you may need to use a different Helm release name or namespace.

This will also automatically disable Dash0 monitoring for all namespaces by deleting the Dash0 monitoring resources in all namespaces. Optionally, remove the namespace that has been created for the operator:

kubectl delete namespace dash0-system
Copy

If you choose to not remove the namespace, you might want to consider removing the secret with the Dash0 authorization token (if such a secret has been created):

console
Copy
1
kubectl delete secret --namespace dash0-system dash0-authorization-secret

If you later decide to install the operator again, you will need to perform the initial configuration steps again:

1. set up a Dash0 backend connection and
2. enable Dash0 monitoring in each namespace you want to monitor, see Enable Dash0 Monitoring For a Namespace.

Automatic Workload Instrumentation

In namespaces that are enabled for Dash0 monitoring, all supported workload types are automatically instrumented by the Dash0 operator, to achieve two goals:

• Enable tracing for supported runtimes out of the box, and
• improve auto-detection of OpenTelemetry resource attributes.

This allows Dash0 users to avoid the hassle of manually adding the OpenTelemetry SDK to their applications, or to set Kubernetes-related resource attributes manually. Dash0 simply takes care of it automatically!

Automatic tracing only works for supported runtimes. For other runtimes, you can add an OpenTelemetry SDK to your workloads by other means.

Auto-detecting OpenTelemetry resource attributes works for all runtimes, that is, for runtimes that are supported by Dash0's auto-instrumentation as well as for workloads to which an OpenTelemetry SDK has been added otherwise. (There is currently one caveat: The resource attribute auto-detection relies on the process or runtime in question to use the Posix function getenvCopy to read environment variables, which is true for almost all runtimes. Two notable exceptions are Python and some statically compiled binaries.)

The Dash0 operator will instrument the following workload types:

• CronJobs
• DaemonSets
• Deployments
• Jobs
• Pods
• ReplicaSets
• StatefulSets

Note that Kubernetes jobs and Kubernetes pods are only instrumented at deploy time, existing jobs and pods cannot be instrumented since there is no way to restart them. For all other workload types, the operator can instrument existing workloads as well as new workloads at deploy time (depending on the setting of spec.instrumentWorkloads.modeCopy in the Dash0 monitoring resource).

The instrumentation process is performed by modifying the Pod spec template (for CronJobs, DaemonSets, Deployments, Jobs, ReplicaSets, and StatefulSets) or the Pod spec itself (for standalone Pods).

The modifications that are performed for workloads are the following:

• add an emptyDirCopy volume named dash0-instrumentationCopy to the pod spec
• add a volume mount dash0-instrumentationCopy to all containers of the pod
• add an init container named dash0-instrumentationCopy that will copy the OpenTelemetry SDKs and distributions for supported runtimes to the dash0-instrumentationCopy volume mount, so they are available in the target container's file system
• add or modifying environment variables (OTEL_EXPORTER_OTLP_ENDPOINTCopy, OTEL_EXPORTER_OTLP_PROTOCOLCopy, LD_PRELOADCopy, and several Dash0-specific variables prefixed with DASH0_Copy) to all containers of the pod
• add the Dash0 injector (see below for details) as a startup hook (via the LD_PRELOADCopy environment variable) to all containers of the pod
• add the following labels to the workload metadata:
  • dash0.com/instrumentedCopy: trueCopy or falseCopy depending on whether the workload has been successfully instrumented or not
  • dash0.com/operator-imageCopy: the fully qualified image name of the Dash0 operator image that has instrumented this workload
  • dash0.com/init-container-imageCopy: the fully qualified image name of the Dash0 instrumentation image that has been added as an init container to the pod spec
  • dash0.com/instrumented-byCopy: either controllerCopy or webhookCopy, depending on which component has instrumented this workload. The controller is responsible for instrumenting existing workloads while the webhook is responsible for instrumenting new workloads at deploy time.

Notes:

1. Automatic tracing will only happen for supported runtimes. Nonetheless, the modifications outlined above are performed for every workload. One reason for that is that there is no way to tell which runtime a workload uses from the outside, e.g. on the Kubernetes level. The more important reason is that runtimes that are not (yet) supported for auto-instrumentation still benefit from the improved OpenTelemetry resource attribute detection.
2. The operator operates sets OTEL_EXPORTER_OTLP_ENDPOINT=http://$(NODE_IP):40318Copy, that is, it tells the workload to send OTLP traffic to the HTTP port of the OpenTelemetry collector pod on the same host, which belongs to the OpenTelemetry collector DaemonSet managed by the operator. It also sets the protocol accordingly by setting OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufCopy. The protocol http/protobufCopy is the recommended default according to the OpenTelemetry specification, and it is widely supported. It does so under the assumption that workloads which have an OpenTelemetry SDK use an SDK that respects OTEL_EXPORTER_OTLP_PROTOCOLCopy and also has support for the http/protobufCopy protocol. For workloads that have an OpenTelemetry SDK that either does not respect OTEL_EXPORTER_OTLP_PROTOCOLCopy (and defaults to grpcCopy) or does not have support for http/protobufCopy, this will lead to the SDK trying to establish a gRPC connection to the collector's HTTP endpoint, that is, the SDK will not be able to emit telemetry. SDKs without support for http/protobufCopy are rather rare, but one prominent example is the Kubernetes ingress-nginx. The recommended approach is to disable workload instrumentation by the Dash0 operator for these workloads; for example by opting out of auto-instrumentation via a workload label (i.e. dash0.com/enable: "false"Copy, see Disabling Auto-Instrumentation for Specific Workloads), or by not installing a Dash0 monitoring resource in the namespace where these workloads are located. The workloads can then be monitored by following the setup described in Sending Data to the OpenTelemetry Collectors Managed by the Dash0 operator to have the workload send telemetry to the collectors managed by the Dash0 operator, using gRPC. Note that this is not relevant for workloads that do not have an OpenTelemetry SDK at all, since they will ignore OTEL_EXPORTER_OTLP_ENDPOINTCopy. In case the Dash0 operator Helm chart has been deployed with operator.collectors.forceUseServiceUrl=trueCopy or operator.collectors.disableHostPorts=trueCopy, OTEL_EXPORTER_OTLP_ENDPOINTCopy is not set to http://$(NODE_IP):40318Copy, but to the HTTP port of the DaemonSet collector's service URL http://${helm-release-name}-opentelemetry-collector-service.${namespace-of-the-dash0-operator}.svc.cluster.local:4318Copy instead.

The remainder of this section provides a more detailed step-by-step description of how the Dash0 operator's workload instrumentation for tracing works internally, intended for the technically curious reader. You can safely skip this section if you are not interested in the technical details.

1. The Dash0 operator adds the dash0-instrumentationCopy init container with the Dash0 instrumentation image to the pod spec template of workloads. The instrumentation image contains OpenTelemetry SDKs and distributions for all supported runtimes and the Dash0 injector binary.
2. When the init container starts, it copies the OpenTelemetry distributions and distributions and the injector binary to a dedicated shared volume mount that has been added by the operator, so they are available in the target container's file system. When it has copied all files, the init container exits.
3. The operator also adds environment variables to the target container to ensure that the OpenTelemetry SDK has the correct configuration and will get activated at startup. The activation of the OpenTelemetry SDK happens via an LD_PRELOADCopy hook. For that purpose, the Dash0 operator adds the LD_PRELOADCopy environment variable to the pod spec template of the workload. LD_PRELOADCopy is an environment variable that is evaluated by the dynamic linker/loader when a Linux executable starts. In general, it specifies a list of additional shared objects to be loaded before the actual code of the executable. In this specific case, the Dash0 injector binary is added to the LD_PRELOADCopy list.
4. At process startup, the Dash0 injector adds additional environment variables to the running process by hooking into the getenvCopy function of the standard library. The reason for doing that at process startup and not when modifying the pod spec (where environment variables can also be added and modified) is that the original environment variables are not necessarily fully known at that time. Workloads will sometimes set environment variables in their Dockerfile or in an entrypoint script; those environment variables are only available at process runtime. For example, the Dash0 injector sets (or appends to) NODE_OPTIONSCopy to activate the Dash0 OpenTelemetry distribution for Node.js to collect tracing data from all Node.js workloads. For JVMs, the same is achieved by setting (or appending to) the JAVA_TOOL_OPTIONSCopy environment variable, namely -javaagentCopy and -Dotel.resource.attributesCopy). The latter is neccessary because JVMs do not rely on getenvCopy to read environment variables.
5. Additionally, the Dash0 injector automatically improves Kubernetes-related resource attributes as follows: The operator sets the environment variables DASH0_NAMESPACE_NAMECopy, DASH0_POD_NAMECopy, DASH0_POD_UIDCopy and DASH0_CONTAINER_NAMECopy on workloads. The Dash0 injector binary picks these values up and uses them to populate the resource attributes k8s.namespace.nameCopy, k8s.pod.nameCopy, k8s.pod.uidCopy and k8s.container.nameCopy via the OTEL_RESOURCE_ATTRIBUTESCopy environment variable. If OTEL_RESOURCE_ATTRIBUTESCopy is already set on the process, the key-value pairs for these attributes are appended to the existing value of OTEL_RESOURCE_ATTRIBUTESCopy. If OTEL_RESOURCE_ATTRIBUTESCopy was not set on the process, the Dash0 injector will add OTEL_RESOURCE_ATTRIBUTESCopy as a new environment variable.

If you are curious, the source code for the injector is open source and can be found here.

Scraping Prometheus Endpoints

The Dash0 operator automatically scrapes Prometheus endpoints on pods annotated with the prometheus.io/*Copy annotations as defined by the Prometheus Helm chart.

The supported annotations are:

• prometheus.io/scrapeCopy: Only scrape pods that have a value of trueCopy, except if prometheus.io/scrape-slowCopy is set to trueCopy as well. Endpoints on pods annotated with this annotation are scraped every minute, i.e., scrape interval is 1 minute, unless prometheus.io/scrape-slowCopy is also set to trueCopy.
• prometheus.io/scrape-slowCopy: If set to trueCopy, enables scraping for the pod with scrape interval of 5 minutes. If both prometheus.io/scrapeCopy and prometheus.io/scrape-slowCopy are annotated on a pod with both values set to trueCopy, the pod will be scraped every 5 minutes.
• prometheus.io/schemeCopy: If the metrics endpoint is secured then you will need to set this to httpsCopy.
• prometheus.io/pathCopy: Override the metrics endpoint path if it is not the default /metricsCopy.
• prometheus.io/portCopy: Override the metrics endpoint port if it is not the default 9102Copy.

To be scraped, a pod annotated with the prometheus.io/scrapeCopy or prometheus.io/scrape-slowCopy annotations must belong to namespaces that are configured to be monitored by the Dash0 operator (see Enable Dash0 Monitoring For a Namespace section).

The scraping of a pod is executed from the same Kubernetes node the pod resides on.

This feature can be disabled for a namespace by explicitly setting prometheusScraping.enabled: falseCopy in the Dash0 monitoring resource.

Note: To also have Kube state metrics available (which are used extensively in Awesome Prometheus alerts) scraped and delivered to Dash0, you can annotate the kube-state-metrics pod with prometheus.io/scrape: "true"Copy and add a Dash0 monitoring resource to the namespace it is running in.

Managing Dash0 Dashboards

You can manage your Dash0 dashboards via the Dash0 operator.

Pre-requisites for this feature:

• A Dash0 operator configuration resource has to be installed in the cluster.
• The operator configuration resource must have the apiEndpointCopy property.
• The operator configuration resource must have a Dash0 export configured with authorization (either tokenCopy or secret-refCopy).
• The operator will only pick up Perses dashboard resources in namespaces that have a Dash0 monitoring resource deployed.
• The operator will not synchronize Perses dashboard resources in namespaces where the Dash0 monitoring resource has the setting synchronizePersesDashboardsCopy set to falseCopy. (This setting is optional and defaults to trueCopy when omitted.)

Furthermore, the custom resource definition for Perses dashboards needs to be installed in the cluster. There are two ways to achieve this:

• Install the Perses dashboard custom resource definition with the following command:

console
Copy
1
kubectl apply --server-side -f https://raw.githubusercontent.com/perses/perses-operator/refs/tags/v0.2.0/config/crd/bases/perses.dev_persesdashboards.yaml

• Alternatively, install the full Perses operator: Go to https://github.com/perses/perses-operator and follow the installation instructions there.

With the prerequisites in place, you can manage Dash0 dashboards via the operator. The Dash0 operator will watch for Perses dashboard resources in all namespaces that have a Dash0 monitoring resource deployed, and synchronize the Perses dashboard resources with the Dash0 backend:

• When a new Perses dashboard resource is created, the operator will create a corresponding dashboard via Dash0's API.
• When a Perses dashboard resource is changed, the operator will update the corresponding dashboard via Dash0's API.
• When a Perses dashboard resource is deleted, the operator will delete the corresponding dashboard via Dash0's API.

The dashboards created by the operator will be in read-only mode in the Dash0 UI.

If the Dash0 operator configuration resource has the datasetCopy property set, the operator will create the dashboards in that dataset, otherwise they will be created in the defaultCopy dataset.

You can opt out of synchronization for individual Perses dashboard resources by adding the Kubernetes label dash0.com/enable: falseCopy to the Perses dashboard resource. If this label is added to a dashboard which has previously been synchronized to Dash0, the operator will delete the corresponding dashboard in Dash0. Note that the spec.instrumentWorkloads.labelSelectorCopy in the monitoring resource does not affect the synchronization of Perses dashboards, the label to opt out of synchronization is always dash0.com/enable: falseCopy, even if a non-default label selector has been set in spec.instrumentWorkloads.labelSelectorCopy.

When a Perses dashboard resource has been synchronized to Dash0, the operator will write a summary of that synchronization operation to the status of the Dash0 monitoring resource in the same namespace. This summary will also show whether the dashboard had any validation issues or an error occurred during synchronization:

Kind: Dash0Monitoring
...
Status:
  Perses Dashboard Synchronization Results:
    my-namespace/perses-dashboard-test:
      Synchronization Status:     successful
      Synchronized At:            2024-10-25T12:02:12Z
Copy

Note: If you only want to manage dashboards, check rules, synthetic checks and views via the Dash0 operator, and you do not want it to collect telemetry, you can set telemetryCollection.enabledCopy to falseCopy in the Dash0 operator configuration resource. This will disable the telemetry collection by the operator, and it will also instruct the operator to not deploy the OpenTelemetry collector in your cluster.

Managing Dash0 Check Rules

You can manage your Dash0 check rules via the Dash0 operator.

Pre-requisites for this feature:

• A Dash0 operator configuration resource has to be installed in the cluster.
• The operator configuration resource must have the apiEndpointCopy property.
• The operator configuration resource must have a Dash0 export configured with authorization (either tokenCopy or secret-refCopy).
• The operator will only pick up Prometheus rule resources in namespaces that have a Dash0 monitoring resource deployed.
• The operator will not synchronize Prometheus rule resources in namespaces where the Dash0 monitoring resource has the setting synchronizePrometheusRulesCopy set to falseCopy. (This setting is optional and defaults to trueCopy when omitted.)

Furthermore, the custom resource definition for Prometheus rules needs to be installed in the cluster. There are two ways to achieve this:

• Install the Prometheus rules custom resource definition with the following command:

console
Copy
1
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.85.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml

• Alternatively, install the full kube-prometheus stack Helm chart: Go to https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack and follow the installation instructions there.

With the prerequisites in place, you can manage Dash0 check rules via the operator. The Dash0 operator will watch for Prometheus rule resources in all namespaces that have a Dash0 monitoring resource deployed, and synchronize the Prometheus rule resources with the Dash0 backend:

• When a new Prometheus rule resource is created, the operator will create corresponding check rules via Dash0's API.
• When a Prometheus rule resource is changed, the operator will update the corresponding check rules via Dash0's API.
• When a Prometheus rule resource is deleted, the operator will delete the corresponding check rules via Dash0's API.

Note that a Prometheus rule resource can contain multiple groups, and each of those groups can have multiple rules. The Dash0 operator will create individual check rules for each rule in each group.

If the Dash0 operator configuration resource has the datasetCopy property set, the operator will create the rules in that dataset, otherwise they will be created in the defaultCopy dataset.

Prometheus rules will be mapped to Dash0 check rules as follows:

• Each rulesCopy list item with an alertCopy attribute in all groupsCopy will be converted to an individual check rule in Dash0.
• Rules that have the recordCopy attribute set instead of the alertCopy attribute will be ignored.
• The name of the Dash0 check rule will be "${group_name} - ${alert}"Copy, where ${group_name}Copy is the nameCopy attribute of the group in the Prometheus rule resource, and ${alert}Copy is value of the alertCopy attribute.
• The intervalCopy attribute of the group will be used for the setting "Evaluate every" for each Dash0 check rule in that group.
• Other attributes in the Prometheus rule are converted to Dash0 check rule attributes as described in the table below.
• Some annotations and labels are interpreted by Dash0, these are described in the conversion table below. For example, to set the summary of the Dash0 check rule, add an annotation summaryCopy to the rulesCopy item in the Prometheus rule resource.
• If exprCopy contains the token $__thresholdCopy, and neither annotation dash0-threshold-degradedCopy nor dash0-threshold-criticalCopy is present, the rule will be considered invalid and will not be synchronized to Dash0.
• If the rule has the annotation dash0-enabled=falseCopy, the check rule will be synchronized but disabled in Dash0. This Prometheus annotation is not to be confused with the Kubernetes label dash0.com/enable: falseCopy, which disables synchronization of the entire Prometheus rules resource (and all its check rules) to Dash0 (see below).
• The group attribute limitCopy is not supported by Dash0 and will be ignored.
• The group attribute partial_response_strategyCopy is not supported by Dash0 and will be ignored.
• All labels (except for the ones explicitly mentioned in the conversion table below) will be listed under "Additional labels".
• All annotations (except for the ones explicitly mentioned in the conversion table below) will be listed under "Annotations".

You can opt out of synchronization for individual Prometheus rules resources by adding the Kubernetes label dash0.com/enable: falseCopy to it. If this label is added to a Prometheus rules resource which has previously been synchronized to Dash0, the operator will delete all corresponding check rules in Dash0. Note that this refers to a Kubernetes label on the Kubernetes resource, and it will affect all check rules contained in this Prometheus rules resource. This mechanism is not to be confused with the Prometheus annotation dash0-enabledCopy, which can be applied to individual rules in a Prometheus rules resource, and controls whether the check rule is enabled or disabled in Dash0. Please also note that the spec.instrumentWorkloads.labelSelectorCopy in the monitoring resource does not affect the synchronization of Prometheus rule resources, the label to opt out of synchronization is always dash0.com/enable: falseCopy, even if a non-default label selector has been set in spec.instrumentWorkloads.labelSelectorCopy.

When a Prometheus rules resource has been synchronized to Dash0, the operator will write a summary of that synchronization operation to the status of the Dash0 monitoring resource in the same namespace. This summary will also show whether any of the rules had validation issues or errors occurred during synchronization:

yaml
Copy
123456789101112131415
Kind: Dash0Monitoring
...
Status:
  Prometheus Rule Synchronization Results:
    my-namespace/prometheus-example-rules:
      Synchronization Status:        successful
      Synchronized At:               2024-10-25T11:59:49Z
      Alerting Rules Total:          3
      Synchronized Rules Total:      3
      Synchronized Rules:
        dash0/k8s - K8s Deployment replicas mismatch
        dash0/k8s - K8s pod crash looping
        dash0/collector - exporter send failed spans
      Invalid Rules Total:           0
      Synchronization Errors Total:  0

Note: If you only want to manage dashboards, check rules, synthetic checks and views via the Dash0 operator, and you do not want it to collect telemetry, you can set telemetryCollection.enabledCopy to falseCopy in the Dash0 operator configuration resource. This will disable the telemetry collection by the operator, and it will also instruct the operator to not deploy the OpenTelemetry collector in your cluster.

Managing Dash0 Synthetic Checks

You can manage your Dash0 synthetic checks via the Dash0 operator.

Pre-requisites for this feature:

• A Dash0 operator configuration resource has to be installed in the cluster.
• The operator configuration resource must have the apiEndpointCopy property.
• The operator configuration resource must have a Dash0 export configured with authorization (either tokenCopy or secret-refCopy).
• The operator will only pick up synthetic check resources in namespaces that have a Dash0 monitoring resource deployed.

With the prerequisites in place, you can manage Dash0 synthetic checks via the operator. The Dash0 operator will watch for synthetic check resources in all namespaces that have a Dash0 monitoring resource deployed, and synchronize the synthetic check resources with the Dash0 backend:

• When a new synthetic check resource is created, the operator will create a corresponding synthetic check via Dash0's API.
• When a synthetic check resource is changed, the operator will update the corresponding synthetic check via Dash0's API.
• When a synthetic check resource is deleted, the operator will delete the corresponding synthetic check via Dash0's API.

The synthetic checks created by the operator will be in read-only mode in the Dash0 UI.

If the Dash0 operator configuration resource has the datasetCopy property set, the operator will create the synthetic checks in that dataset, otherwise they will be created in the defaultCopy dataset.

You can opt out of synchronization for individual synthetic checks resources by adding the Kubernetes label dash0.com/enable: falseCopy to the synthetic check resource. If this label is added to a synthetic chec which has previously been synchronized to Dash0, the operator will delete the corresponding synthetic chec in Dash0. Note that the spec.instrumentWorkloads.labelSelectorCopy in the monitoring resource does not affect the synchronization of synthetic checks, the label to opt out of synchronization is always dash0.com/enable: falseCopy, even if a non-default label selector has been set in spec.instrumentWorkloads.labelSelectorCopy.

When a synthetic check resource has been synchronized to Dash0, the operator will write a summary of that synchronization operation to its status. Note that in contrast to synchronizing Prometheus rules or Perses dashboards (which are third-party custom resources from the perspective of the operator, i.e. they are potentially owned and managed by another Kubernetes operator), the result of the synchronization operation will not be written to the status of the Dash0 monitoring resource in the same namespace, but to the synthetic check resource status directly. The status will also show whether the synthetic check had any validation issues or an error occurred during synchronization.

yaml
Copy
12345
Kind: Dash0SyntheticCheck
...
Status:
  Synchronization Status: successful
  Synchronized At:        2025-09-05T11:47:56Z

Note: If you only want to manage dashboards, check rules, synthetic checks and views via the Dash0 operator, and you do not want it to collect telemetry, you can set telemetryCollection.enabledCopy to falseCopy in the Dash0 operator configuration resource. This will disable the telemetry collection by the operator, and it will also instruct the operator to not deploy the OpenTelemetry collector in your cluster.

Managing Dash0 Views

You can manage your Dash0 views via the Dash0 operator.

Pre-requisites for this feature:

• A Dash0 operator configuration resource has to be installed in the cluster.
• The operator configuration resource must have the apiEndpointCopy property.
• The operator configuration resource must have a Dash0 export configured with authorization (either tokenCopy or secret-refCopy).
• The operator will only pick up Dash0 view resources in namespaces that have a Dash0 monitoring resource deployed.

With the prerequisites in place, you can manage Dash0 views via the operator. The Dash0 operator will watch for view resources in all namespaces that have a Dash0 monitoring resource deployed, and synchronize the view resources with the Dash0 backend:

• When a new view resource is created, the operator will create a corresponding view via Dash0's API.
• When a view resource is changed, the operator will update the corresponding view via Dash0's API.
• When a view resource is deleted, the operator will delete the corresponding view via Dash0's API.

The views created by the operator will be in read-only mode in the Dash0 UI.

If the Dash0 operator configuration resource has the datasetCopy property set, the operator will create the view in that dataset, otherwise they will be created in the defaultCopy dataset.

You can opt out of synchronization for individual views resources by adding the Kubernetes label dash0.com/enable: falseCopy to the view resource. If this label is added to a synthetic chec which has previously been synchronized to Dash0, the operator will delete the corresponding synthetic chec in Dash0. Note that the spec.instrumentWorkloads.labelSelectorCopy in the monitoring resource does not affect the synchronization of views, the label to opt out of synchronization is always dash0.com/enable: falseCopy, even if a non-default label selector has been set in spec.instrumentWorkloads.labelSelectorCopy.

When a view resource has been synchronized to Dash0, the operator will write a summary of that synchronization operation to its status. Note that in contrast to synchronizing Prometheus rules or Perses dashboards (which are third-party custom resources from the perspective of the operator, i.e. they are potentially owned and managed by another Kubernetes operator), the result of the synchronization operation will not be written to the status of the Dash0 monitoring resource in the same namespace, but to the view resource status directly. The status will also show whether the view had any validation issues or an error occurred during synchronization.

yaml
Copy
12345
Kind: Dash0View
...
Status:
  Synchronization Status: successful
  Synchronized At:        2025-09-05T11:47:56Z

Note: If you only want to manage dashboards, check rules, synthetic checks and views via the Dash0 operator, and you do not want it to collect telemetry, you can set telemetryCollection.enabledCopy to falseCopy in the Dash0 operator configuration resource. This will disable the telemetry collection by the operator, and it will also instruct the operator to not deploy the OpenTelemetry collector in your cluster.

Notes on ArgoCD

As many other Helm charts, the Dash0 operator Helm chart regenerates TLS certificates for in-cluster communication, that is, for its services and webhooks. The certificate will be regenerated every time the Dash0 operator Helm chart is applied. For users deploying the Dash0 operator via ArgoCD, and in particular without using ArgoCD's auto-sync feature, the certificates and derived data (ca.crtCopy, tls.crtCopy, tls.keyCopy, caBundleCopy) will show up as a diff in the ArgoCD UI. The certificate is also regenerated everytime the hard refresh option is used in ArgoCD, since this action will trigger rendering the Helm chart templates again, even if nothing has changed in the git repository.

To avoid this, you can instruct ArgoCD to ignore these particular differences. Here is an example for an argoproj.io/v1alpha1.ApplicationCopy resource with ignoreDifferencesCopy:

yaml
Copy
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dash0-operator
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:

  source:
    chart: dash0-operator
    repoURL: https://dash0hq.github.io/dash0-operator
    targetRevision: ...

  # ... your current spec for the dash0-operator ArgoCD application

  # Ignore certificates which are generated on the fly during Helm chart template rendering:
  ignoreDifferences:
    - kind: Secret
      name: dash0-operator-certificates
      jsonPointers:
        - /data/ca.crt
        - /data/tls.crt
        - /data/tls.key
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: dash0-operator-injector
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: dash0-operator-monitoring-mutating
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      name: dash0-operator-operator-configuration-mutating
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: dash0-operator-monitoring-validator
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      name: dash0-operator-operator-configuration-validator
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle

Notes on Kyverno Admission Controller

In clusters that have the Kyverno admission controller deployed, it is highly recommended to either use a volume for filelog offsets instead of the default filelog offset config map, or to exclude ConfigMaps (or all resource types) in the Dash0 operator's namespace from Kyverno's processing. Leaving Kyverno processing in place and using the config map filelog offset storage can lead to severe performance issues, since the default config map for filelog offsets is updated very frequently. This can cause Kyverno to consume a lot of CPU and memory resources, potentially even leading to OOMKills of the Kyverno admission controller.

Notes on Running The Operator on Apple Silicon

When running the operator on an Apple Silicon host (M1, M3 etc.), for example via Docker Desktop, some attention needs to be paid to the CPU architecture of images. The architecture of the Kubernetes node for this scenario will be arm64Copy. When running a single-architecture amd64Copy image (as opposed to a single-architecture arm64Copy image or a multi-platform build containing amd64Copy as well as arm64Copy) the operator will prevent the container from starting.

The reason for this is the interaction between Rosetta emulation and how the operator works. The Dash0 instrumentation image (which is added as an init container and contains the auto-tracing injector) is a multi-platform image, supporting both amd64Copy and arm64Copy. When this image is pulled from an Apple Silicon machine, it automatically pulls the arm64Copy variant. That is, the injector binary that is added via the init container is compiled for arm64Copy. Now, when the application from your amd64Copy application image is started, the injector and the application will be incompatible, as they have been built for two different CPU architectures.

Under normal circumstances, an amd64Copy image would not work on an arm64Copy Kubernetes node anyway, but in the case of Docker Desktop on MacOS, this combination is enabled due to Docker Desktop automatically running amd64Copy images via Rosetta2 emulation.

You can work around this issue by one of the following methods:

• using an amd64Copy Kubernetes node,
• by building a multi-platform image for your application, or
• by building the application as an arm64Copy image (e.g. by using --platform=linux/arm64Copy when building the image).

Notes on Running The Operator on Docker Desktop

The hostmetricsCopy receiver will be disabled when using Docker as the container runtime.

Notes on Running The Operator on Minikube

The hostmetricsCopy receiver will be disabled when using Docker as the container runtime.