Technology
Dash0 Kubernetes Operator
For monitoring your Kubernetes cluster and workloads there is no faster and easier way than using Dash0's Kubernetes Operator.
Overview
Overview
The Dash0 Kubernetes operator is designed to easily gather telemetry from your cluster and its deployed workloads. It comes with built-in best practices for Kubernetes, automatic workload identification and instrumentation, and support for configuration as code.
The Dash0 operator for Kubernetes is 100% open source, made of several OpenTelemetry components optimized to provide a great out-of-the-box experience monitoring your Kubernetes workloads.
You can learn more about the Dash0 K8s operator in our documentation.
Use Cases
- Automatically instrument your Java, .NET, Node.js, Python and Ruby workloads
- Collect logs from your Kubernetes cluster automatically
- Monitor Kubernetes cluster and workload metrics
- Scrape Prometheus metrics
- Synchronize Prometheus alert rules, Perses dashboards, Dash0 views, synthetic checks and notification channels for configuration-as-code
Setup
Overview
Let's get the Dash0 Kubernetes Operator installed. It only takes a few minutes!
Install the Dash0 Operator Helm Chart
Add the Helm chart for the Dash0 operator to your Helm repository.
helm repo add dash0-operator https://dash0hq.github.io/dash0-operator
helm repo updateProvide your Dash0 Authorization token
You need a Dash0 authorization token to connect to Dash0's servers for data transmission. We recommend to store it securely within a Kubernetes secret in the namespace where you will install the Dash0 operator.
kubectl create namespace dash0-system
kubectl create secret generic dash0-authorization-secret \
--namespace dash0-system \
--from-literal=token={{token}}Deploy the Dash0 operator to your Kubernetes cluster
Next, let's deploy the Dash0 operator via its Helm chart.
helm upgrade --install \
--namespace dash0-system \
--set operator.dash0Export.enabled=true \
--set operator.dash0Export.endpoint={{endpoint_otlp_grpc}} \
--set operator.dash0Export.apiEndpoint={{endpoint_api_http}} \
--set operator.dash0Export.secretRef.name=dash0-authorization-secret \
--set operator.dash0Export.secretRef.key=token \
dash0-operator \
dash0-operator/dash0-operatorWhen installing the Dash0 operator on GKE Autopilot, you should also add --set operator.gke.autopilot.enabled=true.
Enabling the Operator for a Namespace
Observability with the Dash0 operator must be enabled for each namespace that you want to monitor with Dash0. To start
monitoring a namespace, create a Dash0Monitoring resource in the namespace, like this:
kubectl apply -n $NAMESPACE -f - <<EOF
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
name: dash0-monitoring-resource
EOFContinuous profiling
To collect continuous profiles, we recommend using the operator-managed eBPF profiler. The operator deploys the profiler on each node and configures the collectors to forward profiles to Dash0, without changes to your application code or container images. Profiling is optional and is not required for the basic setup above.
The eBPF profiler runs as a privileged container with host PID access and read-only mounts of /proc and /sys. Your
operator namespace's Pod Security Admission policy must permit these privileges. The chart does not change namespace
security labels. Review the profiling security
requirements
with your cluster administrator.
The operator-managed eBPF profiler is not yet supported on GKE Autopilot. Do not enable it on Autopilot clusters.
On supported clusters, enable profiling and the managed eBPF profiler when installing or upgrading the operator:
helm upgrade --install \
--namespace dash0-system \
--set operator.profilingEnabled=true \
--set operator.ebpfProfilerEnabled=true \
--set operator.dash0Export.enabled=true \
--set operator.dash0Export.endpoint={{endpoint_otlp_grpc}} \
--set operator.dash0Export.apiEndpoint={{endpoint_api_http}} \
--set operator.dash0Export.secretRef.name=dash0-authorization-secret \
--set operator.dash0Export.secretRef.key=token \
dash0-operator \
dash0-operator/dash0-operatorEnabling Opt-In Language Auto-Instrumentation
Auto-instrumentation for Python and Ruby is opt-in. Use the flags below when installing or upgrading the Helm chart to enable either or both:
| Language | Recommended minimum version | Helm flag |
|---|---|---|
| Python | 3.8+ | operator.instrumentation.enablePythonAutoInstrumentation |
| Ruby | 3.3+ | operator.instrumentation.enableRubyAutoInstrumentation |
Workloads running older versions are unaffected: the instrumentation detects the runtime version at startup and deactivates itself gracefully, so it is safe to enable these flags in clusters with mixed runtime versions.
For example, to enable both:
helm upgrade --install \
--namespace dash0-system \
--set operator.dash0Export.enabled=true \
--set operator.dash0Export.endpoint={{endpoint_otlp_grpc}} \
--set operator.dash0Export.apiEndpoint={{endpoint_api_http}} \
--set operator.dash0Export.secretRef.name=dash0-authorization-secret \
--set operator.dash0Export.secretRef.key=token \
--set operator.instrumentation.enablePythonAutoInstrumentation=true \
--set operator.instrumentation.enableRubyAutoInstrumentation=true \
dash0-operator \
dash0-operator/dash0-operatorPython and Ruby auto-instrumentation only support the http/protobuf OTLP transport protocol. If a workload's
container sets OTEL_EXPORTER_OTLP_PROTOCOL=grpc, telemetry will not be exported. Remove that override from the pod
spec to let the operator configure the correct protocol automatically.
Enabling these flags on an existing installation causes all pods in namespaces with instrumentWorkloads.mode=all to
restart immediately. Pods in namespaces with mode=created-and-updated pick up the change on their next redeployment.
Learn more about the Dash0 operator
Consult the Dash0 documentation to learn more about the features of the Dash0 operator.
It is recommended to also revisit the section Recommended Settings for Production Use to learn about the most important configuration settings you might want for a robust experience (e.g. filelog offset volumes, cert-manager integration, configuring resource limits).
Dashboards
Check Rules
[Dash0 Operator] High error log rate in Dash0 operator manager
The operator manager pod is logging errors at an elevated rate
The operator manager pod {{ $labels.k8s_pod_name }} logs errors in cluster {{ $labels.k8s_cluster_name }}.
sum by (k8s_cluster_name, k8s_pod_name) (increase({otel_metric_name = "dash0.logs", service_namespace="dash0-operator", service_name="operator-manager", otel_log_severity_range =~ "^(ERROR|FATAL)$"}[1m])) > $__threshold[Dash0 Operator] IaC synchronization issues
The Dash0 operator reports issues when synchronizing resources with the Dash0 API
The Dash0 operator reports issues when synchronizing resources with the Dash0 API in cluster {{$labels.k8s_cluster_name}}.
(sum by (k8s_cluster_name) (increase({otel_metric_name = "dash0.logs", otel_log_severity_range =~ "^(WARN|ERROR|FATAL)$", service_namespace="dash0-operator", service_name="operator-manager", error = "validation issues and/or synchronization issues occurred"}[5m])) or sum by (k8s_cluster_name) (increase({otel_metric_name = "dash0.logs", otel_log_severity_range =~ "^(WARN|ERROR|FATAL)$", service_namespace="dash0-operator", service_name="operator-manager", otel_log_body =~ ".*validation issues and/or synchronization issues occurred.*"}[5m]))) > $__threshold[Dash0 Operator] Operator is down
The Dash0 operator manager pod is not ready, according to the k8s.container.ready metric
The Dash0 operator manager pod in cluster {{ $labels.k8s_cluster_name }} is not ready, according to the k8s.container.ready metric.
max by (k8s_cluster_name) ({otel_metric_name = "k8s.container.ready", otel_metric_type = "gauge", service_namespace = "dash0-operator", service_name = "operator-manager", k8s_container_name = "manager"}) < $__threshold[Dash0 Operator] Pods are in CrashLoopBackOff
Dash0 operator pods are in a CrashLoopBackOff state
Pods in {{ $labels.k8s_cluster_name }}/{{ $labels.k8s_namespace_name }} are in a CrashLoopBackOff.
sum by (k8s_cluster_name, k8s_namespace_name) ({otel_metric_name = "k8s.container.status.reason", otel_metric_type = "sum", service_namespace = "dash0-operator", k8s_container_status_reason = "CrashLoopBackOff"}) >= $__thresholdDash0 Operator PrometheusRule reconciliation high rate
Dash0 Operator PrometheusRule reconciliation rate is unusually high
Alert triggers when PrometheusRule reconciliation rate is unusually high, which may indicate issues with the operator or configuration changes.
rate({otel_metric_name="dash0.operator.manager.prometheusrule.reconcile_requests"}[5m]) > $__threshold
