Dash0 Kubernetes Operator

Description

The Dash0 Kubernetes operator installs an OpenTelemetry collector into your cluster that sends data to your Dash0 ingress endpoint, with authentication already configured out of the box. Additionally, it will enable gathering OpenTelemetry data from applications deployed to the cluster for a selection of supported runtimes, plus automatic log collection and metrics.

The Dash0 Kubernetes operator is currently available as a technical preview.

Supported runtimes for automatic instrumentation:

• Node.js 18 and beyond

  Prerequisites

• Kubernetes >= 1.xx
• Helm >= 3.x, please refer to Helm's documentation for more information on installing Helm.

To use the operator, you will need provide two configuration values:

• endpointCopy: The URL of the Dash0 ingress endpoint backend to which telemetry data will be sent. This property is mandatory when installing the operator. This is the OTLP/gRPC endpoint of your Dash0 organization. The correct OTLP/gRPC endpoint can be copied fom https://app.dash0.com -> organization settings -> "Endpoints". Note that the correct endpoint value will always start with ingress.Copy and end in dash0.com:4317Copy. Including a protocol prefix (e.g. https://Copy) is optional.
• Either tokenCopy or secretRefCopy: Exactly one of these two properties needs to be provided when installing the operator.
  • tokenCopy: This is the Dash0 authorization token of your organization. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". The prefix Bearer Copymust not be included in the value. Note that when you provide a token, it will be rendered verbatim into a Kubernetes ConfigMap object. Anyone with API access to the Kubernetes cluster will be able to read the value. Use a secret reference and a Kubernetes secret if you want to avoid that.
  • secretRefCopy: A reference to an existing Kubernetes secret in the Dash0 operator's namespace. The secret needs to contain the Dash0 authorization token. See below for details on how exactly the secret should be created and configured.

Installation

Before installing the operator, add the Dash0 operator's Helm repository as follows:

01
helm repo add dash0-operator https://dash0hq.github.io/dash0-operator
helm repo update

Now you can install the operator into your cluster via Helm with the following command:

01234567
helm install \
  --namespace dash0-system \
  --create-namespace \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.token=REPLACE THIS WITH YOUR DASH0 AUTH TOKEN \
  dash0-operator \
  dash0-operator/dash0-operator

Instead of providing the auth token directly, you can also use a secret reference:

012345678
helm install \
  --namespace dash0-system \
  --create-namespace \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.secretRef.name=REPLACE THIS WITH THE NAME OF AN EXISTING KUBERNETES SECRET \
  --set operator.dash0Export.secretRef.key=REPLACE THIS WITH THE PROPERTY KEY IN THAT SECRET \
  dash0-operator \
  dash0-operator/dash0-operator

See the section Using a Kubernetes Secret for the Dash0 Authorization Token for more information on using a Kubernetes secrets with the Dash0 operator.

Last but not least, you can also install the operator without providing a Dash0 backend configuration:

01234
helm install \
  --namespace dash0-system \
  --create-namespace \
  dash0-operator \
  dash0-operator/dash0-operator

However, you will need to create a Dash0 operator configuration resource later that provides the backend connection settings. That is, providing --set operator.dash0Export.enabled=trueCopy and the other backend-related settings when running helm installCopy is simply a shortcut to deploy the Dash0 operator configuration resource automatically at startup.

On its own, the operator will not do much. To actually have the operator monitor your cluster, two more things need to be set up:

1. a Dash0 backend connection has to be configured and
2. monitoring workloads has to be enabled per namespace.

Both steps are described in the following sections.

Configuration

Configuring the Dash0 Backend Connection

You can skip this step if you provided --set operator.dash0Export.enabled=trueCopy together with the endpoint and either a token or a secret reference when running helm installCopy. In that case, proceed to the next section, Enable Dash0 Monitoring For a Namespace.

Otherwise, configure the backend connection now by creating a file dash0-operator-configuration.yamlCopy with the following content:

0123456789101112
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      # Replace this value with the actual OTLP/gRPC endpoint of your Dash0 organization.
      endpoint: ingress... # TODO needs to be replaced with the actual value, see below

      authorization:
        # Provide the Dash0 authorization token as a string via the token property:
        token: auth_... # TODO needs to be replaced with the actual value, see below

You need to provide two configuration settings:

• spec.export.dash0.endpointCopy: The URL of the Dash0 ingress endpoint backend to which telemetry data will be sent. This property is mandatory. Replace the value in the example above with the OTLP/gRPC endpoint of your Dash0 organization. The correct OTLP/gRPC endpoint can be copied fom https://app.dash0.com -> organization settings -> "Endpoints". Note that the correct endpoint value will always start with ingress.Copy and end in dash0.com:4317Copy. Including a protocol prefix (e.g. https://Copy) is optional.
• spec.export.dash0.authorization.tokenCopy or spec.export.dash0.authorization.secretRefCopy: Exactly one of these two properties needs to be provided. Providing both will cause a validation error when installing the Dash0Monitoring resource.
  • spec.export.dash0.authorization.tokenCopy: Replace the value in the example above with the Dash0 authorization token of your organization. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". The prefix Bearer Copymust not be included in the value. Note that the value will be rendered verbatim into a Kubernetes ConfigMap object. Anyone with API access to the Kubernetes cluster will be able to read the value. Use the secretRefCopy property and a Kubernetes secret if you want to avoid that.
  • spec.export.dash0.authorization.secretRefCopy: A reference to an existing Kubernetes secret in the Dash0 operator's namespace. See the section Using a Kubernetes Secret for the Dash0 Authorization Token for an example file that uses a secretRefCopy. The secret needs to contain the Dash0 authorization token. See below for details on how exactly the secret should be created and configured. Note that by default, Kubernetes secrets are stored unencrypted, and anyone with API access to the Kubernetes cluster will be able to read the value. Additional steps are required to make sure secret values are encrypted. See https://kubernetes.io/docs/concepts/configuration/secret/ for more information on Kubernetes secrets.

After providing the required values, save the file and apply the resource to the Kubernetes cluster you want to monitor:

0
kubectl apply -f dash0-operator-configuration.yaml

The Dash0 operator configuration resource is cluster-scoped, so a specific namespace should not be provided when applying it.

Enable Dash0 Monitoring For a Namespace

For each namespace that you want to monitor with Dash0, enable workload monitoring by installing a Dash0 monitoring resource into that namespace:

Create a file dash0-monitoring.yamlCopy with the following content:

0123
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource

Save the file and apply the resource to the namespace you want to monitor. For example, if you want to monitor workloads in the namespace my-nodejs-applicationsCopy, use the following command:

0
kubectl apply --namespace my-nodejs-applications -f dash0-monitoring.yaml

If you want to monitor the defaultCopy namespace with Dash0, use the following command:

0
kubectl apply -f dash0-monitoring.yaml

The Dash0 monitoring resource supports additional configuration settings:

• spec.instrumentWorkloadsCopy: A namespace-wide opt-out for workload instrumentation for the target namespace. There are threepossible settings: allCopy, created-and-updatedCopy and noneCopy. By default, the setting allCopy is assumed.If this setting is omitted, the value allCopy is assumed and new/updated as well as existing Kubernetes workloads will be intrumented by the operator to send telemetry to Dash0, as described above.More fine-grained per-workload control over instrumentation is available by setting the label dash0.com/enable=falseCopy on individual workloads.The behavior when changing this setting for an existing Dash0 monitoring resource is as follows:
  • allCopy: If set to allCopy (or omitted), the operator will:
    • instrument existing workloads in the target namespace (i.e. workloads already running in the namespace) when the Dash0 monitoring resource is deployed,
    • instrument existing workloads or update the instrumentation of already instrumented workloads in the target namespace when the Dash0 Kubernetes operator is first started or restarted (for example when updating the operator),
    • instrument new workloads in the target namespace when they are deployed, and
    • instrument changed workloads in the target namespace when changes are applied to them. Note that the first two actions (instrumenting existing workloads) will result in restarting the pods of the affected workloads.
  • created-and-updatedCopy: If set to created-and-updatedCopy, the operator will not instrument existing workloads in the target namespace. Instead, it will only:
    • instrument new workloads in the target namespace when they are deployed, and
    • instrument changed workloads in the target namespace when changes are applied to them. This setting is useful if you want to avoid pod restarts as a side effect of deploying the Dash0 monitoring resource or restarting the Dash0 Kubernetes operator.
  • noneCopy: You can opt out of instrumenting workloads entirely by setting this option to noneCopy. With spec.instrumentWorkloads: noneCopy, workloads in the target namespace will never be instrumented to send telemetry to Dash0.
  • When this setting is updated to spec.instrumentWorkloads=allCopy (and it had a different value before): All existing uninstrumented workloads will be instrumented.
  • When this setting is updated to spec.instrumentWorkloads=noneCopy (and it had a different value before): The instrumentation will be removed from all instrumented workloads.
  • Updating this value to spec.instrumentWorkloads=created-and-updatedCopy has no immediate effect; existing uninstrumented workloads will not be instrumented, existing instrumented workloads will not be uninstrumented. Newly deployed or updated workloads will be instrumented from the point of the configuration change onwards as described above.

Here is an example file for a monitoring resource that sets the spec.instrumentWorkloadsCopy property to created-and-updatedCopy:

01234567
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource
spec:
  # Opt-out settings for particular use cases. The default value is "all". Other possible values are
  # "created-and-updated" and "none".
  instrumentWorkloads: created-and-updated

Using a Kubernetes Secret for the Dash0 Authorization Token

If you want to provide the Dash0 authorization token via a Kubernetes secret instead of providing the token as a string, create the secret in the namespace where the Dash0 operator is installed. If you followed the guide above, the name of that namespace is dash0-systemCopy. The authorization token for your Dash0 organization can be copied from https://app.dash0.com -> organization settings -> "Auth Tokens". You can freely choose the name of the secret and the key of the token within the secret.

Create the secret by using the following command:

0123
kubectl create secret generic \
 dash0-authorization-secret \
 --namespace dash0-system \
 --from-literal=token=auth_...your-token-here...

With this example command, you would create a secret with the name dash0-authorization-secretCopy in the namespace dash0-systemCopy. If you installed (or plan to install) the operator into a different namespace, replace the --namespaceCopy parameter accordingly.

The name of the secret as well as the key of the token value within the secret must be provided when referencing the secret during helm installCopy, or in the YAML file for the Dash0 operator configuration resource (in the secretRefCopy property).

For creating the operator configuration resource with helm installCopy, the command would look like this, assuming the secret has been created as shown above:

01234567
helm install \
  --namespace dash0-system \
  --set operator.dash0Export.enabled=true \
  --set operator.dash0Export.endpoint=REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT \
  --set operator.dash0Export.secretRef.name=dash0-authorization-secret \
  --set operator.dash0Export.secretRef.key=token \
  dash0-operator \
  dash0-operator/dash0-operator

If you do not want to install the operator configuration resource via helm installCopy but instead deploy it manually, and use a secret reference for the auth token, the following example YAML file would work work with the secret created above:

0123456789101112
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # TODO REPLACE THIS WITH YOUR DASH0 INGRESS ENDPOINT

      authorization:
        secretRef:
          name: dash0-authorization-secret
          key: token

When deploying the operator configuration resource via kubectlCopy, the following defaults apply:

• If the nameCopy property is omitted, the name dash0-authorization-secretCopy will be assumed.
• If the keyCopy property is omitted, the key tokenCopy will be assumed.

With these defaults in mind, the secretRefCopy could have also been written as follows:

012345678910
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # TODO needs to be replaced with the actual value, see above

      authorization:
        secretRef: {}

Note: There are no defaults when using --set operator.dash0Export.secretRef.nameCopy and --set operator.dash0Export.secretRef.keyCopy with helm installCopy, so for that approach the values must always be provided explicitly.

Note that by default, Kubernetes secrets are stored unencrypted, and anyone with API access to the Kubernetes cluster will be able to read the value. Additional steps are required to make sure secret values are encrypted. See https://kubernetes.io/docs/concepts/configuration/secret/ for more information on Kubernetes secrets.

Dash0 Dataset Configuration

Use the spec.export.dash0.datasetCopy property to configure the dataset that should be used for the telemetry data. By default, data will be sent to the dataset defaultCopy.

0123456789101112
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration
spec:
  export:
    dash0:
      endpoint: ingress... # see above

      dataset: my-custom-dataset # This optional setting determines the Dash0 dataset to which telemetry will be sent.

      authorization: # see above
        ...

Disable Self-Monitoring

By default, self-monitoring is enabled for the Dash0 Kubernetes operator as soon as you deploy a Das0 operator configuration resource with an export. That means, the operator will send self-monitoring telemetry to the Dash0 Insights dataset of the configured backend. Disabling self-monitoring is available as a setting on the Dash0 operator configuration resource. Dash0 does not recommend to disable the operator's self-monitoring.

Here is an example with self-monitoring disabled:

012345678
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0OperatorConfiguration
metadata:
  name: dash0-operator-configuration-resource
spec:
  selfMonitoring:
    enabled: false
  export:
    # ... see above for details on the export settings

Disable Dash0 Monitoring For a Namespace

If you want to stop monitoring a namespace with Dash0, remove the Dash0 monitoring resource from that namespace. For example, if you want to stop monitoring workloads in the namespace my-nodejs-applicationsCopy, use the following command:

kubectl delete --namespace my-nodejs-applications Dash0Monitoring dash0-monitoring-resourceCopy

or, alternatively, by using the dash0-monitoring.yamlCopy file created earlier:

kubectl delete --namespace my-nodejs-applications -f dash0-monitoring.yamlCopy

Upgrading

To upgrade the Dash0 Kubernetes Operator to a newer version, run the following commands:

01
helm repo update dash0-operator
helm upgrade --namespace dash0-system dash0-operator dash0-operator/dash0-operator

Uninstallation

To remove the Dash0 Kubernetes Operator from your cluster, run the following command:

0
helm uninstall dash0-operator --namespace dash0-system

Depending on the command you used to install the operator, you may need to use a different Helm release name or namespace.

This will also automatically disable Dash0 monitoring for all namespaces by deleting the Dash0 monitoring resources in all namespaces. Optionally, remove the namespace that has been created for the operator:

0
kubectl delete namespace dash0-system

If you choose to not remove the namespace, you might want to consider removing the secret with the Dash0 authorization token (if such a secret has been created):

0
kubectl delete secret --namespace dash0-system dash0-authorization-secret

If you later decide to install the operator again, you will need to perform the initial configuration steps again:

1. set up a Dash0 backend connection and
2. enable Dash0 monitoring in each namespace you want to monitor, see Enable Dash0 Monitoring For a Namespace.