Filtering collected telemetry in the Dash0 operator

In the same vein as the Spam filters you find in Dash0, we exposed the possibility of configuring the OpenTelemetry Collector filter processor via the Dash0MonitoringCopy resource in the Dash0 operator.

With the new filter capabilities of the Dash0 operator, you can discard some telemetry before it even leaves your cluster:

dash0-monitoring.yaml
123456789101112131415161718192021222324
apiVersion: operator.dash0.com/v1alpha1
kind: Dash0Monitoring
metadata:
  name: dash0-monitoring-resource
spec:
  filter:
    traces:
      span:
      - 'attributes["http.route"] == "/ready"'
      - 'attributes["http.route"] == "/metrics"'
      spanevent:
      - 'attributes["grpc"] == true'
      - 'IsMatch(name, ".*grpc.*")'
    metrics:
      metric:
      - 'name == "k8s.replicaset.available"'
      - 'name == "k8s.replicaset.desired"'
      datapoint:
      - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
      - 'resource.attributes["service.name"] == "my_service_name"'
    logs:
      log_records:
      - 'IsMatch(body, ".*password.*")'
      - 'severity_number < SEVERITY_NUMBER_WARN'

The configurations above will apply only to telemetry coming from the namespace in which this Dash0MonitoringCopy resource resides.

You can also export the OpenTelemetry Transformation Language (OTTL) expressions from your Spam filters in Dash0, and apply it to the relevant namespaces in a “belt and suspenders” kind of fashion.

Having the option to filter telemetry both at the Dash0 backend as well as directly in your Kubernetes cluster might seem redundant, but it is actually useful for a couple of reasons:

• Using the spam filter in Dash0 is a quick and easy way to remove telemetry that has little or no value, and to reduce your Dash0 bill.
• However, if some components in your cluster really produce a lot of noisy telemetry, you would still pay for egress as the data is sent to Dash0, only to have Dash0 discard it immediately.
• Filtering directly in your cluster enables you to optimize your cloud costs further by removing those egress costs as well.
• Filtering in your cluster can also help remove sensitive information before it leaves your cluster – the log record filter above which drops lines with “password” is an example for that. (Ideally, telemetry with sensitive information wouldn’t even be produced at all, but mistakes happen.)

Note: You should be able to reuse the configurations you may already have (or find on the Internet or generate via AI) for the filter processor of the OpenTelemetry Collector.