• 3 min read

Spam filters: the point-and-click way to stay within your observability budget

OpenTelemetry, Prometheus and other open-source projects have made telemetry open, easy to collect and plentiful. Too plentiful, sometimes: complaints about observability budgets out of control are everywhere and all at once.

The OpenTelemetry Collector and SDKs have several mechanisms to allow you to collect less telemetry, but the configurations can be toilsome and error-prone. Wouldn’t it be easier if you could just analyze which telemetry you do not want to pay for, the same way you analyze it to troubleshoot issues?

An end to spammy telemetry

About a year ago, way before we officially launched Dash0 (my, doesn’t time fly?!?), we published our vision for what makes an observability tool truly OpenTelemetry-native. And in our vision, there is the following:

“OpenTelemetry-native tools should provide easy-to-use facilities for users to balance their needs with the costs of processing and storing the telemetry. For example, the tool should provide an easy way to configure telemetry sampling to control costs. This goes beyond the common head/tail sampling by leveraging correlations between signals (e.g., exemplars) to retain as much meaning as possible while reducing the volume of stored telemetry.”

We believe in the paragraph above every bit as the day we wrote it. And today, we follow through with Spam Filters. With two clicks, you can define filtering rules, executed by the Dash0 ingestion pipeline, to drop logs, metric data points and spans before Dash0 charges you for it:

With Dash0 powerful filtering capabilities, it’s trivial to select all Kubernetes probes that succeeded. We want to know only about the ones that fail, and throw away the rest.
One click, and the rule is ready. Press the Flag as spam button, and no more spans like these will make it to your Dash0 bill!

It of course works also for log records and metrics:

Flagging logs as spam in Dash0.

The case of metrics is especially interesting: sometimes, what you want is to drop the entirety of a metric, and in that case you flag it as spam by its name. Some other times, you care about that metric only in specific circumstances and, in a very Dash0-y fashion, you drop all metric datapoints where deployment.environment.name!=prod.

Flagging metrics by name as spam in Dash0; it also works to drop metric datapoints by specific attribute values!

Spam filters are specified at the level of the Dash0 dataset, and can be removed as easily as they can be created:

The new spam filter in your organization settings.

Do try this at home

It will not surprise you to know that we build spam filters so that they are compatible with OpenTelemetry best practices, and specifically with the way you can perform similar filtering using the OpenTelemetry Collector. With one button, you can see equivalent configurations in OTelBin:

It turns out that what is a couple of filters in Dash0, maps to a complex OpenTelemetry Transformation Language (OTTL) expression. It could be simplified, if we made assumptions about where in your telemetry are the correct attributes; but better play it safe.

You can replicate the same filters to your own setup, and be doubly sure that no unwanted telemetry will be billed for at the end of the month!

What’s next?

We have a lot more ideas about what we could do to give you better control of your telemetry. We won’t say much now, because we prefer to first ship, then let Dash0 speak for us.

One thing, however, is certain: you deserve as much control as possible in all aspects of your observability. At Dash0, we are committed to giving this control to you, nicely wrapped in a lovable user experience, with deep compatibility with the open-source ecosystem, and without bill shock at the end of the month.

Just the way it should be.

    Related Reads