AutoMerge is generally available

Changelog

Recent changes and improvements to Dash0

Join us on a journey through Observability and OpenTelemetry as we improve our product. Stay connected with our Newsletter or RSS for fresh updates.

AutoMerge is generally available

AutoMerge, Dash0's approval agent, is out of beta. It scores the risk of every pull request and approves the low-risk ones, so your reviewers spend their time on the changes that need them.

The AutoMerge overview of a repository in Auto-merge mode, with tiles for auto-approved pull requests, intervention rate, time to merge and agreement with humans, auto-approved pull requests over time, criterion scores against the passing threshold, and merged pull requests by type

AutoMerge is Dash0's approval agent. It scores the risk of every pull request and approves the low-risk ones, so your engineers spend their review time where it matters and your team ships faster without giving up control. It has been in beta since 6 November, and from today it is generally available.

AutoMerge checks each pull request against the rules you set for its repository: protected paths, criteria that AI evaluators score, and a threshold. A pull request that AutoMerge does not approve goes through your regular review.

Start in Dry run

Connect GitHub or GitLab and pick your repositories. In Dry run, AutoMerge comments its decision on every pull request and approves or merges nothing. The comment shows the combined score against your threshold, the score and weight of each criterion, and the evidence behind each score. Its comments already help your reviewers, before you let it approve or merge anything.

An AutoMerge comment on a GitHub pull request in Dry run. It would approve with a score of 88.8 out of 100 against a threshold of 75, lists five criteria with their weights and scores, and explains the Blast radius score of 96

Move up when it has earned it

Dash0 compares every decision with what your reviewers did next and shows the result as Agreement with humans. When the decisions match your reviewers, switch the repository to Auto-approve, where AutoMerge approves and a person merges, or to Auto-merge, where AutoMerge approves and merges. Path rules give a path its own level. Mark migrations, infrastructure or billing code as Protected, and a pull request that touches them always goes to a person.

Each decision is on the repository's Pull requests tab, with the verdict, the deciding criterion and the score of every criterion. If a decision was wrong, say so with "Was this decision right?".

The decision record of an approved pull request: a combined score of 88.2 against a threshold of 70, the deciding criterion, and the score of each of four criteria

Your own criteria

Every repository starts with three criteria: Blast radius, Evidence quality and Change footprint. Change them or add your own. Each criterion has its own AI evaluator that scores the pull request from 0 to 100 and explains the score. You write its prompt, pick the tools it may use to read the code, the pull request and the history of the repository, set its share of the combined score, and set a floor below which it rejects the pull request by itself. You also set the threshold for the repository. Improve configuration asks Agent0 to propose a change to the configuration, which you preview before you apply it.

The criteria of a repository's AutoMerge configuration, with one criterion open to show its name, its 40% share, its rejection floor of 60, its prompt and the tools it may use, and the Improve configuration button

Test a change before it runs

The Testing tab replays a configuration change on past pull requests. It shows which decisions change and how the new decisions compare with your reviewers, while the running configuration stays as it is.

A test result on the Testing tab that compares configuration v4 with v1 on 25 recent merges, with 8 changed decisions and the outcomes approved and agreed, approved but should not have, held and agreed, and held but could have merged

Pricing

Each pull request that AutoMerge scores costs 1 AI credit, regardless of the number of evaluations. Dry run is charged like the other modes, because AutoMerge still scores the pull request. A test costs 1 AI credit for each pull request it replays. To stop the charges for a repository, set its mode to Off.

Organizations that existed before today get 14 days of AutoMerge free, from the first pull request it scores. The Dash0 trial covers new organizations. When your organization reaches its monthly AI credit limit, AutoMerge stops scoring and says so on the pull request.

Supported

AutoMerge works with repositories connected through the GitHub integration or the GitLab integration. On GitLab it scores merge requests the same way it scores pull requests. On GitHub, an admin approves the permissions AutoMerge needs for the GitHub App. On GitLab, the service account's token needs the api scope, so that AutoMerge can comment, approve and merge.

Read the AutoMerge documentation, or follow Set Up AutoMerge to start your first Dry run.

Read more

Investigate Kubernetes workloads in k9s

Find out whether a Kubernetes performance problem affects one pod or an entire workload. Explore resource trends, request errors and latency directly in k9s with Dash0.

When a deployment starts having problems, current CPU and memory usage only tell part of the story. Has resource usage been climbing? Is one replica struggling while the others are healthy? Are requests getting slower or failing?

The Dash0 CLI answers them in k9s. Explore up to 24 hours of CPU and memory trends alongside request rates, errors, latency and log counts by severity. No need to leave your terminal and find the right dashboard.

The Dash0 CLI page for a deployment of three pods in k9s, in per-pod view over 1 hour: memory at 706 MB with the busiest pod at 85% of its limit, and CPU and memory charts with one line per pod.

Compare individual pods to spot uneven resource usage, or switch between total, average, minimum and maximum values for a workload. The view refreshes every 30 seconds, and shortcuts open the corresponding metrics, traces and logs in Dash0 for further investigation.

You can also run the commands directly in your terminal. JSON output provides the same time series and Dash0 links for scripts and coding agents.

sh
dash0 -X k8s pods watch <pod> -n <namespace>
dash0 -X k8s deployments watch <deployment> -n <namespace> --per-pod
dash0 -X k8s pods watch <pod> -n <namespace> -o json

The commands are experimental and require the -X flag. They use Kubernetes metrics collected by the Dash0 operator.

To get started, add the k9s plugin configuration to your plugins.yaml. The example uses Shift-D as the shortcut. See the Dash0 in k9s guide for installation and setup.

Read more

Get an email when a notification channel stops delivering

Dash0 now emails your organization's admins when a notification channel stops delivering alerts. The email names the channel and the reason, and links to the channel so you can fix it.

Until now, a notification channel that stopped delivering showed its problem only inside Dash0, in the channel's delivery status. If a Slack app lost access or a webhook token expired, your alerts could stop arriving before anyone opened the settings and noticed.

Now Dash0 emails every admin of your organization when one of its notification channels stops delivering:

  • When it is sent. The channel's delivery status is Stopped or Refused, or alerts to the channel kept failing for 2 hours.
  • What it says. The channel, its type, when the problem started, the last failure, and the reason.
  • How to fix it. Open channel settings takes you to the channel. Fix the destination, then click Send test notification.
  • Once per problem. Each broken channel gets one email. Dash0 emails again only if the channel recovers and later stops delivering again.
  • No email while you set up a channel. Dash0 waits 30 minutes after a change to a channel's configuration, and skips disabled channels.

Read the docs: Monitor Notification Channel Delivery

Read more

Explore your Kubernetes pods, live or historical

Kubernetes gets its own section under Infrastructure: a live map of every pod, a pod table that takes k9s keys, and a side panel that shows why a pod failed.

Kubernetes used to be one resource type among many on the Resources page. It now has its own section under Infrastructure, built around the question you open it with: which pods are unhappy, and why. The pages run live by default, and the same control rewinds them to any past window, so you can ask that question of the cluster as it is now or as it was when it broke.

The Kubernetes Map view, with pods grouped into two clusters and colored by health, and the Needs attention list beside it

Start wide, then zoom in

The Map draws every pod in every cluster, grouped by cluster and namespace and colored by health. Click a cluster to zoom into it, a namespace to go further, a pod to open it. Beside the map, Needs attention says what is wrong and how many pods it affects: failed, pending, in an unknown phase, not ready, near the CPU limit, near the memory limit.

The Table trades the overview for detail. Every pod is a row carrying its namespace, its age, its status, how often it has restarted, how much CPU and memory it is using, and how much of its limit that is. Group the rows by cluster, namespace or node, and filter them by name, cluster, namespace, node or IP.

The Kubernetes pod table sorted by status, with three pods in Error at the top above running pods, and namespace, age, restarts and CPU usage columns

The k9s keys, in the browser

The table takes the keys you already type. j and k move, g and Shift+G jump to the ends, Enter or d opens a pod, y its YAML, l its logs, e its events, o its node, p its parent workload, c copies the name and / filters. Shift+N, Shift+A, Shift+S, Shift+C and Shift+M sort by name, age, status, CPU and memory.

Why that pod failed

Open a pod and the side panel answers it without sending you anywhere else.

  • Containers gives each container's current state and, once object collection is on, its last termination, with the reason, exit code and signal.
  • Events sits next to the CPU, memory and network charts. Kubernetes records no event when a container is killed, so Dash0 reconstructs those rows from the container's status metrics and marks them as approximate.
  • Logs are filtered to the pod, and to one container when you pick one. A container that terminated without writing anything says so, and links to the event that explains it.
  • Traces group into operations with p50, p95 and error counts, and open in place.
  • Related resources walks the cluster, its nodes and the namespace's pods on them, and the workload down through its ReplicaSets.
The pod side panel for a failed pod, with the related resources tree showing other failing pods on its node, the hover card on its container in Error with the reason and exit code, and an events table whose first row is the reconstructed container termination

The YAML, as it is in the cluster

The YAML tab shows the object's manifest as it was last collected, stamped with when. An object that has since been deleted still shows its last known state.

Manifests reach Dash0 as log records from the OpenTelemetry Collector's k8sobjects receiver, so any collector you already run can send them. The Dash0 operator runs that receiver for you once you turn Kubernetes object collection on for the cluster and for the namespaces you want manifests from. Before anything leaves your cluster, it masks the values of environment variables and removes the annotations that kubectl and similar tools use to store a serialized copy of the whole manifest.

The YAML tab of a failed pod, showing its collected manifest with line numbers and an Observed timestamp

Live, or any moment you choose

The pages follow your cluster as it changes. Pause them when you want to read a row without it moving under you, or pick a past window, such as yesterday or this day last week, and see the cluster as it was then, with a banner saying so and one click back to live.

Workloads, Nodes and Storage are next.

Read more
Mateusz Wojciechowski

Mateusz Wojciechowski

Shani Shiri

Shani Shiri

Tomas Zaicevas

Tomas Zaicevas

Evgeni Wachnowezki

Evgeni Wachnowezki

Andrea Chomiak

Andrea Chomiak

Simo Paraschi

Simo Paraschi

Add automations to your check rules

Link existing automations or create one from a Dash0 template without leaving the check rule, and choose how the alert and the automation's result are delivered.

When a check fails, an automation can investigate it for you. Until now, you set one up on the Automations page and then had to work out which check rules it covered.

A check rule now has an Automations section after its Notifications. It lists the automations that run when the rule fails. You can link an existing automation, or create one from a Dash0 template, such as Investigate, without leaving alerting. Before you create it, the template panel shows how many runs and credits to expect each month. If an automation runs for every failed check or matches by label, you can stop it for a single rule. Linking works even on rules you can only read, as long as you can create automations.

For each pair of check rule and automation, you choose how the alert and the automation's result are delivered:

  • Same notification channel, when automation finishes: the alert goes out as usual, and the result follows as a new message. This is the default.
  • Hold off the alert until the automation completes: one alert goes out, with the result, when the run finishes.
  • Different notification channels: the alert goes to the check rule's channels, and the result goes to the automation's Send to.

A held alert is still sent if the run is skipped or fails. Results follow silences, so a silenced failed check does not leak into your channels.

See About Automations to learn more.

Read more

Bulk silence failed checks

Select failed checks and silence them together, up to 200 in one silence. The Silences Hub shows a bulk silence as one row.

Until now, a silence covered one failed check. When one problem failed many checks, you had to silence each one separately.

Now you can silence many failed checks at once:

  • Select and silence together. In the failed checks list, select the failed checks and click Silence. Choose how long, and add a reason, which is required. One silence covers up to 200 failed checks.
  • Review before you confirm. The dialog lists the failed checks that will be silenced. It flags the failed checks that have resolved, and Remove all takes them out. Failed checks that are already silenced show Silenced already and stay out of the silence unless you tick them.
  • See a bulk silence as one row. Its row in the Silences Hub shows how many failed checks it covers. Expand the row to see each failed check, and click one to open its detail page.
  • Remove a failed check from a silence. In the expanded row, hover over a failed check and click its trash icon, Remove from silence. Its notifications resume, and the other failed checks stay silenced. Removing the last failed check ends the silence.
  • Add to an active silence. Select failed checks and click Add to silence. Pick an active silence that covers failed checks, review the failed checks, and add them. The silence keeps its end time.
  • End silences for a selection. Select failed checks and click End silence. The dialog lists them under Will stop being silenced. Only the failed checks you leave ticked stop being silenced. A silence that also covers other failed checks keeps silencing them.
  • Silenced now or earlier. In the failed checks list, a failed check that a silence covers right now shows Silenced now. One that a silence covered earlier in the selected time range, but not now, shows Was silenced.
Silence failed checks dialog for five selected failed checks, with a duration, a required reason, a note that 1 failed check has resolved with a Remove all button, and one failed check tagged Silenced already

Read the docs: Silence Failed Checks

Read more

Latency SLOs on any histogram

Add a duration to an SLO and count only the requests that succeeded within it. It works on span metrics and any cumulative histogram, and you do not write any PromQL.

The Good events card of the SLO form for the demo-webshop-search service on dash0.spans.red. Duration is set to at most 300 ms, the success criteria is otel.span.status.code != ERROR, and the PromQL preview shows the histogram_fraction query the form writes.

A request that succeeds after eight seconds is still a bad experience, but an error-rate SLO counts it as good. To build a latency SLO in Dash0 until now, you wrote the PromQL yourself, with a bucket filter that had to match a bucket bound exactly or a histogram_fraction query.

The SLO form now has a Duration row in the Good events and Bad events cards. Pick a histogram metric, such as dash0.spans.red or http.server.request.duration, and type a duration like 300 ms. A request then counts as good only when it matches your success criteria and finishes within that time. The form writes the query and shows it as you type. It works with span metrics, OpenTelemetry histograms with fixed buckets, and native or exponential histograms.

See the documentation to set one up.

Read more

Start Agent0 automations from changes in Linear

Automations now start when issues, comments, projects, initiatives or releases change in Linear, so Agent0 picks up the work without anyone handing it over.

Work tracked in Linear still waits for someone to notice it, read it and start on it.

Automations can now start when something changes in Linear: an issue is created or updated, a comment, attachment or reaction is added, a project or initiative gets an update, or a release goes out. Narrow the trigger by team, project, label, status, author or name, with exact values or regular expressions. Anything that doesn't match is skipped. When an event matches, Agent0 reads the item and does what the automation asks, such as opening a pull request with a fix, building a dashboard, or posting a root-cause report to Slack.

For example, an issue trigger that listens for the In Progress status starts Agent0 as soon as your team picks up a ticket.

See the documentation to set one up.

Read more

Instrument pods instead of workloads with the Dash0 operator

The operator can now instrument pods at creation and leave your deployments untouched.

Starting with version 0.160.0, the Dash0 operator can instrument pods when they are created, instead of modifying the pod templates of deployments, stateful sets and other workloads.

To switch a namespace over, set spec.instrumentWorkloads.mode: pods in its Dash0 monitoring resource. Since the workload resources are never modified, tools like Argo CD, Flux or Terraform no longer see drift caused by the operator. Pods created by other controllers, for example Argo Rollouts, are instrumented as well.

The operator doesn't restart pods to instrument them: existing pods pick up the instrumentation the next time they are recreated. The mode is set per namespace, so you can switch namespaces one at a time. We recommend running two operator replicas (operator.replicaCount=2) so that the webhook stays available during operator updates.

For details, see the documentation.

Read more

Continuous profiling in Dash0

Find the code behind resource consumption with OpenTelemetry profiles. Filter by workload, compare profiles, and investigate CPU time, memory allocations, and other measurements.

Resource charts show which workloads consume resources. Profiles show the code paths behind that consumption.

Dash0 now has a Profiling Explorer for investigating OpenTelemetry profiles. Analyze CPU time, memory allocations, and other measurements in the same explorer. Dash0 supports all OpenTelemetry profile types; the types available in your dataset depend on what your profiler collects and sends.

Dash0 Profiling Explorer showing workload filters, an On-CPU time-series chart, and an aggregate flame graph

From workload to code

Filter profiles by workload attributes, such as Kubernetes namespace, deployment, or pod. Split the chart into series with Sum by to find the workloads contributing most to the selected measurement, then inspect their call stacks.

Choose the view that answers your question:

  • Flame Graph: Find expensive call paths across the selected samples. Reverse the call-stack order to bring leaf functions together across their callers and find the code doing the work directly.
  • Table: Search for a function and compare its Self contribution with its Total, which includes its callees. Values follow the selected profile's units, whether time, bytes, or counts.
  • Flame Chart: Inspect samples along a time axis and group them by attributes to investigate bursts or sustained activity.
  • Sandwich: Follow the sampled callers and callees around a selected frame to understand which operations lead into it and where it delegates work.

Compare profiles and focus on the difference

Choose A Baseline and B Comparison to compare datapoints over time or different workloads. A differential flame graph shows which call paths contribute less or more in B: green for less, red for more.

Separate baseline and comparison On-CPU charts above a differential flame graph with green and red frames

Use stack filters to keep relevant call paths, or frame filters to hide individual frames, such as runtime internals. These filters change the code view, not the stored profiles.

Compare the same profile type and units, and account for differences in traffic and workload scope before treating a change as a regression or an improvement.

Bring your OpenTelemetry profiles

On Kubernetes, the Dash0 Operator profiling integration can deploy an opt-in managed profiler or forward profiles from a separately deployed profiler. Profile collection must be enabled separately from tracing and metrics instrumentation.

Open Telemetry → Profiling, choose a dataset containing profiles, and select a time range and Profile type to start investigating.

Learn more about continuous profiling in Dash0.

Read more

Continuous profiling with the Dash0 Operator

The Dash0 Operator can now deploy and manage the Parca eBPF profiler for you and collect CPU profiles from your monitored workloads, without any code changes.

Traces tell you which request was slow. Profiles tell you which function burned the CPU time. Until now, getting profiles from Kubernetes into Dash0 meant deploying an eBPF profiler yourself and wiring it up to the collectors of the Dash0 Operator.

Starting with version 0.159.0, the Dash0 Operator does this for you. It runs the Parca agent as a DaemonSet, sends its profiles through the operator's collector on the same node, and exports them to Dash0.

What you get

  • No code changes: The profiler uses eBPF to sample the processes on each node. There is no library to add and no image to rebuild.
  • Broad runtime coverage: Native code such as Go, Rust and C/C++, as well as Java, Python, Node.js, .NET, Ruby, PHP, Perl, Erlang/Elixir and LuaJIT.
  • Kubernetes context: The operator enriches profiles with Kubernetes metadata, just like your traces, metrics and logs. Profiles follow the same namespace selection: the operator keeps workload profiles only from namespaces that have a Dash0Monitoring resource, and the filter and transform settings of these resources apply to profiles as well.
  • Readable native stack traces: The profiler uploads debug information of native binaries to Dash0 through a proxy managed by the operator, so that Dash0 can resolve native frames to function names. The proxy uses the same credentials as your Dash0 exports.

How do you try it?

Upgrade the Dash0 Operator for Kubernetes to v0.159.0 or later and set operator.ebpfProfilerEnabled=true when installing or upgrading the Helm chart. If you manage the Dash0OperatorConfiguration resource yourself instead of via Helm, also set spec.profiling.enabled and spec.profiling.ebpfProfiler.enabled to true there.

The profiler is opt-in and off by default. Sampling frequency, off-CPU sampling, the enabled runtimes, resources and scheduling are configurable on a self-managed Dash0OperatorConfiguration resource. See the documentation for all options.

Since the agent runs with privileged access to your nodes, review the security requirements before you enable it. Support for GKE Autopilot will follow in a later release.

Read more

AutoMerge approves your pull requests

AutoMerge scores every pull request with AI evaluators and approves the ones that clear your rules. It starts in Dry run, where it only comments its decision.

Coding agents made writing a change cheap. Reviewing it did not get cheaper, so pull requests wait for a person, including the routine ones nobody needed to read line by line.

AutoMerge, now in beta in Darkplane, takes that routine share off your reviewers. AI evaluators score each pull request against criteria you define. Each criterion returns a score from 0 to 100 with a rationale. When the weighted score reaches your threshold and no criterion falls below its floor, AutoMerge approves. Everything else goes to a person.

Darkplane AutoMerge score cards for a pull request routed to a human and an approved pull request

Every repository starts with three criteria: blast radius, evidence quality, and change footprint. Treat them as a starting point and fit them to your repository:

  • Add the questions your reviewers ask, and remove the ones that don't fit.
  • Write the prompt each evaluator follows, and choose the tools it may use to read the repository and the pull request.
  • Weight the criteria by what matters most, and set a floor that rejects a pull request on one criterion alone.
  • Tell the evaluators what the code doesn't show, such as which directories carry the highest consequences.

You decide how much it may do

Each repository runs in one of four modes: Off, Dry run, Auto-approve, or Auto-merge. Path rules give specific paths their own level. Mark migrations, infrastructure, or billing as Protected, and a pull request that touches them goes to a person without being scored. A pull request always runs at the least autonomous level of any file it touches.

AutoMerge fails closed. If a criterion can't produce a score, nothing is approved. It doesn't approve a commit whose CI is failing, and it never approves a pull request again after a reviewer dismissed its approval. Its check is advisory, so it blocks no merge unless you make it a required check in GitHub.

Start in Dry run

In Dry run, AutoMerge comments its decision on each pull request and approves or merges nothing. Dash0 grades every decision against what your reviewers did next and shows the result as Agreement with humans. Once its decisions match your reviewers, switch the repository to Auto-approve. Before you change criteria or the threshold, test the change against past pull requests on the Testing tab.

The AutoMerge overview of a repository in Auto-merge mode, with tiles for auto-approved pull requests, intervention rate, time to merge, and agreement with humans, charts of auto-approvals and criterion scores over time, a breakdown of which criterion decided each outcome, and the repository's path rules

AutoMerge works with repositories connected through the GitHub integration. Read Set Up AutoMerge to start your first Dry run, or see the AutoMerge overview.

Read more