How to monitor and debug Terragrunt & Terraform/OpenTofu using OpenTelemetry

Introduction

Terraform, OpenTofu, and Terragrunt are popular infrastructure management and deployment tools.

Terraform, developed by HashiCorp, is a widely-used Infrastructure as Code (IaC) tool that allows users to define, provision, and manage cloud resources across multiple platforms using a declarative language called HashiCorp Configuration Language (HCL). HashiCorp was bought in 2024 by IBM and changed the Terraform license.

OpenTofu is an open-source fork of Terraform and now part of the Linux Foundation. It maintains backward compatibility with Terraform, allowing for seamless transitions and supporting existing workflows.

Terragrunt is a thin wrapper around Terraform/OpenTofu, that provides additional features to improve infrastructure management. It promotes DRY (Don't Repeat Yourself) configurations, simplifies remote state management, and offers versioning capabilities for Terraform/OpenTofu modules.

We use Terragrunt for managing all of our static cloud infrastructure. To help with debugging and improving our Terragrunt workflow we are sending OpenTelemetry logs, metrics and traces from Terragrunt to Dash0.

This blog post explains how to configure Terragrunt to collect OpenTelemetry data when running everything on your local machine. We use Atlantis to plan & apply all Terragrunt/Terraform changes via GitHub pull requests. We will explain that setup in a future blog post.

Getting started

Currently Terragrunt only supports sending telemetry data to a locally running OpenTelemetry collector. At Dash0 we run Atlantis on an EC2 instance with a collector to collect all Terragrunt metrics, logs and traces.

Diagram showing local setup of Terraform, Terragrunt and the OpenTelemetry collector

Prerequisites

To experiment with the tools in this blog post you need:

Docker installed locally
Existing Terragrunt project
Terragrunt version 0.72.0 or newer
Terraform
A free Dash0 trial account, https://www.dash0.com/sign-up

Step 1: Run OpenTelemetry collector

Please download this OpenTelemetry collector YAML configuration first and save it locally as config-blog-post.yaml, https://www.otelbin.io/s/8fd6d3e8f31f48b04ae3b018bf691d4860815fc5

Then replace <AUTH_TOKEN> in config-blog-post.yaml with the auth token from your Dash0 organization. If you do not have an organization you can quickly spin up a free two week trial under https://www.dash0.com/sign-up

Then you can start the collector using the following docker command:

0
docker run -p 4317:4317 -p 4318:4318 -v $(pwd)/config-blog-post.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.114.0

Step 2: Set environment variables

Open a terminal and go to a Terragrunt directory where you would typically run your terragrunt commands. Set the following environment variables:

0123456
export TERRAGRUNT_TELEMETRY_TRACE_EXPORTER=http
export TERRAGRUNT_TELEMETRY_TRACE_EXPORTER_HTTP_ENDPOINT=localhost:4318
export TERRAGRUNT_TELEMETRY_TRACE_EXPORTER_INSECURE_ENDPOINT=true

export TERRAGRUNT_TELEMETRY_METRIC_EXPORTER=grpcHttp
export TERRAGRUNT_TELEMETRY_METRIC_EXPORTER_INSECURE_ENDPOINT=true
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

Step 3: Run terragrunt

Run terragrunt plan/apply commands. Terragrunt will automatically send all telemetry data to the locally running collector which will forward the data to the Dash0 OpenTelemetry backend. The following arguments make sure that terragrunt writes structured logs in JSON format.

012
terragrunt run-all plan \
--terragrunt-log-format=json \
--terragrunt-non-interactive

While running terragrunt, it will automatically send all OpenTelemetry data to the running collector instance. This collector then forwards the data to your Dash0 organization.

Using OpenTelemetry data from terragrunt

Support for exporting OpenTelemetry data was added at the beginning of 2024 to Terragrunt, see https://github.com/gruntwork-io/terragrunt/issues/2919

Here is the initial list of what data is being collected:

What commands are running and in which folders.
- Whether each command succeeded or failed.
- How long each command took.
What steps Terragrunt did internally (e.g., file system scanning, config parsing, auto-init, hook execution, etc) to process the command.
- How long each step took.
- Whether each step succeeded or failed.
What TF modules and providers, and at which versions, were used.
- How long it took to download them.
- How big they are on disk.

Example use cases

Example 1: Get error messages from spans in the tracing view

From the spans you get detailed information on errors with the full error message.

Error messages from spans in the tracing view

Example 2: Create dashboards for terragrunt exceptions

Based on the span data you can create charts and alerts when exceptions occur during terragrunt plan and apply lifecycles. Here is an example PromQL query:

01234567
max by (service_name, otel_span_event_name) (
  increase(
    {
      otel_metric_name = "dash0.span.events",
      service_name = "terragrunt"
    }[$__rate_interval]
  )
)

This is what a chart would look like for the PromQL query above.

Dashboards for terragrunt exceptions

Example 3: Get notified when changes to resources are applied

012345678
sum by (service_namespace, service_name, dir, args) (
  increase(
    {
      otel_metric_name = "dash0.spans",
      service_name = "terragrunt",
      args =~ ".*apply.*"
    }[$__interval]
  )
)

Example 4: Get notified when changes to critical production resources are applied

This is useful when you want to be notified when critical resources are updated, like the Kubernetes cluster.

0123456789
sum by (service_namespace, service_name) (
  increase(
    {
      otel_metric_name = "dash0.spans",
      service_name = "terragrunt",
      dir =~ ".*production.*eks_cluster.*",
      args =~ ".*apply.*"
    }[$__interval]
  )
)

Example 5: Terragrunt debugging metrics

In our integration hub we added a Terragrunt dashboard that is using most of the metrics that are available. The dashboard helps you identify which versions of Terragrunt are used, to make sure everything is consistent. Additionally it helps understanding the durations of Terragrunt runs as well as if the various caches are used and working correctly.

Terragrunt debugging metrics dashboard

Spans

You can easily find all Terragrunt spans by using the following filters:

service.name = terragrunt

deployment.environment.name = terragrunt-local

Then you can find only Terragrunt related traces and slice and dice through them using Dash0.

Span overview showing all Terragrunt spans in the current timeframe

Metrics

You can find all Terragrunt metrics data in the metrics explorer by using the following filters: