Amazon CloudWatch Transaction Search (X-Ray)

Overview

Integrating CloudWatch Transaction Search with Dash0 involves two main steps:

1. Enable Transaction Search: Configure AWS to capture 100% of spans and store them in the aws/spans CloudWatch log group
2. Set up Firehose delivery: Create a Firehose stream and subscription filter to forward span data from the aws/spans log group to Dash0

This integration allows you to analyze your complete distributed traces in Dash0 while maintaining cost-effective span collection through CloudWatch.

Prerequisites

Before starting, ensure you have:

• Applications instrumented with OpenTelemetry or X-Ray that are sending traces or AWS Services with X-Ray tracing enabled
• Appropriate IAM permissions to enable Transaction Search and manage CloudWatch Logs
• A Dash0 organization with access to authentication tokens

Step 1: Enable Transaction Search

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
2. From the navigation pane, under Settings, choose X-Ray traces.
3. Choose Transaction search > View settings.

4. Click on Edit to configure transaction search.
5. Select the checkbox to ingest spans as structured logs.
6. Enter a percentage of spans to be indexed as trace summaries (1% is recommended for most use cases and is free).

7. Click Save to activate Transaction Search.

Note: After enabling Transaction Search, it can take up to 10 minutes for spans to become available in the aws/spans log group.

Step 2: Create a Firehose Delivery Stream

Follow these steps to create a Firehose delivery stream that will send span data to Dash0:

1. Choose Source and Destination

1. Open the Amazon Data Firehose console.
2. Choose Create delivery stream.
3. Choose Direct PUT as the source.
4. Choose HTTP Endpoint as the destination.

Or open this direct link to create a Firehose stream which preselects the HTTP Endpoint destination.

2. Configure Stream Settings

1. Give your Firehose stream a descriptive name (e.g., dash0-cloudwatch-spans).
2. Leave the other source settings as default.

3. Configure Destination Settings

In the HTTP Endpoint URL field, enter the AWS CloudWatch Spans via Firehose HTTP endpoint from your Dash0 organization settings under Endpoints:

Authentication:

Create a Dash0 authentication token for the Firehose stream. By default, data will be sent to the default dataset. If you want to send data to another dataset, you can restrict the token access to that specific dataset. It's recommended to grant only the ingestion permission to this token.

You can either:

• Choose Use access key and enter the authentication token directly as the Access Key:
• Use AWS Secrets Manager to retrieve the token programmatically.
  • the secret must be a key-value pair with the key api_key and the value being your Dash0 authentication token.

4. Configure Advanced Settings

• Compression: Data sent from CloudWatch Logs to Firehose is already compressed with gzip, so additional compression is not needed.
• Buffering: You can keep the default buffering settings or adjust based on your needs.

5. Configure Backup Settings

Create or choose an existing S3 bucket to store data in case of delivery failures.

6. Review and Create

Review your settings and click Create Firehose stream.

NOTE: You might get an error on the first attempt to create the Firehose stream. Please try again after a few seconds by clicking the "Create Firehose stream" button again.

Step 3: Create a CloudWatch Subscription Filter

Now you'll create a subscription filter to forward spans from the aws/spans log group to your Firehose delivery stream.

1. Create IAM Role for CloudWatch Logs

Create an IAM role that allows CloudWatch Logs to send data to Firehose:

1. Go to IAM > Roles and create a new role.
2. Choose Custom trust policy and use this trust policy:

Note: Replace <region> and <account-id> with your actual values.

3. Attach the AmazonKinesisFirehoseFullAccess policy, or create a custom policy with minimal permissions:

Note: Replace <region>, <account-id>, and <delivery-stream-name> with your actual values.

4. Name the role (e.g., CloudWatchLogsFirehoseRole) and create it.

2. Create the Subscription Filter

1. In the AWS console, go to CloudWatch > Logs > Log groups.

2. Find and select the aws/spans log group.

   Note: This log group is automatically created when you enable Transaction Search. If you don't see it, ensure Transaction Search is enabled and wait up to 10 minutes.

3. Click Actions > Create subscription filter > Create Amazon Data Firehose subscription filter.

4. In Choose destination, select the Firehose delivery stream you created earlier.

5. In Grant permission, select the IAM role you created.

6. Configure the filter:

   • Log format: Choose Other
   • Filter pattern: Leave empty to forward all span data, or use a filter pattern to filter specific spans

7. Give your subscription filter a name and click Create.

Enable System Fields for Multi-Account/Region Attribution (Recommended)

When sending spans into a Firehose delivery stream, you should enable system fields to ensure proper resource attribution in Dash0. This adds cloud.account.id and cloud.region attributes to your spans, making it easy to identify the source account and region for each span.

Step 4: Set retention policy for the aws/spans log group

To manage costs and avoid excessive data retention, set a retention policy for the aws/spans log group:

1. In the CloudWatch Logs console, select the aws/spans log group.
2. Click on Actions > Edit retention.
3. Choose a retention period (e.g., 1 day) to automatically delete old spans.
4. Click Save.

Verification

To verify your setup is working:

1. Check span ingestion: In CloudWatch Logs, verify that the aws/spans log group is receiving span data.
2. Monitor Firehose delivery: Check your Firehose delivery stream metrics for successful deliveries.
3. Verify in Dash0: Look for span data appearing in your Dash0 dashboard.

Explore Spans in Dash0

Once everything is configured, your CloudWatch Transaction Search spans will be available in Dash0 for advanced analysis. You can:

• Search spans by attributes, trace IDs, or business context
• Correlate spans with other telemetry data (logs, metrics)
• Analyze distributed traces with complete visibility
• Create dashboards and alerts based on span data

The spans will include all the rich metadata from your applications, including custom attributes, business context, and full trace hierarchy information in the OpenTelemetry format.