Instrumenting Spring applications with OpenTelemetry and Cloud Native Buildpacks

Containerizing applications is not always as straightforward as one might think. It is especially challenging to build high-quality container images that can be easily configured, maintained in the face of dependencies needing upgrades, and provide great security.

In this post we show a way to run Spring Boot applications in a well-built container image, will full OpenTelemetry support.

The content of this blog post is also available as a tutorial at: https://github.com/dash0hq/spring-demo

About…

This blog post combines three things: Spring Boot, Cloud Native Buildpacks, and OpenTelemetry. Let’s have a quick introduction to all three.

About Spring Boot

Spring Boot is the best way to:

… create stand-alone, production-grade Spring based Applications that you can 'just run'.

From Spring Boot

In a nutshell, Spring Boot is a framework with comprehensive facilities for configuring and building Spring-based applications written in Java or Kotlin (or both!) so that all you need to do to make them run, is execute java -jar.Copy

If you are not familiar with Spring or Spring Boot, then go check out the breathtaking dev-rel work of Josh Long a.k.a. @Starbuxman. We'll wait.

About Cloud Native buildpacks

Cloud Native Buildpacks provide a comprehensive set of building blocks to create professional-grade container images to run your applications. For example, consider something as simple as running a Java application in a container. It's not much, right? You make sure there is a JRE or a JDK in the container image, slap your executable Jar file (maybe built with Spring Boot) in it, set the entry point, and you are done, right?

Well, it depends. A simple Java application that does not need to serve much load, will be fine with that. But then, you might not have had to fine-tune the memory settings of the Java Virtual Machine that much. (In which case: count yourself lucky, although it's a lot of fun (TM) you are missing out on.) The Java cloud native buildpack can do that for you using the Java memory calculator, and then you just need to tweak a couple of relatable settings like thread count, as opposed to cracking out the calculator, a reference guide of the memory layout of Java virtual machines, and going to town on it. The Java cloud native buildpack also makes it very easy to set up SSL CA certificates, the necessary tooling to be able to remotely attach to your JVM for debug purposes with jattachCopy, and a bajillion other things.

Oh, and there is the OpenTelemetry buildpack to ensure that the OpenTelemetry Java agent is in the container image, ready to rock. And if you are now wondering "OK, what's OpenTelemetry", read on ;-)

About OpenTelemetry

What's OpenTelemetry? In short, OpenTelemetry is the best thing to happen in observability in a while:

OpenTelemetry (OTel) is an open-source observability framework. It solves the telemetry collection and transmission aspects of observability, but it is not a backend, analytics or visualization tool. OpenTelemetry is part of the Cloud Native Computing Foundation (CNCF), which also hosts projects like Kubernetes or Prometheus, and it is the second most active CNCF project just behind Kubernetes.

From What is OpenTelemetry?

OpenTelemetry provides specifications and high-quality implementations of ways to collect logs, metrics and distributed tracing data (a.k.a. spans) from your applications. The OpenTelemetry Java agent, specifically, is a one-stop shop to monitor your Java applications. And we are going to use it in this blog post to monitor a simple Spring Boot application with barely a few configurations.

Building container images with Spring Boot and Cloud Native Buildpacks

All the code referenced in this section, alongside instructions to try it on your own, is available at https://github.com/dash0hq/spring-demo.

Building a Spring Boot container image with Cloud Native Buildpacks

Cloud Native Buildpacks have a tool, called pack, to build container images with any cloud-native buildpack. But in our case, since we are focusing on Spring Boot, we can use the integration in the spring-bootCopy Maven plugin (which is also available for Gradle btw; but the author of this post is deeply nostalgic about the golden age of XML, in which “markup languages made sense”, and Maven scratches that itch). With the right configurations in pom.xmlCopy, building a container image using cloud native buildpacks is as easy as running:

./mvnw package spring-boot:build-imageCopy

Configuring which buildpacks to use

The build process works based on the configurations provided in the pom.xmlCopy, and specifically:

pom.xml
01234567891011121314
<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <configuration>
    <image>
      <buildpacks>
        <buildpack>gcr.io/paketo-buildpacks/java</buildpack>
        <buildpack>gcr.io/paketo-buildpacks/opentelemetry</buildpack>
      </buildpacks>
      <env>
        <BP_OPENTELEMETRY_ENABLED>true</BP_OPENTELEMETRY_ENABLED>
      </env>
    </image>
  </configuration>
</plugin>

The Maven configurations to build the container image with cloud native buildpacks. The author of this post needs more angular brackets in their life.

What Maven will do for us, is to build a container image using cloud native buildpacks that, will do three things:

1. Set up the container image to run Java applications shipped as executable Jar files, which is the preferred way of running Spring Boot applications
2. Add the OpenTelemetry Java agent and logic to configure it to run in the Java Virtual Machine inside the container
3. Add the Spring Boot application to the container image, so that it will be run on startup of the container

Anatomy of an image built with Cloud Native Buildpacks

The output of the build process is rather long, but the really interesting part is which layers are part of the container image:

spring-boot plugin build logs
012345678910
[INFO] [creator] Reusing layer 'paketo-buildpacks/ca-certificates:helper'
[INFO] [creator] Reusing layer 'paketo-buildpacks/bellsoft-liberica:helper'
[INFO] [creator] Reusing layer 'paketo-buildpacks/bellsoft-liberica:java-security-properties'
[INFO] [creator] Reusing layer 'paketo-buildpacks/bellsoft-liberica:jre'
[INFO] [creator] Reusing layer 'paketo-buildpacks/executable-jar:classpath'
[INFO] [creator] Reusing layer 'paketo-buildpacks/spring-boot:helper'
[INFO] [creator] Reusing layer 'paketo-buildpacks/spring-boot:spring-cloud-bindings'
[INFO] [creator] Reusing layer 'paketo-buildpacks/spring-boot:web-application-type'
[INFO] [creator] Reusing layer 'paketo-buildpacks/opentelemetry:helper'
[INFO] [creator] Reusing layer 'paketo-buildpacks/opentelemetry:opentelemetry-java'
[INFO] [creator] Reusing layer 'buildpacksio/lifecycle:launch.sbom'

The many, many layers of a container image built with cloud native buildpacks.

Container images are generally layered: the content of an image at runtime is effectively the “merge” of multiple archives containing the files that will end up in the container. Actually, a container image in the OCI Image format, the format used by every container image you likely come across, is pretty much a tar file with a specific structure, in which each layer is another tar file.

In cloud native buildpacks, pretty much each layer adds a functionality:

• certificate authority SSL certificates (paketo-buildpacks/ca-certificatesCopy)
• a Java Virtual Machine (paketo-buildpacks/bellsoft-libericaCopy, a build of the JVM that has been the go-to in the Cloud Foundry ecosystem) and JVM security settings
• facilities to run executable Jar files (paketo-buildpacks/executable-jarCopy)
• facilities to run Spring boot applications (paketo-buildpacks/spring-bootCopy)
• and last but not least, the OpenTelemetry Java agent (paketo-buildpacks/opentelemetryCopy)

All the buildpacks above, except for the opentelemetryCopy one, are added to the image by the Paketo javaCopy buildpack that combines them. Besides, depending on the build parameters given to the javaCopy buildpack, and not depicted in our example, it might also add other layers to provide, for example, monitoring facilities that are specific to Azure (paketo-buildpacks/azure-application-insightsCopy), or Google (paketo-buildpacks/google-stackdriverCopy), and more. The full list is available here.

The build process of this container image is reproducible, and when new versions of the build packs are released, e.g., with a new build of the OpenTelemetry Java agent, the JVM, or improved settings, the only thing you need to do is run the build process again!

For more details on cloud native buildpacks, refer to the excellent documentation on the buildpacks.io website.

Bringing it out for a spin

Deploying a demo Java application on Kubernetes is relatively simple with a deployment:

deployment.yaml
0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: demo
  name: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - image: demo:0.0.1-SNAPSHOT
        name: demo
        ports:
        - containerPort: 8080
        env:
          - name: K8S_NAMESPACE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: K8S_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: K8S_POD_UID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
          - name: K8S_CONTAINER_NAME
            value: demo
          - name: OTEL_JAVAAGENT_ENABLED
            value: 'true'
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            # Replace this with the actual OTLP endpoint you want to use
            value: *otlp-endpoint
          - name: OTEL_EXPORTER_OTLP_HEADERS
            # Replace this with the HTTP headers your OTLP endpoint requires for authorization
            value: *otlp-auth-headers
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: k8s.namespace.name=$(K8S_NAMESPACE_NAME),k8s.pod.name=$(K8S_POD_NAME),k8s.pod.uid=$(K8S_POD_UID),k8s.container.name=$(K8S_CONTAINER_NAME)
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 8080
          initialDelaySeconds: 3
          periodSeconds: 3

A Kubernetes deployment for our Spring Boot application that does a LOT of work to configure the OpenTelemetry Java agent. This listing uses two YAML anchors (*otlp-endpoint, *otlp-auth-headersCopy) to ensure that you do not deploy it with partial configurations.

In the YAML listing above, you see quite a few environment variables that configure the OpenTelemetry Java agent:

• OTEL_JAVAAGENT_ENABLEDCopy is the opt-in environment variable required by the OpenTelemetry buildpack to activate at runtime the OpenTelemetry Java agent.
• OTEL_EXPORTER_OTLP_ENDPOINTCopy and OTEL_EXPORTER_OTLP_HEADERSCopy tell the OpenTelemetry Java agent where to send the telemetry, and which authentication information to send along. You can get the actual values to use with Dash0 from the Java onboarding documentation.
• the K8S_NAMESPACE_NAME, K8S_POD_NAME, K8S_POD_UIDCopy and K8S_CONTAINER_NAMECopy environment variables expose into the pod important metadata (namespace name, pod name and uid, and container name), the values of which are provided at runtime by Kubernetes’ Downward API.
• The OTEL_RESOURCE_ATTRIBUTESCopy instructs the OpenTelemetry Java agent to add key resource attributes from the k8s.* namespace of the OpenTelemetry semantic conventions based on the values of the K8S_NAMESPACE_NAME, K8S_POD_NAME, K8S_POD_UIDCopy and K8S_CONTAINER_NAMECopy environment variables. These values are more than enough, for example, to let the k8sattribute processor of the OpenTelemetry collector reliably fill the gaps like deployment name, deployment UID, etc.

By the way: the $(ENV_VAR)Copy notation is one of my favorite Kubernetes things: it allows you to define “dependent environment variables”, interpolating the value of a variable into another. It’s pretty much the same thing you would do in a Unix shell with $ENV_VARCopy (or ${ENV_VAR}Copy if you have refined shell tastes and enjoy curly brackets in your life as much as I do angular brackets).

Dependent environment variables are pretty much the best way to add to pods environment variables that combine some static part and Kubernetes secrets mounted as environment variables. This is, for example, what you should consider doing for the OTEL_EXPORTER_OTLP_HEADERS Copyenvironment variable if, you know, you wanted to send an HTTP header like:

HTTP headers
0
Authorization=Bearer $(MY_DASH0_TOKEN)

And it works out of the box in Dash0

As you can expect, all the telemetry sent by the OpenTelemetry Java agent works out of the box in Dash0.

Java Virtual Machine and Kubernetes metrics in Dash0.

Logging and tracing work too, including their correlation.

Happy OpenTelemetry-native observability!

One last thing…

Cloud native buildpacks are amazing, and you really should try them.

But if you don’t want to or cannot use them but still want all the awesomeness of effortless Java monitoring on Kubernetes, try out the Dash0 operator: since release 0.37.0, it applies all the techniques for configuring the Java agent shown above and effortlessly adds the OpenTelemetry Java agent to your pods, irrespective of how you build your container images!