Journald stores logs in a compressed binary format in the /var/log/journal/ directory, which saves disk space and improves efficiency. Its structured logging supports multi-line text, which is usually difficult to deal with, requiring complex configurations in logging agents to deal with correctly. And it’s fast.
In this blog post, we’ll show you how to send journald logs to Dash0. Dash0 has excellent built-in support for journald, automatically mapping all metadata to the OpenTelemetry semantic conventions, providing a delightful experience for your journald log analytics.
Getting started with the OpenTelemetry journald receiver
For this blog post, we will show how to collect journald logs using the OpenTelemetry journald receiver on an AWS EC2 instance. The post covers the following key steps:
- Installing the OpenTelemetry Collector
- Setting permissions
- Configuring the OpenTelemetry Collector
- Tuning the configuration for specific use cases, such as filtering logs by unit or priority.
The full documentation for the journald receiver can be found here: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.116.0/receiver/journaldreceiver#readme
Step 1: Install OpenTelemetry Collector
As a first step you have to install the OpenTelemetry Collector binary. For Ubuntu running on an amd64 architecture, you can run the following commands:
0123$ sudo apt-get update$ sudo apt-get -y install wget$ wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.deb$ sudo dpkg -i otelcol-contrib_0.116.1_linux_amd64.deb
For all other setups you can find the installation instructions within the Collector’s installation guidelines.
Step 2: Setup user permissions
The OpenTelemetry collector will need some specific privileges to be able to get logs from journald. And this is accomplished by granting these privileges to the otelcol-contrib user by adding it to the systemd-journal group:
012$ sudo adduser otelcol-contrib systemd-journalAdding user `otelcol-contrib' to group `systemd-journal' ...Adding user otelcol-contrib to group systemd-journal
Run the following manual check to verify that journalctl works for the otelcol-contrib user:
0123456$ sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contribDec 16 20:39:48 ip-10-2-199-35 sudo[147296]: pam_unix(sudo:session): session closed for user rootDec 16 20:39:54 ip-10-2-199-35 sudo[147305]: root : TTY=pts/1 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/su -s /bin/bash -c 'journalctl --lines 5' otelcol-contribDec 16 20:39:54 ip-10-2-199-35 sudo[147305]: pam_unix(sudo:session): session opened for user root(uid=0) by ubuntu(uid=0)Dec 16 20:39:54 ip-10-2-199-35 su[147307]: (to otelcol-contrib) root on pts/2Dec 16 20:39:54 ip-10-2-199-35 su[147307]: pam_unix(su:session): session opened for user otelcol-contrib(uid=996) by root(uid=0)
Step 3: Add journald to collector config.yaml
Now we need to instruct the OpenTelemetry Collector to use the newly-acquired privileges to access journald logs.
The OpenTelemetry configuration is stored under /etc/otelcol-contrib/config.yaml . Modify it as shown below. Note that configuration only scrapes journald logs from the server, batches them and sends them to an OpenTelemetry backend like Dash0; if you have pre-existing configurations, you will have to merge them
01234567891011121314151617181920212223242526272829extensions:bearertokenauth/dash0:scheme: Bearertoken: <AUTH_TOKEN>receivers:journald:directory: /var/log/journalprocessors:batch: {}exporters:otlp/dash0:auth:authenticator: bearertokenauth/dash0endpoint: ingress.eu-west-1.aws.dash0.com:4317service:extensions:- bearertokenauth/dash0pipelines:logs:receivers:- journaldprocessors:- batchexporters:- otlp/dash0
Collector example configuration, https://www.otelbin.io/s/0246ac5590651e0955f0b48733c5280f94174513
By the way, OTelBin.io is a tool we provide with ❤️ to the OpenTelemetry community to help configure the OpenTelemetry collector. Give it a try at!
Step 4: Restart Collector
Restart the Collector for the new configuration to become active.
0$ sudo systemctl restart otelcol-contrib
Now you can see the logs appearing in Dash0.
We automatically extract attributes and the severity from journald logs to enrich them with context.
Step 5: Tuning the configuration
You may further tune which logs are collected. The following example shows how to collect only logs for the systemd unit “atlantis” (i.e., a specific workload managed by systemd) with priority “info”.
0123456receivers:journald:directory: /run/log/journalunits:- atlantispriority: info
Conclusion
Journald is a logging system with many desirable properties: it supports structured logs, works transparently with multi-line logs, and it’s easy to integrate through the OpenTelemetry Collector with Dash0.
We hope this post helps you get better insights on what is going on with your applications and systems.
If you’re ready to experience next-generation observability, try Dash0 today!
