Customizing Observability Kit

How to add custom instrumentation traces and metrics to an application.

Customizing Traces
Creating Custom Metrics
System & Process Metrics

The default instrumentation of Observability Kit provides a reasonable amount of data out-of-the-box. However, it can’t cover all the specifics of an application. Writing custom instrumentation allows the collection of application-specific telemetry data for possible problematic places or performance hotspots of an application.

Customizing Traces

Custom instrumentation allows the execution of specific pieces of code to be recorded as spans. These are automatically be added to the current trace, and provide more details, such as attributes and errors.

Custom spans can be added either by using annotations, or by using the OpenTelemetry API to create and end spans, manually. The following sections gives instructions for each of these methods.

Using Annotations

The easiest way to add custom spans is to annotate methods whose executions should be recorded. Annotations require the OpenTelemetry instrumentation annotations dependency to be in the project.

<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-instrumentation-annotations</artifactId>
    <version>1.18.0</version>
</dependency>

Note	Use Matching Versions The OpenTelemetry API version should match the version that’s used by Observability Kit. The used version is available in the release notes and it’s also visible when running the Java agent.

Now methods that should be recorded, and can be annotated with the @WithSpan annotation.

import io.opentelemetry.instrumentation.annotations.WithSpan;

public class ImageListView {
  @WithSpan
  public void fetchImages() {
      // execution code
  }
}

Method parameters can be added as attributes to the span by using the @SpanAttibute annotation like so:

import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.instrumentation.annotations.WithSpan;

public class ImageListView {
  @WithSpan
  public void fetchImages(@SpanAttribute("fetch.url") String fetchUrl) {
      // execution code
  }
}

For more details, see the OpenTelemetry Annotations docs.

Creating Spans Manually

Spans can be created manually using the OpenTelemetry API. This provides finer control over when spans are started, ended, and what data they should contain.

Creating manual spans requires adding the OpenTelemetry API dependency to the project:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.18.0</version>
</dependency>

Note	Use Matching Versions The OpenTelemetry API version should match the version that’s used by Observability Kit. The used version is available in the release notes and it’s also visible when running the Java agent.

To create a span manually, get a tracer instance from the OpenTelemetry API, and then start and end a span around the code that should be traced. For example:

public class ImageListView {
    void fetchImages() {
        Tracer trace = GlobalOpenTelemetry.getTracer("app-instrumentation",
        "1.0");
        final Span span = trace.spanBuilder("My Problematic Task").startSpan();
        try {
            // Do possibly problematic task
        } finally {
            span.end();
        }
    }
}

The started span is automatically added to the currently active trace on the thread.

Attributes can be added to the span for more details on the state of the application. Attributes can be non-null string, boolean, floating point and integer values, or an array of these.

public class ImageListView {
    void fetchImages(String fetchUrl) {
        Tracer trace = GlobalOpenTelemetry.getTracer("app-instrumentation",
        "1.0");
        final Span span = trace.spanBuilder("My Problematic Task").startSpan();
        try {
            // Do possibly problematic task

            // Set attribute on span
            span.setAttribute("fetch.url", fetchUrl);
        } finally {
            span.end();
        }
    }
}

Events can be also recorded on spans to get a timestamp when something happened during the execution. Like spans, events may also contain attributes that are associated with an event.

public class ImageListView {
    void fetchImages(String fetchUrl) {
        Tracer trace = GlobalOpenTelemetry.getTracer("app-instrumentation",
        "1.0");
        final Span span = trace.spanBuilder("My Problematic Task").startSpan();
        try {
            // Record start of the operation as span
            span.addEvent("Image fetch start");

            // load images...

            // Add event with attributes
            Attributes attributes = Attributes.builder().put("images.count", 42).build();
            span.addEvent("Images loaded", attributes);
        } finally {
            span.end();
        }
    }
}

Spans should be marked as errors if the code execution fails. Exceptions can be recorded in a span event, which provides detailed information such as exception type and stack trace when looking at the traces.

public class ImageListView {
    void fetchImages(String fetchUrl) {
        Tracer trace = GlobalOpenTelemetry.getTracer("app-instrumentation",
        "1.0");
        final Span span = trace.spanBuilder("My Problematic Task").startSpan();
        try {
            // Do possibly problematic task
            span.addEvent("Image fetch start");
            // load from external
            span.addEvent("Images loaded");
        } catch(Exception exception) {
            // Handle exception
            // Mark the span as having an error
            span.setStatus(StatusCode.ERROR, exception.getMessage());
            // Add exception trace to the span
            span.recordException(throwable);
        } finally {
            span.end();
        }
    }
}

For more details about manual instrumentation, see the OpenTelemetry manual instrumentation docs.

Long Running Spans

At times, the important information isn’t the method execution time, but in the execution of a threaded piece of code.

For the annotation @WithSpan, if the annotated method returns a future or promise, then the span ends only when the future completes. For supported future types, see Creating spans around methods with @WithSpan

    @WithSpan
    private CompletableFuture<JsonValue> getJsResult(String js) {
        return getElement().executeJs(js).toCompletableFuture();
    }

For manual implementation, it’s possible to keep the span open until a threaded task completes.

    private void getWindowWidth() {
        Tracer trace = GlobalOpenTelemetry.getTracer("app-instrumentation",
        "1.0");
        final Span span = trace.spanBuilder("Fetch window width").startSpan();

        getJsResult("return window.outerWidth").whenComplete((value, throwable) -> {
            System.out.println(value);
            span.end();
        });
    }

    private CompletableFuture<JsonValue> getJsResult(String js) {
        return getElement().executeJs(js).toCompletableFuture();
    }

Creating Custom Metrics

Creating manual metrics requires adding the OpenTelemetry API dependency to the project:

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.18.0</version>
</dependency>

It’s possible to make synchronous and asynchronous metric instrumentation. Synchronous is when measurements are recorded as they happen. Asynchronous is related to when a collection of measurements is invoked.

class MyMetrics {
    void generateMetrics() {
        Meter meter = GlobalOpenTelemetry.meterBuilder("app-instrumentation")
            .setInstrumentationVersion("1.0.0").build();
    }
}

With the Meter, you can now create metric instrumentation for Counter, UpDownCounter, Gauge and Histogram. Each can be either a Long or a Double metric.

Counter: records only positive values.
UpDownCounter: records positive and negative values.
Gauge: measures an instantaneous value with an asynchronous callback.
Histogram: records measurements that are most useful to analyze as a histogram distribution.

Adding a synchronous count for amount of generated images:

class MyMetrics {
    static LongCounter counter;
    void generateMetrics() {
        // ...
         counter = meter
            .counterBuilder("generated_image")
            .setDescription("Amount of images generated")
            .setUnit("1").build();
    }

    public void generateImage() {
        // generation code
        counter.add(1);
    }
}

Adding an asynchronous gauge for measuring open sessions would could be the following:

class MyMetrics {
    void generateMetrics() {
        // ...
        meter.gaugeBuilder("vaadin.session.count").ofLongs()
            .setDescription("Number of open sessions").setUnit("count")
            .buildWithCallback(measurement -> {
                measurement.record(getOpenSessions());
            });
    }
}

Metrics can also be annotated with attributes to help describe what the metric represents.

For more details about manual metrics, see OpenTelemetry metrics

System & Process Metrics

It’s possible to collect systems and process metrics using Operating System and Hardware Information (OSHI) with OpenTelemetry instrumentation.

First, the application needs to get the opentelemetry-oshi and oshi-core dependencies:

<dependency>
  <!-- contains the implementation of the process- and systems-metrics collection for JavaAgent -->
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-oshi</artifactId>
  <version>1.23.0-alpha</version>
  <scope>compile</scope>
</dependency>
<dependency>
  <groupId>com.github.oshi</groupId>
  <artifactId>oshi-core</artifactId>
  <version>5.3.1</version>
</dependency>

Then, to enable and register the metrics and observers, call SystemMetrics.registerObservers() and ProcessMetrics.registerObservers() once.

For instance, here is how it might look in a Spring Boot application:

@SpringBootApplication
public class YourApplication implements AppShellConfigurator {

    static {
        SystemMetrics.registerObservers(GlobalOpenTelemetry.get());
        ProcessMetrics.registerObservers(GlobalOpenTelemetry.get());
    }
    //  other code
}

Documentation