Machine Mind
14 min readplatform-engineering

Custom Spring Boot Observability Starter: When Every Team's Instrumentation Becomes Your Bottleneck

Building a custom Spring Boot observability starter to enforce consistent tracing, metrics, and context propagation across microservices without vendor lock-in — with real trade-offs from fintech and telecom scale.

On this page

# The Problem

Your telecom carrier processes 2 million call detail records (CDRs) per hour across six microservices: ingestion, validation, enrichment, rating, ledger, and billing. A customer calls complaining about an erroneous charge. You trace their call: it entered the pipeline at 14:32:17 UTC. By 14:32:22, it was marked BILLED. But somewhere between enrichment and rating—a 3-second window—the trace goes dark. Two disjoint traces where there should be one continuous flow.

You open Jaeger. The enrichment service shows the call’s trace ID. The rating service shows a different trace ID for the same call. Context was lost in an async enrichment callback that queued to Kafka. The enrichment team instrumented their callback with Micrometer @Observed but didn’t use ContextPropagatingTaskDecorator. The rating team did. Different approaches, both reasonable-looking, but they broke distributed tracing across team boundaries. Now you’ve spent 90 minutes debugging what should have been a 10-minute investigation.

This is what happens when you leave observability as a per-team concern. Each service adds its own @Observed annotations, uses slightly different context propagation logic, and tunes cardinality differently. The mesh is theoretically observable, but the traces are fragmented, context leaks at async boundaries, and your SLO dashboards show double-counted spans from overlapping instrumentation. When you’re processing millions of CDRs per day, every second of debug delay multiplies cost.

A custom observability starter solves this by making tracing and metrics a platform-level concern. Every service in your mesh gets the same opinion about what to instrument, how to propagate context, how to avoid cardinality explosions, and how to extend it without forking.

# Why Build vs. Buy

Vendor-provided auto-instrumentation (Datadog, New Relic, Splunk) solves observability. Their agents inject bytecode, intercept calls, and emit spans—no code changes needed. You pay per GB and get their UI bundled.

Why we built instead:

The big cloud observability vendors are optimized for black-box applications. They emit a lot of data—every DB query, every HTTP call, every exception. In a telecom context, that cardinality explodes. A CDR (call detail record) processing pipeline might touch a customer ID dimension table on every row. Auto-instrumentation sees that as 2 million independent queries per hour and emits 2 million spans per hour. Your cardinality limits hit in minutes.

You also lose control. Your vendor’s defaults might capture PII or highly sensitive telecom fields (calling line ID, called number, IMSI/IMEI). You can filter in their UI, but that’s reactive. You want to decide at instrumentation time what gets captured. For regulatory compliance (TCPA, GDPR), that matters.

Finally: you own the code. When a bug in the starter breaks observability for all your services, you’re the on-call engineer. But you’re also the one who can ship a fix in 30 minutes instead of waiting for a vendor release and staging environment validation. For a platform team, that ownership is often the right trade-off.

The downside: you maintain it. Every minor Spring Boot release, every Micrometer version bump, you validate the starter still works. A critical bug in the starter affects every service in your mesh. The context propagation logic has to cover Reactor, @Async, virtual threads, and ForkJoinPool—that’s a surface area.

# Architectural Vision

The starter sits between your application code and the Micrometer Observation + OpenTelemetry layers. It’s not a replacement for either—it’s a wrapper that enforces consistent defaults and provides extension points so teams can override without forking.

graph TD
    subgraph App["Application Layer"]
        AppCode["Spring Boot App<br/>@Transactional @Observed"]
    end
    
    subgraph Starter["Observability Starter"]
        Config["Auto-Configuration<br/>@ConditionalOnMissingBean"]
        Context["Context Propagation<br/>Reactor + @Async"]
        Extension["Extension Points<br/>Predicate/Filter/Handler"]
    end
    
    subgraph Micrometer["Micrometer & OpenTelemetry"]
        MM["Micrometer<br/>Observation API"]
        OTel["OpenTelemetry<br/>Exporters"]
    end
    
    subgraph Backends["Observability Backends"]
        Jaeger["Jaeger/Tempo"]
        Prometheus["Prometheus"]
    end
    
    AppCode --> Config
    AppCode --> Extension
    Config --> MM
    Context --> MM
    Extension --> MM
    MM --> OTel
    OTel --> Jaeger
    OTel --> Prometheus

The starter provides three things:

  1. Sensible defaults — every service gets the same @Observed setup on HTTP handlers, repository methods, and async tasks.
  2. Context propagation — trace IDs flow from sync code into Reactor Context, into @Async tasks, even into ForkJoinPool work-stealing threads.
  3. Extension points — teams can define ObservationPredicates to exclude certain paths, ObservationHandlers to extract custom attributes, and ObservationFilters to rename or suppress spans without editing the starter source.

# Key Design Decisions

Auto-configuration strategy

Spring Boot’s @ConditionalOnClass, @ConditionalOnMissingBean idiom is the foundation. The starter auto-configures:

  • ObservationRegistry customization (registers filters, predicates, handlers)
  • TraceIdGenerator (defaults to OpenTelemetry’s UUID-based generator)
  • ContextPropagator beans for W3C Trace Context and W3C Baggage
  • MeterRegistry instrumentation (CPU, memory, GC)

Teams can override any bean by defining their own in application.yml or a @Configuration class. No forking required.

The observation lifecycle

Here’s what happens when a span is created:

sequenceDiagram
    participant Request as HTTP Request
    participant App as App Code
    participant Registry as Observation Registry
    participant Predicate as Predicate
    participant Filter as Filter
    participant Handler as Handler
    participant Backend as Backend
    
    Request ->> App: HTTP handler
    App ->> Registry: Observation.start()
    Registry ->> Predicate: test(name, context)
    
    alt Predicate rejects
        Predicate -->> Registry: false
        Registry -->> App: no-op observation
    else Predicate accepts
        Predicate ->> Registry: true
        Registry ->> Filter: process(scope)
        Filter ->> Handler: handle(scope)
        Handler ->> Backend: emit span
        App ->> App: business logic
        App ->> Registry: Observation.stop()
        Registry ->> Handler: handle(context)
        Handler ->> Backend: finalize span
        Backend -->> App: done
    end

Context propagation across async boundaries

The hardest part: trace IDs must flow from the original HTTP thread into async tasks. Spring Boot 3.0+ provides TaskExecutor integration via ContextPropagatingTaskDecorator, but you have to opt in per threadpool. The starter makes it automatic.

When you define @Bean public TaskExecutor in your application, the starter wraps it with ContextPropagatingTaskDecorator unless you’ve already done so. Same for ThreadPoolExecutor used in @Async methods.

For Reactor (the hard case), the starter registers a ContextPropagationContext supplier that hooks into Reactor’s Context API. When a trace scope is created, its attributes (trace ID, span ID) are stored in Reactor’s Context automatically. Downstream operators see them.

The trade-off: automatic wrapping assumes you want context propagated everywhere. If you have hot loops doing @Async work millions of times per second, the TaskDecorator overhead (copying the context map) matters. You can disable it for specific thread pools via properties and manage context manually.

# Extension Points

ObservationPredicate

Decides whether an observation scope should be recorded. Use it to exclude health checks, readiness probes, and other noisy low-value calls.

@Bean
public ObservationPredicate observationPredicate() {
  return (name, context) -> {
    // Skip /actuator/* paths
    if (name.contains("http.server") && 
        context.getAttribute("http.url") != null) {
      String url = (String) context.getAttribute("http.url");
      if (url.startsWith("/actuator")) return false;
    }
    return true;
  };
}

ObservationFilter

Mutates attributes on an observation before it’s recorded. Rename attributes, add custom tags, suppress high-cardinality values.

@Bean
public ObservationFilter observationFilter() {
  return (context) -> {
    // Bucket transaction amounts to $1K ranges
    if (context.getAttribute("transaction.amount") != null) {
      long amount = (Long) context.getAttribute("transaction.amount");
      long bucket = (amount / 1000) * 1000;
      context.addAttribute("transaction.amount.bucket", bucket);
    }
  };
}

ObservationHandler

Custom code that runs when a scope is created, stopped, or encounters an error. Use it to log, send alerts, or update custom metrics.

@Bean
public ObservationHandler<Observation.Context> customHandler() {
  return new ObservationHandler<Observation.Context>() {
    @Override
    public void onStart(Observation.Context context) {
      // Custom logic
    }
    @Override
    public void onStop(Observation.Context context) {
      // Custom logic
    }
    @Override
    public boolean supportsContext(Observation.Context context) {
      return true;
    }
  };
}

ObservationRegistryCustomizer

Fine-grained control: add or remove handlers, set sampler strategies, configure exporters.

@Bean
public ObservationRegistryCustomizer<MeterRegistry> registryCustomizer() {
  return (registry) -> {
    registry.observationConfig()
      .observationHandler(myHandler)
      .sampler(Sampler.alwaysSample()); // Override default sampler
  };
}

The extension point model:

  • ObservationPredicate — decides whether to record
  • ObservationFilter — mutates attributes before recording
  • ObservationHandler — runs custom code on lifecycle events
  • ObservationRegistryCustomizer — configures the registry itself

Teams implement these as beans and the starter discovers them automatically via @ConditionalOnMissingBean.

# Reactive & Async Propagation

This is where teams diverge most. Reactive code (Reactor, Project Loom virtual threads) and traditional @Async need different context propagation strategies.

Reactor (the easy case now, hard historically)

Spring Framework 6.1+ and Spring Boot 3.2+ integrated Micrometer’s ContextPropagationContext directly into Reactor. When you create an observation scope, its attributes are automatically stored in Reactor’s Context and flow through all downstream operators (map, flatMap, switchMap). No manual plumbing.

@Service
public class OrderService {
  @Autowired private Observation observation;
  
  public Mono<Order> createOrder(Order order) {
    // Observation scope is created; trace ID flows into Reactor context
    return observation.observe(() ->
      orderRepository.save(order)
        .flatMap(saved -> 
          // Trace ID is visible here
          notificationService.notify(saved)
        )
    );
  }
}

@Async (the complex case)

@Async methods run on a thread pool (TaskExecutor). Without context propagation, the trace ID stays on the caller’s thread. The async work is invisible to tracing.

The starter wraps your TaskExecutor with ContextPropagatingTaskDecorator, which copies the MDC and Spring’s SecurityContext (and, in Spring 6.1+, ContextPropagationContext) to the thread pool thread. The trace ID follows.

But there’s a catch: ForkJoinPool (used by CompletableFuture and parallelStream()) is also a Executor, but Spring doesn’t auto-wrap it. If your service uses CompletableFuture for async work, the starter can’t help—context won’t propagate. You have to either:

  1. Avoid CompletableFuture; use @Async instead.
  2. Manually wrap your CompletableFuture chains with context copy/restore logic (tedious).
  3. Accept that ForkJoinPool work loses trace context (the pragmatic choice for non-critical paths).

Real example: a telecom billing system processes CDRs in parallel using parallelStream() on a ForkJoinPool. The main trace covers the CDR ingestion. Each record’s processing (validation, enrichment, ledger update) runs on the ForkJoinPool and loses the trace context. The starter can’t help. Solution: convert to a TaskExecutor-backed @Async loop, accept the sequential overhead, and regain tracing.

# Data Access Layer Coverage

JDBC DataSource (the problem)

javax.sql.DataSource is wrapped by Spring with a JdbcObservationFilter, which emits spans for each SQL query. Default behavior captures the SQL and result count. But on a high-throughput service, every unique SQL query becomes a cardinality explosion.

Example: a payments ledger service runs SELECT * FROM ledger WHERE account_id = ? millions of times per day. Each account_id is a different query in the span attributes. You hit your cardinality limits in hours.

The starter lets you filter or rename these attributes:

@Bean
public ObservationFilter jdbcFilter() {
  return (context) -> {
    if ("sql.query".equals(context.getName())) {
      // Remove the WHERE clause; just keep the table name
      String sql = (String) context.getAttribute("sql.query");
      String normalized = sql.replaceAll("WHERE.*", "WHERE ...").intern();
      context.addAttribute("sql.query", normalized);
    }
  };
}

R2DBC (the solution)

R2DBC (reactive relational database connectivity) ships with first-class Micrometer Observation support. Statements are wrapped automatically, span attributes are pre-normalized, and cardinality is sane.

Trade-off: not all JDBC drivers have R2DBC equivalents (Oracle’s R2DBC support came late). If you’re on PostgreSQL or MySQL, R2DBC is available and observability is better. If you’re on an older proprietary database, you’re stuck with JDBC.

The starter doesn’t force R2DBC, but it strongly recommends it and configures JDBC cardinality filtering to match R2DBC’s behavior.

Data access observability paths:

  • JDBC Path: Raw SQL + account_id in spans → cardinality explosion
  • JDBC + Starter Filter: Normalized SQL → sane cardinality
  • R2DBC Path: Native support, pre-normalized → sane cardinality by default

# Rollout Plan

You don’t flip a switch and add the starter to 200 services overnight. Context propagation bugs, cardinality explosions, and extension point conflicts will surface only under production load.

Phase 1: Adoption by one team (weeks 1–4)

The platform team dogfoods the starter in one non-critical service (e.g., user preferences microservice). Goals:

  • Verify context propagation works with their tech stack (Spring Cloud Gateway, Reactor, PostgreSQL).
  • Validate that Jaeger/Tempo shows correct traces.
  • Load-test to catch cardinality issues early.

Learnings: the preferences team uses CompletableFuture in a recommendations enrichment pipeline. Context is lost. They either switch to @Async or accept the tracing gap. Decision: accept it (it’s low-priority work).

Phase 2: Broader adoption (weeks 5–12)

Roll out to 3–5 services across different teams: payments, ledger, notifications, compliance. Each team configures the starter slightly differently (different ObservationPredicates, different sampler strategies).

Learnings:

The ledger team’s observations on UPDATE account SET balance = balance + ? create one span per unique balance delta. High-cardinality disaster. The starter’s ObservationFilter saves them; they normalize deltas to $100 buckets.

The payments team runs two rate-limiting endpoints side by side (old and new). Both are instrumented. Overlapping @Observed annotations cause spans to be double-counted in Prometheus. Solution: add an ObservationPredicate to suppress the old endpoint’s observations.

The compliance service has audit logging (@Observed on every repository method) and fine-grained tracking (another layer of observability for reporting). Too many observations, Prometheus metrics grow 10x. Solution: introduce a sampler that records 10% of low-cardinality compliance observations and 100% of high-cardinality suspicious transaction flags.

Phase 3: Ecosystem-wide (months 4–6)

Roll out to all services. By this point, the starter has stabilized, and the platform team has published a runbook: “What to do when your observations aren’t showing up”, “How to fix cardinality explosions”, “When to use predicates vs. filters”.

Real example: a major telecom processing 10 million CDRs per day rolls out the starter across their main pipeline (ingestion, validation, enrichment, ledger, billing). Initial cardinality issues: validation step touches 50K equipment IDs per second. They filter observations to suppress equipment-specific tags. Ledger step touches customer service plans (40K plans). They normalize to plan buckets. After 4 weeks of tuning, observability is good, cardinality is predictable.

# Trade-offs & Failure Modes

Duplicate observations (silent SLO corruption)

If two teams instrument the same code path, you get two spans for one operation. Prometheus metrics double-count duration and error rates.

Example: the ledger team adds @Observed to TransactionRepository.save(). Later, the core platform team adds the same annotation in a base repository class. Now both fire. Metrics for save duration look good, but they’re double. SLO alerts fire because “p99 duration increased”, but it’s an instrumentation artifact.

Detection: compare span counts in Jaeger to API call counts. If spans > calls, you have duplication.

Fix: the starter includes an optional DuplicateObservationDetector that logs when two observations with the same name are active on the same thread. It’s disabled by default (low-overhead but not free) and can be enabled per service.

# In application.yml
spring.observability.duplicate-detection-enabled: true

Missing context in async retry queues

A payments service publishes a retry event to Kafka when a ledger update fails. The consumer reads the event, but the trace ID is not in the Kafka header (because the producer didn’t serialize it). The consumer starts a fresh trace. Two disjoint traces where there should be one.

Fix: the starter provides an ObservationContextMessagePropagator bean that automatically serializes trace context into Kafka headers (via Spring Cloud Stream / Spring Kafka).

# Automatic, but only if you use Spring Kafka or Spring Cloud Stream
spring.application.name: payments-service
# The starter detects Kafka and adds the propagator

Cardinality explosion from normalized-too-late filters

You define an ObservationFilter that normalizes high-cardinality values. But the span is already created and exported before the filter runs. You’re still emitting millions of unique metric names to Prometheus.

Lesson: use ObservationPredicate (which runs before observation creation) to prevent high-cardinality scopes from being recorded at all. Use ObservationFilter only to tag transformations, not to suppress.

Wrong (ineffective):

@Bean
public ObservationFilter wrongApproach() {
  return context -> {
    if (isTooHighCardinality(context)) {
      // Too late! Span already created and emitted
      context.addAttribute("filtered", true);
    }
  };
}

Right (efficient):

@Bean
public ObservationPredicate rightApproach() {
  return (name, context) -> {
    if (name.equals("sql.query") && isTooHighCardinality(context)) {
      return false; // Don't record this observation at all
    }
    return true;
  };
}

Virtual threads and context leaks

With Project Loom’s virtual threads, you can create millions of threads with negligible overhead. But if context propagation isn’t set up correctly, each virtual thread gets a copy of the trace context, and you lose the parent-child relationship in Jaeger.

Spring 6.1+ handles this automatically via ContextPropagationContext and VirtualThreadTaskExecutor. The starter configures it. But if you use raw Thread.startVirtualThread() without the Spring abstraction, context doesn’t flow.

Mitigation: the starter docs emphasize using Spring’s TaskExecutor abstraction, not raw threads. For teams that ignore this, tracing gaps appear in Jaeger.

# Cost & Ownership

Who maintains this?

The platform team. It’s a platform investment, not a framework. Three engineers: one for Micrometer/OpenTelemetry knowledge, one for Spring Boot and context propagation, one for incident response.

How is it distributed?

Publish to your internal Artifactory / Nexus as com.yourcompany.platform:spring-boot-starter-observability:1.0.0. Every service includes it as a compileOnly or implementation dependency.

Versioning: semantic versioning. Minor version bumps when you add new extension points. Patch bumps for bug fixes. When Spring Boot or Micrometer releases a major version, you bump the starter’s major version and validate it across a tier of services before rolling out.

Failure mode: the starter has a bug

A critical bug in context propagation logic is discovered: trace IDs are being dropped in 0.01% of async tasks (enough to break financial reconciliation). You cut a patch release, publish to Artifactory, and all services pick it up in their next deploy.

If you’re using a vendored observability agent, this is the vendor’s problem. With a custom starter, it’s yours. You handle the incident in 2 hours. That’s the ownership cost.

Failure mode: the starter has a breaking change

You upgrade to Micrometer 1.13, which changes the ObservationHandler interface signature. The starter is compatible, but your team’s custom ObservationHandler breaks. You have to backport the custom handler to the new API, or the service can’t deploy.

This is why the starter is versioned carefully. You don’t bump Micrometer versions in the starter without extensive validation and a clear upgrade path for custom handlers.

# Closing

If I were to start over, I’d push harder on the separation between “what to observe by default” and “how teams override those defaults”. We learned this too late: a single application.properties file with 47 observability flags is harder to maintain than a few well-designed extension points. Teams are smarter than you think. Give them the levers, set reasonable defaults, and they’ll extend it without breaking things.

Also: observe in production, not in development. Early emphasis on “let’s instrument everything in the test environment” wasted weeks. Real issues surface when you’re processing millions of transactions per day, and that’s when you need to tune. Start with broad observations, filter aggressively, and add precision where it matters.

The other thing: context propagation is harder than metrics. Metrics are fire-and-forget. Context has to flow through async boundaries, and there are always new frameworks (Vert.x, Kotlin coroutines, custom thread pools) that break it. You can’t predict all of them. The starter provides good defaults, but every new technology that touches async gets a post-mortem: “Why didn’t trace context follow?”

Still, building this saved the organization millions in observability tooling costs and weeks of debugging time. A platform team that owns observability is worth the investment.

# References

—views

Click to show appreciation

Related reading

Discussion