The 4 Layers of OpenTelemetry Adoption
How companies adopt OpenTelemetry and learnings from speaking to users.
Most companies treat OpenTelemetry like a one-time project. It's not.
After speaking with enough teams going through OTel adoption, we started seeing a pattern.
It's four sequential phases, each requiring different skills and manual work, revealing different blind spots, and failing in different ways.
From conversations, we mapped out four layers to the OpenTelemetry lifecycle:
- App-level instrumentation
- Infra-level instrumentation
- Pipeline Reliability
- Data Parity
This is what we’ve learned:
Layer 1: App-level Instrumentation
This is about the SDK.
You learn the terms, the language differences, the nuance, and use cases; you map what to instrument. Maybe you spin up Jaeger. Real dev work.
The learning curve is steep.
You're reading documentation scattered across GitHub repos and the OTel docs site, and going through vendor-specific guides that may or may not apply to your setup.
Most OpenTelemetry initiatives begin here because Layer 1 exposes the necessary concepts from the get-go.
Perhaps you aim for one proof point: a cross‑service trace of a critical path, end‑to‑end, with the right telemetry visible.
Set the scope, milestones, and team.
What we also learn is that Layer 1 is a test for organizational readiness.
You're not just asking developers to learn new tools. You're asking them to adopt new concepts.
Logs worked fine. Now they need to understand spans, context propagation, and the semantic convention sh*t show.
The biggest challenge isn’t always technical; it’s poor communication about why you're doing this, what each team gains when it works.
Without it, you create "diva" teams who refuse to participate. Sub-optimal teamwork cascades through every layer after.
Ultimately, Layer 1 is also about getting everyone to care.
Because when teams inevitably underestimate the timeline and difficulty, that early alignment is what keeps the project moving.
Layer 2: Infra-level instrumentation
This is where you drift off the SDK and into collector-driven instrumentation.
Layer 2 is proper DevOps/SRE work, centered on the OpenTelemetry Collector; deploying and configuring collectors to handle both infrastructure and app-level telemetry.
It generally runs in parallel with Layer 1, though it does introduce a more complex orchestration and relies heavily on concepts most learn during Layer 1.
You’re not really adding SDKs to apps here; you’re shaping flow.
Sidecar, daemon, gateway, load balancing, sampling rules.
Deployment choices matter a great deal; Go all-DaemonSet, and memory consumption sneaks up on you. Go all-sidecar and you end up with preposterous config files.
Like that one Dark Knight quote: "You either die a hero, or you live long enough to see your YAMLs explode.”
Layer 2 is the wiring: deciding what to collect and how collectors process and route telemetry. Add sampling rules, data filtering, compliance controls - so every pipeline honors the same standards.
In practice, Layer 2 sets the intent that Layer 3 will verify in reality. Think of it as design/build: you implement here, then prove and operate there.
Layer 3: Pipeline reliability
Layer 3 is where the work shifts to proving the pipeline itself is healthy.
This layer is about confidence. Observability for your observability. Is your K8s telemetry flowing correctly? Are collectors processing data as expected? Does telemetry pass safely end-to-end?
You want a live view of the whole pipeline so you can pinpoint where data changed or vanished.
Investing here is the difference between knowing your system is healthy and just hoping it is; risking silent data loss, undetected outages, and misleading dashboards.
This is where OpenTelemetry OpAMP helps: fleet‑wide health and the ability to manage and troubleshoot configurations remotely and safely, and see the effective config an agent is actually running, so you can prevent and fix drifts.
Layer 3 is the verification and operations loop for what Layer 2 built. It answers “Did our Layer 2 intent hold in production?”
Together, Layers 2 and 3 form a cycle: design in Layer 2, validate in Layer 3, adjust Layer 2, and repeat - until telemetry arrives end‑to‑end with no surprises.
Layer 4 - Data parity
This layer is for teams migrating off a vendor. The question is: Does the new stack behave like the old one? Can we, now, get the insights we want?
Practically, you validate correctness and completeness.
Totals and values of metrics should match. Logs should arrive. Queries should work equivalently. If labels or fields don’t propagate - or their naming differs - you can’t slice, dice, or search reliably.
Ensure the migrated data behaves the same so you can confidently turn off the old stack.
This is “split‑screen” work: previous stack on one side, new stack on the other.
I think the dream is to snap your fingers and see hundreds of your Datadog dashboards appear in your new stack.
Since we’re not there (yet), you need to do the work: transform your dashboards and tables so they map cleanly to the new stack, and prove parity across common views before you retire the old.
The Lifecycle
At some point, layer 4 becomes obsolete. When you successfully migrate to your new stack, life feels a little easier.
The work then shifts to maintaining and operating layers 1 through 3.
When a new service appears, you cycle back through the layers: you instrument it (Layer 1) and verify its telemetry flows correctly (Layer 3). You can revisit Layer 2 if the service needs some special handling.
Configuration drift creeps in. Your payment services start throwing errors, and you're adjusting sampling rates, tweaking pipelines, and ensuring the right data still makes it through.
Even with a single collector, complexity compounds. Some services run in hosted setups, others in containers configured differently.
Add more teams, more services, more edge cases - what started as a simple setup now demands ongoing refinement and adaptation.
OpenTelemetry is not a one-time project. It's a never-ending cycle.
4 → Lawrence (marketing stuff)
If you're stuck somewhere in these four layers - migrating to OpenTelemetry or running it across a complicated distributed system - get into the private beta!
We're building the tools to help teams install, maintain, and troubleshoot OpenTelemetry at scale.