OpenTelemetry in production: the open-source stack you still have to operate

In a large organization, observability tools change. That is normal: requirements evolve, contracts change, and different teams need different tools.

What bothers me is when every backend change sends us back into dozens or hundreds of applications. We repeat the same migration, with the same risks. That is an endless cycle I would rather avoid.

That is why I proposed OpenTelemetry during this migration. Not to replace Dynatrace, force Splunk out, or mandate a Grafana stack. Dynatrace remains an expected destination; existing Grafana and Splunk use cases do not disappear just because we adopt a common standard.

Applications emit signals under a shared contract. The platform collects, transforms and routes those signals to whichever tools are needed. When a destination changes, I would rather update the platform than reinstrument every service.

Of course, changing backends does not magically migrate dashboards, alerts or proprietary features. But at least it does not tie each application back to a single provider.

There is a trade-off: we now have to operate that shared layer — OpenTelemetry, Alloy, VictoriaMetrics, Tempo, Loki and Grafana. Disks, queries, backups, capacity and incidents do not vanish with a more flexible architecture. This was never about getting ‘Dynatrace for free’. The last part of this series is about the actual cost of that flexibility.

It worked outside production. We had forgotten about volume#

We ran into this with OpenTelemetry auto-instrumentation on Kubernetes.

The Operator injects an initContainer and a shared emptyDir volume to make the instrumentation libraries available inside the application container. That emptyDir was not declared by our jobs: it came with instrumentation.

Outside production, traces arrived and pods started as expected. Nothing unusual.

Production had a completely different rhythm: lots of jobs, short-lived pods constantly coming and going, and an extra initialization phase for each instrumented pod. The I/O cost was repeated every time.

At that scale, disk I/O eventually saturated and some pods were left waiting. We disabled the affected injection to get things back to normal, then revisited the setup.

It was not one emptyDir behaving badly or a mysterious Operator bug. It was the cumulative cost of a perfectly normal mechanism across workloads that start and disappear very frequently.

I had checked that instrumentation worked. I had not checked carefully enough what it cost at real production volumes. Oops.

Observability consumes resources too, even before the application starts. We need to test it at the right scale.

Non-production does not reflect production volumes#

The same gap shows up on the backend side.

A query that answers instantly outside production with a small dataset can become very expensive with several weeks of production data. A buffer that looks harmless completely changes behaviour when throughput increases.

We had very slow Grafana dashboards, and sometimes queries that put VictoriaMetrics, Loki, or Tempo under pressure.

It was not “Grafana being slow”. The questions asked of the backends were too expensive.

The 1d, 7d, or 30d ranges are a good example. If the dashboard recomputes the same heavy aggregations on every open, the cost comes back for every user and every refresh.

Recording rules made a real difference#

For metrics, we moved part of those calculations into recording rules evaluated by vmalert.

The idea is fairly basic: periodically compute an expensive expression, write the result as a series, then have the dashboard read that series.

You avoid reconstructing the same answer constantly.

That does not remove the need to think about the query. A poorly designed recording rule can also create too many series. But on repeated aggregations and long views, the gain is clear.

That is also why I treat dashboards as platform code. A poorly designed dashboard can put substantial pressure on the backends without changing a single line of application code.

Observe the observability#

VictoriaMetrics, Tempo, Loki, and Alloy therefore have their own de facto SLOs, even if we do not always call them that.

At a minimum I watch:

CPU and memory;
disk saturation;
ingestion throughput;
export errors;
queues and buffers;
query latency;
active series;
stored volumes;
rule status.

Otherwise you discover the problem when dashboards stop answering. And diagnosing a platform with a broken observability platform is rarely the best moment of the day.

Retention must follow usage#

Another point easy to underestimate: not everything deserves the same retention.

A detailed trace is very useful during a recent incident. Its value often drops fairly quickly. An aggregated metric can stay relevant much longer for following a trend or preparing capacity planning.

Logs have yet another lifecycle.

So I prefer to reason signal by signal rather than copy a “90 days everywhere”.

That is also the value of multi-backend routing: each destination receives what serves its usage, with the appropriate retention.

Splunk, for example, still serves dashboards that have not yet been migrated. I am not trying to turn it off on principle. As long as it serves a purpose, the platform sends it the data it needs. If those use cases move elsewhere, we can phase out that destination without changing the applications’ instrumentation.

Open source does not mean free#

A stack operated in-house has a very real cost:

storage and compute;
network;
backups;
upgrades;
capacity;
incidents;
rules and dashboards;
security;
human time.

A small team can operate it, but you have to want to own that topic.

Conversely, a SaaS or managed offering buys a lot less operations. That service obviously has a price, often with ingestion, retention, and extra features to account for.

So the useful comparison is not “free open source versus paid SaaS”.

I compare full costs instead:

how much it costs to operate the platform yourself;
how much the managed service costs with real volumes;
which skills you need to keep in-house;
what contractual dependency you accept;
how much a future migration will cost.

Depending on team size, traffic, and constraints, the answer can be very different.

In this specific case, the trade-off remains good#

The client can continue moving toward Dynatrace without forcing teams to abandon their existing Grafana or Splunk use cases overnight.

The platform takes on a clear responsibility: evolve collection, transformation and destinations. Applications should not need to know which observability tool the organization chooses next.

That separation does not make backends interchangeable. Provider-specific queries, dashboards and alerts still need migration work. But this should not mean going back into the instrumentation of 200 services every time tooling changes.

The other side of the deal is that we have to maintain that shared layer. That is a trade-off I accept: fewer provider dependencies inside applications, more operating responsibility on the platform.

What I would do differently next time#

I would keep OpenTelemetry and the separation between instrumentation and destinations. I would make that contract and its limits explicit earlier: what the platform can change, what remains backend-specific, and who owns each adaptation.

I would first distinguish long-running services from short-lived Jobs and CronJobs. The same instrumentation has a very different cost when pods are created again and again.

For ephemeral workloads, I would look at the real pod creation rate, initContainer duration, ephemeral-storage use, and disk I/O per node. Then I would compare runs with and without injection under comparable load. No made-up benchmark: we need the cluster’s actual workload profile.

I would roll out instrumentation in batches, explicitly exclude jobs where the signal is not worth the cost, and make reverting the injection straightforward. At each step I would track startup times and pods left waiting.

For Grafana, I would also test dashboards earlier against realistic data volumes and retention periods, with recording rules for repeated metric aggregations, instead of finding out their cost later.

I would not ban initContainers or emptyDir. I would just measure the cost of repeating them much earlier.

What I would do differently in greenfield#

On a new platform, I would start stricter:

limited attributes by default;
no dynamic dimension without a reason;
PII policy before opening the flows;
sampling defined from the start;
cardinality and storage budgets;
recording rules planned for important views;
a justified destination for each signal.

In brownfield, we had to do almost the opposite: keep first, understand the usages, migrate, then close progressively.

Both paths can arrive at the same place. They just do not have the same risk.

What I keep from this experience#

The starting point was not to replace everybody’s tools. It was to avoid another campaign of application changes every time a large organization makes a new observability tooling decision.

OpenTelemetry helps us escape that loop: services describe what they do, while the platform owns collection, filtering and routing. It separates responsibilities; it does not promise effortless migrations.

We also learned the cost of that separation in practice: the initContainers and emptyDir volumes introduced by auto-instrumentation have a cost when short-lived pods multiply. So do high-cardinality labels, expensive dashboards, retention and data volumes.

I would make the same architectural choice again. I would just test realistic volumes earlier and treat operating that shared layer as part of the design from day one.

OpenTelemetry in production: the open-source stack you still have to operate#