OpenTelemetry in production: the open-source stack you still have to operate

Once OpenTelemetry, Alloy, VictoriaMetrics, Tempo, Loki, and Grafana are installed, you still have to run all of that correctly.

That is the less visible part of the project.

The stack is open source. Disks, compute, backups, version upgrades, overly heavy queries, and incidents are not.

In the context I support, that trade-off is worth it. But I would never present this architecture as a way to get “Dynatrace for free”.

Observability can break something#

We saw that fairly early with Kubernetes auto-instrumentation.

Injection via the OpenTelemetry Operator made us use emptyDir on some ephemeral jobs. On lighter environments, nothing particular. In production, with real traffic, disk I/O eventually saturated.

From there, the problem was no longer very theoretical: pods stayed pending and the platform started to suffer.

We disabled the affected part to restore production, then came back to the topic afterwards.

I do not blame the Operator. The mechanism worked elsewhere. This case mainly recalls that an observability layer also consumes CPU, RAM, network, and disk.

You have to load it, limit it, and test it like any other platform component.

Dev does not tell the story of production volumes#

The same gap shows up on the backend side.

A query that answers instantly with a few days of dev data can become very expensive with several weeks of production. A buffer that looks harmless completely changes behaviour when throughput increases.

We had very slow Grafana dashboards, and sometimes queries that put VictoriaMetrics, Loki, or Tempo under pressure.

It was not “Grafana being slow”. The questions asked of the backends were too expensive.

The 1d, 7d, or 30d ranges are a good example. If the dashboard recomputes the same heavy aggregations on every open, the cost comes back for every user and every refresh.

Recording rules made a real difference#

For metrics, we moved part of those calculations into recording rules evaluated by vmalert.

The idea is fairly basic: periodically compute an expensive expression, write the result as a series, then have the dashboard read that series.

You avoid reconstructing the same answer constantly.

That does not remove the need to think about the query. A poorly designed recording rule can also create too many series. But on repeated aggregations and long views, the gain is clear.

That is also why I treat dashboards as platform code. A Grafana JSON can bring an observability stack to its knees as surely as a bad cron.

Observe the observability#

VictoriaMetrics, Tempo, Loki, and Alloy therefore have their own de facto SLOs, even if we do not always call them that.

At a minimum I watch:

CPU and memory;
disk saturation;
ingestion throughput;
export errors;
queues and buffers;
query latency;
active series;
stored volumes;
rule status.

Otherwise you discover the problem when dashboards stop answering. And diagnosing a platform with a broken observability platform is rarely the best moment of the day.

Retention must follow usage#

Another point easy to underestimate: not everything deserves the same retention.

A detailed trace is very useful during a recent incident. Its value often drops fairly quickly. An aggregated metric can stay relevant much longer for following a trend or preparing capacity planning.

Logs have yet another lifecycle.

So I prefer to reason signal by signal rather than copy a “90 days everywhere”.

That is also the value of multi-backend routing: each destination receives what serves its usage, with the appropriate retention.

Splunk, for example, is not our historical target. It remains temporarily because some dashboards have not been migrated yet. When those usages disappear, the dependency can leave with them.

Open source does not mean free#

A stack operated in-house has a very real cost:

storage and compute;
network;
backups;
upgrades;
capacity;
incidents;
rules and dashboards;
security;
human time.

A small team can operate it, but you have to want to own that topic.

Conversely, a SaaS or managed offering buys a lot less operations. That service obviously has a price, often with ingestion, retention, and extra features to account for.

So the useful comparison is not “free open source versus paid SaaS”.

I compare full costs instead:

how much it costs to operate the platform yourself;
how much the managed service costs with real volumes;
which skills you need to keep in-house;
what contractual dependency you accept;
how much a future migration will cost.

Depending on team size, traffic, and constraints, the answer can be very different.

In this specific case, the trade-off remains good#

Here, the architecture lets the client move toward Dynatrace without losing Grafana usages that remain useful.

Collection is more homogeneous. Splunk can leave progressively instead of disappearing in the middle of a migration. Teams have richer traces to diagnose code and architecture. And a backend change no longer necessarily requires going back through 200 services.

On the other side, you have to maintain the stack. That is the price.

I find the exchange interesting because you are buying reversibility and control, not just fewer licences.

What I would do differently in greenfield#

On a new platform, I would start stricter:

limited attributes by default;
no dynamic dimension without a reason;
PII policy before opening the flows;
sampling defined from the start;
cardinality and storage budgets;
recording rules planned for important views;
a justified destination for each signal.

In brownfield, we had to do almost the opposite: keep first, understand the usages, migrate, then close progressively.

Both paths can arrive at the same place. They just do not have the same risk.

What I keep from this experience#

OpenTelemetry does not remove the complexity of observability.

It mainly avoids putting all that complexity into application code.

The service produces signals. The platform decides how to collect, filter, and route them. The client keeps the choice of its backends.

Then you still have to do the work: cardinality, PII, sampling, capacity planning, storage, rules, dashboards, and incidents.

That suits me much better this way. The complexity exists anyway. Might as well put it in a layer you can steer.

OpenTelemetry in production: the open-source stack you still have to operate#