eBPF, BYOC, ClickHouse, Telemetry Pipelines: Which Ones Are Actually Worth It?

Originally posted in Obics.io

The observability world is changing in very interesting ways. In the last decade we’ve experienced the rise of the OpenTelemetry standard and the rise of open source solutions like Elastic and Grafana.

Now, the observability world is experiencing a new wave of technologies:

• eBPF based instrumentation
• Bring your own Cloud (BYOC)
• Columnar data stores, most prominently ClickHouse
• Telemetry Pipelines

There’s also the AI SRE wave, which is more of a “consume observability” use case, whereas I want to talk about technologies that change the core infrastructure of how we gather and store telemetry.

Those solutions are endlessly advertised in social media and conferences. But are they really providing the promised value?

Let’s try and take an unbiased view of what each of those offers and when it’s worth it to use them.

eBPF

eBPF (Extended Berkeley Packet Filter) lets you run small sandboxed programs inside the Linux kernel, attached to hooks like syscalls, network events, or function entry points. For observability, that means an agent sitting on the node can watch every HTTP request, every database call, and every socket your services open, without anyone adding a single line of instrumentation to the application. That’s the pitch: “zero-code instrumentation”. You install a DaemonSet, and a few minutes later you have an APM working, standard metrics, latency percentiles, and error rates.

There are also some high claims about eBPF performance: no chance of application crashes, smaller CPU overhead, better security. All of them make a lot of sense, but all cited sources that I found were biased, so take it with a grain of salt. 1 eBPF Foundation 2 groundcover

Should you use eBPF?

My personal opinion is that (of course) it depends.

The general notion in terms of architecture is somewhat appealing, but the bottom line isn’t a big deal. Sure, you’re doing less in the application layer but more in the kernel level. You reduce dependency on the application but increase it on deployment infrastructure. I don’t see an obvious advantage one way or another.

One big problem with eBPF is that by definition, at the kernel level you don’t have context from the application level. When you’re logging an HTTP request to POST myapp.com/api/cart, you can’t populate attributes like tenant, country, or whatever else. In the kernel, you have only the network details. So the solution would be to inject headers into the request. But that just means you’re back to doing application instrumentation.

For an existing stack, if it works, don’t break it I say. Unless you have a good reason. So here are some good reasons to switch to eBPF:

• You have frequent crashes from current auto instrumentation
• You need to monitor low level network or low level CPU stacks
• You have special security concerns

For a new stack, eBPF makes sense if you think you’ll need low level network observability or low level performance profiling. Think of high-throughput services like telco companies, streaming providers, CDNs, etc. Less required for a run-of-the-mill application that mostly deals with request-response to a database.

BYOC

Bring-Your-Own-Cloud means that the telemetry data is stored in your cloud account, on your own S3 bucket. You give the vendor a scoped IAM role, and the telemetry never leaves your perimeter.

Two things make this attractive. First of all, you are in control of data compliance. If you’re in healthcare, finance, or defense, or you’re dealing with geo-specific regulations, it might be very relevant.

The second issue is cost. Telemetry data gets enormous, and when the vendor charges a per-GB fee with a hefty margin, the bill also gets enormous. At high volume the difference isn’t a few percent, it’s a different order of magnitude.

BYOC disadvantages

The costs don’t disappear with BYOC, they just change shape. You still pay the vendor, usually by the number of nodes, and now you also own a cloud bill that scales with your telemetry. You end up paying two bills to two vendors for your observability. Although likely the sum of those bills will be smaller.

Feature parity is a catch worth checking explicitly. Legacy observability vendors, those who started without BYOC but now offer BYOC, will have less capability compared to the SaaS version.

Is it worth it?

If you’re in a regulated industry, or have data residency requirements, you’re probably self-hosting the observability either way, so moving doesn’t offer any advantage in terms of compliance. And to be fair, the major vendors already got compliance covered. They know all the regulations, got all the compliance standards, and will store the data in the right country.

For brown-field projects that have a 7 or 8-figure bill in DataDog, it’s absolutely worth it to do a cost simulation. You can potentially save a huge amount. DataDog and Grafana are now onboard the BYOC train, and there are multiple new providers like groundcover whose offer is BYOC only, which means it is a first-class citizen in that platform.

Now it’s a matter of math. Do you have many small nodes on k8s or a few big monolith nodes that each produce a lot of telemetry.

For green-field, I wouldn’t consider BYOC at all. First, most new companies are doing microservice architecture on kubernetes, which means you might not even get a smaller bill. Secondly, if the cost is the concern, there are plenty of cheap observability vendors that might not be as nice as DataDog or Dynatrace but still more mature than the newer BYOC companies.

Columnar data stores

This is the least visible of the four, but the most consequential.

Classic log tooling was built on inverted indexes, Elasticsearch being the canonical example. That design is fantastic for finding one document among billions and expensive for everything else. But to allow such search power, you’ll index 3 or 4 times more data than you ingest.

Columnar stores flip the tradeoff. Data is stored one column at a time rather than one row at a time, so a query that touches three fields reads three columns instead of every row of every log. Since a column holds values of the same type repeating, compression ratios of 10x to 30x are normal.

ClickHouse is the dominant choice, but the family includes Druid, DuckDB, and plain Parquet files on object storage.

The practical effect is that storing becomes much cheaper and still offers great search capabilities. That’s why there are many new vendors that offer 3x-10x cheaper deals than New Relic, Splunk, or DataDog. For most of them you probably wouldn’t know they are based on ClickHouse, like Dash0, Better Stack, Coralogix, SigNoz, HyperDX / ClickStack (from ClickHouse DB themselves).

Is it worth it?

The technology of columnar databases itself is amazing and changes the observability industry as we know it. But if you’re shopping for a new observability vendor, it shouldn’t matter if it’s built on ClickHouse under the hood or not. Enough to say it is a powerful enough engine to support your analytics and search needs. And if it allows DataDog competitors to charge 70% less, I wouldn’t complain.

As for hosting your own ClickHouse instance, that’s an entirely different question. ClickHouse by itself is a database, not an observability specialized solution, so if you’re going that route I suggest installing an observability open source project relying on ClickHouse, like SigNoz or HyperDX. Having said that, those tools are going to be much less mature than, say, a Grafana instance or an ELK stack. Even more so, if you’re comparing to SaaS vendors like DataDog or Dynatrace. So as things stand now, if you’re going open source, just go with Grafana.

Telemetry Pipeline

A telemetry pipeline sits between your agents and your observability vendors and lets you modify the data in flight: redact it, sample it, filter it, convert logs into metrics, fan it out to several destinations, and drop a cheap copy into your own object storage for later. Cribl, Mezmo, and Edge Delta all live in this space.

The canonical open source equivalent is the OpenTelemetry Collector, which you can deploy yourself, and it already has support for most of the capabilities like filtering, enrichment, sampling, etc. Not to say that managing a telemetry pipeline yourself is an easy task. It isn’t.

Out of all the advertised value propositions, I think the following are good reasons to use a telemetry pipeline:

• Mid-migration: Moving from one vendor to another? There will be a transition period when you’ll be sending data to both targets. Telemetry pipeline can make that process much easier; change toggles in the pipeline solution with much more granularity and you don’t have to depend on your application deployment for changes.
• Cost reduction: filter out redundant, repetitive, and low value telemetry before incurring the observability vendor rate.

Is it worth it?

I think the mid-migration is a good use case for telemetry pipeline, especially if you’re in enterprise, migrating multiple tools in a multi-month or even multi-year project.

When it comes to cost-reduction I would split the use-case into 2 parts.

1. If your observability needs are mostly for SIEM, telemetry pipeline can be great. Some telemetry sources, like Microsoft Active Directory or Salesforce, emit enormous amounts of data, and you better do something about that before they reach Splunk, or pay millions. That’s Cribl’s claim to fame and how they were one of the fastest rising startups in 2021-2023.

2. If you’re a tech company, where your observability needs are for your own application code, telemetry pipeline is not the best choice. If you have, for example, a duplication, a redundant debug log, or a metric with huge cardinality, why would you pay a 3rd party telemetry pipeline to constantly filter it out, when you can just fix it at the source. It’s your own code, so go ahead and remove the duplication, lower the debug log severity level, or remove some tags to lower cardinality.

That’s why we created Obics — to develop the AI layer that helps you to identify those redundancies and fix them with minimal effort.

So are they worth it?

All four are real technologies that solve real problems, and none of them is the silver bullet the conference talks imply.

If I had to compress it:

Note that three of those technologies are dealing with the observability high cost, and two of the solutions (eBPF and columnar stores) are about performance.

This reflects what’s top of mind in the space. We already solved other problems, like a common observability standard (OTEL), what observability features are best practice (logs, infra monitoring, APM, alerts), and how an observability solution should look like.

So before you evaluate any of them, name the problem first. Crashing agents or low-level network blind spots? Look at eBPF. Data residency, or a bill with too many digits at high volume? Run the BYOC math. A multi-year migration between vendors, or a SIEM drowning in Active Directory logs? A telemetry pipeline earns its keep. Columnar stores are the one you don’t have to decide on at all, you just benefit from the cheaper vendors they enabled. And if none of those problems hurt you right now, the correct move is to do nothing and let the space mature for another year.

What none of them solve is the telemetry itself. They make expensive data cheaper to store, cheaper to move, and cheaper to query, but they don’t make it better. The duplicate log line, the metric with runaway cardinality, the debug statement someone left at info level two years ago, all of those survive every technology on this list. The cheapest telemetry is the telemetry you never emitted, and that’s the part we’re working on at Obics.