The market for observability tooling

Once a startup reaches any modicum of success, observability becomes a must have tool for their engineering organization (and they will usually adopt it before then anyway). Downtime is basically the worst possible thing that can happen to an engineering team and they will do, and pay for, pretty much anything that will help them proactively avoid it. This, among other dynamics, is why Datadog can charge millions of dollars and observability can make up 20-30% of a company’s entire infrastructure bill.

That being said…companies in this category are relatively mature, and there aren’t necessarily giant differences in their feature sets that matter to buyers. A lot of a buyer’s choice comes down to open source vs. closed source. All in all, you're looking at 20+ legitimate, mature vendor options here. Welcome to the wild world of observability.

So far in this category, we’ve covered Datadog, Splunk, Elastic, and New Relic.

Logs vs. metrics vs. APM#

Broadly, developers think of observability in terms of 3 types of use cases or goals. Understanding the differences between these will help you understand why a company might go for Datadog instead of the Elastic stack, or New Relic and the Grafana stack together.

Logs#

The log is the basic unit of observability, which we covered in the post about Splunk. To quote myself:

In a really ideal world, all of the data we want to analyze would magically sit in perfectly manicured tables with nice column names and no missing data. Unfortunately, reality isn’t so rosy; data is only as good and clean as what generates it, and our systems can get pretty dirty. Data is really (usually) just a record of what happened, and the most popular way of generating and storing that today is a log. A log is just a line that says what happened, where, and when, and includes any necessary other information.

There are logs for everything: server logs, access and authorization logs, event logs, availability logs, resource logs…the list goes on.

A log might get generated when a user makes a request, when someone enters the wrong password, or when one server fails to connect to another server. Pretty much anything that happens in code generates a log. These things are the currency of observability.

But a log is a very basic unit. Inspecting them individually might be useful if you’re trying to figure out what happened on a particular day at a particular time, but to get a bigger picture of what’s going on in your system, you need to aggregate those logs together.

Main use case for logs: individual inspection of what happened for a particular user, or on a particular server or device.

Metrics#

A metric is one level higher than a log: it’s a rollup, or aggregate, of what’s happening to a particular point in your system. You’d use a metric to answer a questions like:

• How many requests did our backend serve yesterday?
• What’s the average response time for those requests?
• What’s the max CPU utilization for a particular server?

These are important to monitor continually, because they’ll tell you when something is working or something is going wrong. If the average response time for your API requests spikes to 10x the normal level, you’ve got a problem.

Recall that logs are the basic units here – metrics are built on top of them. Once you figure out what something is wrong, you might want to dive into the logs to figure out exactly what is happening and why.

Main use case for metrics: ongoing higher level views of particular resources.

APM#

Application Performance Monitoring might just be a marketing term, but it’s a popular one. It refers to monitoring the performance of your application specifically, to the exclusion of infrastructure related things like CPU utilization or how much storage your server is using. APM related things to track include:

• API request response time
• Application load times
• Any errors happening in the application

APM kind of spans both metrics and logs. It just means that both of them are specifically talking about what’s happening inside your app, not any other part of your stack like your servers or database.

Main use case for APM: knowing what’s going on inside of your application.

Back to the big picture. Of course, developers need all of these things. And while companies in this category may have started as a product focused on one in particular, today products like Datadog, New Relic, and Splunk can do all of these things. For example, Datadog has several product lines that correspond to all of these use cases:

• Datadog Log Management
• Datadog Metrics
• Datadog APM

This is why as far as I can tell, most developers seem to view Splunk, New Relic, and Datadog as roughly feature equivalent, give or take a few things like Splunk allowing you to deploy the software in your own data centers.