Elasticsearch’s primary use cases#
Elasticsearch’s primary use cases revolve around things that commonly need, you guessed it, search. One big theme centers around infrastructure management, but teams also use it for security and even user facing search engines.
1) Application and server logs#
Your application – as well as the server(s) that you run it on – shoot out tons and tons of logs. They’re basically just text that say what’s going on, like “we’re installing this” or “this thing failed” or “we’re starting up this program.” But in those logs, my friend, is gold. When things go wrong and you’re trying to figure out why, these are some of the first things engineers will turn to.
With Elasticsearch, you can stream your logs from the source, store them indefinitely (or set a retention window), and search them granularly. In this screenshot, the user is searching for logs that relate to their Kafka cluster with the word “error” in them.
Note that the screenshots in this section are actually of Kibana, not Elasticsearch. You can think of Elasticsearch as the backend that stores the data and powers search, while Kibana is the frontend for visualization and UI / filtering. They are almost always a package deal.
2) Infrastructure metrics#
If you’re running your application on complex infrastructure like Kubernetes, or really any distributed system, developers will collect metrics on the performance of each individual element in that system like CPU usage or utilization. Elasticsearch allows you to store that data and use Kibana to visualize it hierarchically.
3) SIEM#
SIEM stands for Security Information and Event Management, and it’s the practice of documenting and analyzing any access to your internal systems. At larger organizations, IT admins look at logins from external computers and other data points to find patterns and prevent breaches. You can store this type of data in Elasticsearch and easily visualize and filter it in Kibana:
These are 3 primary examples, but there’s so much more you can do with Elasticsearch. Elastic has a great post on their blog explaining some of the cool stuff their users do with the product.
The ELK Stack and Elastic’s business model#
The Elastic or ELK stack#
While Elasticsearch is the database and search engine here, it’s meant to be used as part of a stack of other related, open source tools. Elastic calls it the Elastic stack , but long time readers will recall that they used to call it the ELK Stack which is...a name. It used to stand for Elasticsearch, Kibana, Beats, and Logstash, or in other words their acronym department is out to lunch? Anyway, it’s worth understanding the 3 tools in there that aren’t Elasticsearch:
Kibana is an open source visualization tool meant to be used on top of Elasticsearch. It’s really most accurately described as the frontend for Elasticsearch, and makes it much more interactive. You can search and filter in the UI, build visualizations and graphs, and even do Machine Learning like anomaly detection. Most everyone using Elasticsearch is also using Kibana.
LogStash is a utility for helping you actually get your data into Elasticsearch. You install it on the servers from which you desire to send data from, and it helps you collect that data, transform it, and then send it to your Elasticsearch cluster. It has pre-built connectors for common log formats and can infer structure that helps you store this stuff in Elasticsearch properly. No screenshots here, as it’s a headless tool.
You can think of Beats as a lightweight version of Logstash. It does the same thing – help send data from your sources to Elasticsearch – but it’s a series of smaller, single purpose utilities instead of a bigger general purpose one. The smaller size and ease of use can be beneficial if the data you’re sending to ES is simple.
Elastic’s business model#
All of the ELK stack products are open source and free, if you so desire to run them on your own infrastructure. Elastic makes money in the same way as most commercial open source vendors like Confluent or MongoDB – by charging you to abstract the infrastructure and run it for you. There’s a lot of configuration that goes into self hosting this technology and fine tuning it to your use cases, and a lot of teams don’t want the hassle.
Using Elastic’s cloud product, you pay for the infrastructure you use (more money for larger VMs, etc.). They have a few tiers of plans that start at $20/mo or so and go way up to hundreds of thousands of dollars.
Elastic Cloud has grown significantly, now accounting for 43% of total revenue in 2024 — up from just 22% in 2020 . The majority of new customers are choosing Elastic Cloud rather than self-managed deployments. For enterprise customers, Elastic will deploy these products on your own infrastructure or do a hybrid managed cloud situation.
Elastic's competition + license change#
The search and analytics space is a crowded one, and Elastic has a good deal of competition particularly from major cloud providers.
Most notably (and egregiously) is Amazon, who has a history of drama with Elastic, specifically free riding on Elastic’s open source software and contributing 0 back to it (which they also did with MongoDB et al). Essentially, AWS launched their own managed version of the open source Elasticsearch software and didn’t collaborate at all on it .
Elastic sued them (!) for trademark infringement in 2021, and led Elastic to change their license to prevent things like this from happening in the future. Though Elastic recently decided to change their license back to a more permissive one, OSS license switches is a trend we’ve seen a lot of the past few years: Hashicorp, Redis, and MongoDB to name a few.
