You wouldn’t think there’d be so much fanfare – and such colossal budgets – around data storage. And yet there are 6 public companies in the U.S. that have built billion+ dollar database businesses, from $ORCL to $ESTC to $SNOW. Not to mention the public clouds that sell database products ($AMZN, $GOOG, $DOCN, etc.), plus supporting tooling for infrastructure ($DDOG, $NEWR, etc.). How should you understand the differences between all of these seemingly similar databases? What do developers actually use databases for anyway? And why would one be better than another? This post will run through everything that an investor needs to know to effectively understand the difference between different databases and how they’re positioned in the market. So far in this category, we’ve covered Elastic, MongoDB, Snowflake, and Confluent, plus AWS. If you’re still getting up to speed on what databases are, you’ll want to read Technically’s [beginner's guide to databases](/posts/beginners-guide-to-databases). We also have more in depth posts on a few of the categories listed there, like production databases,relational databases, NoSQL databases, and more. You can also refer back to the [database database](/database-database) if you like having a cheat sheet. ## What developers use different databases for There’s an incredible diversity of database use cases, and it can get overwhelming to understand them all. Why would you use Redis for this? Couldn’t you just use Postgres? Do I need a Vector Database? If you’re trying to understand a new data store, or the difference between two, ask these two questions. ### How does the data get used? The first question to ask is: what’s the point of this data in the first place? What apps does it power, what does it help a developer accomplish? Is it for: **A SaaS-style app, like Gmail or Salesforce?** This is a production database, and you have many options, from relational databases like PostgreSQL and MySQL to NoSQL platforms like MongoDB or Firebase. **Long analytics queries from a data team?** This is a data warehouse, and you might be picking between Snowflake, Redshift, BigQuery, and Clickhouse. **A real-time credit card fraud detection system?** This could be any combination of things, including an in-memory database like Redis, a streaming solution like Kafka and Confluent, and an operational data store like [Materialize](https://materialize.com/). **A RAG pipeline for a Large Language Model?** This could be a dedicated Vector Database like [Pinecone](https://www.pinecone.io/), or one of the large platform databases like PostgreSQL (with [Timescale](https://www.timescale.com/), perhaps). Typically, earlier in a company lifecycle, they’ll have a simpler, smaller set of use cases. Any company with a web application is going to need a database to store their user and app data. Then maybe as they grow, they add a data team who is curious about analytical trends over time. At huge scale, they want to start checking for payment fraud. It’s mostly larger companies who have *many* of these use cases at once. This is why it’s not always MongoDB vs. Redis for a large organization: they could conceivably be using a dozen different databases for different use cases all at once. It’s important to note that you *can* use a regular old relational database for *all* of these things – it just won’t work as well as dedicated solutions. PostgreSQL *can* be a data warehouse, and was for many years; but analytics is important enough to most companies to pay up for Snowflake. The reverse is not true: Snowflake *cannot* be a production database for your SaaS app. ### What does the data look like? “Data” is not monolithic. Depending on what you’re using the data for (see above), the shape, size, and nature of data is wildly different. A few questions worth asking: - Is the data large, or small? - Is it well organized, or haphazardly put together? - Does the data need to be 100% correct at all times, or can it be mostly correct? - Does it need to be near real-time, or can queries take 10 seconds? - Are there any regulatory constraints like HIPAA it needs to adhere to? Data that’s highly organized, needs fast queries, and is of moderate size is a good fit for relational databases. Data that’s small and needs to be real-time is a good fit for in-memory databases. NoSQL is a good fit for huge, disorganized data. The list goes on. In the beginner’s guide to databases, I organized all database use cases into 3 categories: 1. **Databases that power a user-facing app** – production databases that store the data you need for your app to run. 2. **Databases that power analytics** – databases for analysis, machine learning, and anything a data team does. 3. **Databases that power operations** – databases for monitoring, logs, security, and any internal processes that enable the above. I’m writing *this* post a couple of years later and I think the lines are more blurry, especially with the “one database to rule them all” wave happening. Speaking of which… ### One database to rule them all What has made life difficult for analysts over the past few years – and honestly, developers too – is that larger database companies have been **expanding** the scope of their use cases. Products like MongoDB are trying to be all in one databases, solving for multiple different use cases instead of the initial one they were created for (in MongoDB’s case, transactional NoSQL). Look at MongoDB’s homepage, and how their messaging reflects the “one database to rule them all” idea: ![MongoDB's landing page](/images/posts/the-data-store-market/typ_Image_1.gif) From developers I’ve spoken to, a lot of this is just marketing – MongoDB is still severely feature limited in some of these areas like time series or AI. But it doesn’t have to be *as good* as use-case-specific competitors, it just needs to be *good enough* for someone who is already (considering) using MongoDB for something else. ## Open source vs. managed service vs. closed source Setting up a database on good infrastructure used to be a major ordeal involving a lot of specialized expertise. Today, developers can pick from an incredibly diverse spectrum of “hard mode” to “easy mode” depending on how much they’re willing to pay (and company constraints). ![the spectrum between open source and managed databases](/images/posts/the-data-store-market/U0R_Image_2.jpeg) ### Left side: open source, self managed On the **left side** of the spectrum you’ve got your hard mode: open source databases, running on your own infrastructure or low level cloud infrastructure. This is where you’ll see software like PostgreSQL, MySQL, Elasticsearch, Redis, running on a company’s data centers or on a virtual machine on AWS. It is entirely *your job* to install the software, keep it updated, choose the right infrastructure, and network it with the rest of your app. By the way, if you want *real* hard mode, you can try a database that’s designed to run as a distributed system across multiple servers in the cloud like Kafka. As far as I’m aware every major open source Kafka deployment that’s handling meaningful throughput has a dedicated engineering team whose sole job is to make sure it works. In theory, there are three major benefits to the left side of this spectrum: 1. It’s the **cheapest**. Open source software is free. You just pay for the infrastructure, and (more subtly) the team it takes to keep it running. 2. It gives you the most **flexibility**. You can customize open source software however you’d like, run it on whatever infrastructure in whatever cloud you’d like, and enable or disable whatever features you like. 3. It’s the most **secure**. Here, your software and infrastructure are your own. You can’t get breached because your vendor slacked off (other than your cloud vendor). I say “in theory” because vendors on the right side will tell you that in the long run, it’s actually cheaper and more secure to outsource infrastructure to them. Speaking of which… ### Right side: closed source, fully managed On the **right side**, you’ve got your easy mode: serverless closed source databases, running on someone else’s infrastructure that you do not need to worry about. This is where you’ll see software like MongoDB, Snowflake, Confluent, and smaller upstarts like [Planetscale](https://planetscale.com/) and [Supabase](https://supabase.com/). Some of these vendors are taking an open source database like Postgres and building proprietary systems to **run and manage** it for you. Confluent runs on open source Kafka, Planetscale runs on open source MySQL and Vitess, and Supabase plus several AWS offerings run on open source PostgreSQL. But others have built **entirely closed source**, proprietary databases like Snowflake, Oracle, and SQL Server. When you choose something on the right side of the spectrum, you get one, very important benefit: **peace of mind**. You don’t have to worry about setting up or managing infrastructure *at all* – in many cases you can go from idea to working database in just a few minutes. But with that peace of mind comes a separation from the underlying software, and thus less flexibility (it’s someone else’s product, you can’t change it) and *significantly* higher cost. Although these vendors would tell you that your total cost of ownership is lower than open source, since you don’t need to hire and maintain an infrastructure team to manage your database. In practice many companies choose things in the middle of this spectrum, not just at the poles. And a single open source database can show up all over it, depending on who is selling it and how. Consider PostgreSQL, for example: - If you run vanilla open source Postgres on a basic server, you’re somewhere around the left side. - If you want to run distributed Postgres on many servers, you’re all the way on the left side (you may have fallen off into the depths). - If you use [RDS](https://aws.amazon.com/rds/postgresql/), you’re now closer to the middle. It’s managed, but you still have infrastructure decisions to make. - If you use [Neon](https://neon.tech/), you’re now all the way on the right side of the spectrum. It’s still Postgres, but it’s fully managed and serverless. All of this comes down to what tradeoffs an organization wants to make, and how much they’re willing to pay (and sacrifice) to outsource database management to a vendor. ## Final words on data stores Here’s the Gartner Magic Quadrant for cloud database management systems in 2024: ![Gartner data store magic quadrant](/images/posts/the-data-store-market/7S9_Image_3.png) It’s crowded, and barely even scratches the surface of the databases out there. For example, they classify Elastic as an “Insight Engine” (whatever that means) and not a data store. It obviously doesn’t include any of the fast growing database startups, even though they are legitimately making a splash in the enterprise. Analyzing databases is hard. It’s busy out there. Hope this helps.