What's Kafka and what does Confluent do?

The core Confluent product: data streaming

Kafka, and thus Confluent, exists to solve two fundamental problems facing almost every data infrastructure team at every company.

1. There’s a lot of data, and it’s all happening very quickly

As storage has gotten cheaper, we’ve been collecting more and more data. Most software companies record every single website visit and click, and some go even deeper. Once you have more than few users interacting with your product, you’re talking about millions of different events per day. Storing and managing that size and velocity of data is hard.

2. Data needs to move around to be valuable

Even if you’re a wiz at collecting and storing your data, there’s a problem: you’re going to need to move it around for it to be valuable. Where data gets initially collected and stored is rarely where it’s going to be useful.

Kafka solves these problems by creating a central registry for all of this data – you can think of it like one of those conveyor belt sushi places. Any consumers that need to use the data (like apps, databases, or ML models) can just take the plate they need (although really, they’re just taking a copy of it). This is sometimes called a publish-subscribe model, often shortened to pub-sub.