Why do we need a vector database in the first place?#
Using your data to improve AI models#
The point of a vector database is to make it easier for you to integrate your company’s data into a large language model. But why would you want to do that in the first place? The general answer: to make models more accurate and customized to your specific needs.
Back in prehistoric machine learning days, every model you built was trained on your unique data. Today, most people use off-the-shelf foundation models like GPT-4, Claude, or Gemini. These models are trained on the entirety of the internet! And they don’t have access to your company’s internal data, which is where the real value is. You can get decent generic responses out of them for some tasks, but for real business use cases you won’t get anywhere without your data being integrated somehow.
There are two state of the art ways that teams are powering LLMs with their proprietary data. The first is fine tuning, where you actually retrain a model to take your data into account, updating the model weights as you go and creating an entirely new model. The second, and more popular for now, is RAG, or Retrieval Augmented Generation. It’s a clever way of including relevant data for your prompt inside the prompt itself, without needing to retrain the model.
A primer on vectors and embeddings#
The data that you might use to power an AI model is usually a subset of data that your company is already storing: like user interactions, customer support transcripts, product documentation. So why do you need a specialized database to store this stuff? Why can you just use the existing database(s) that it sits in?
The answer lies in the format of data that ML and AI models use. Models can only work with numbers: not text, images, or videos like us human beings. If you want to give a model a piece of data, it needs to be in a numeric format, end of story. A fancy word for a group of numbers in computer science is a vector, hence the moniker vector database.
Luckily for us, pretty much all types of data can be converted into a numeric format using a process called embedding. An embedding takes a dataset like this:
And turns it into something more machine readable like this:
[0,3,5,3,1,2,7,38,45,5,7,3,4]
[3,2,3,4,55,1,52,23,78,11,1]
[2346,23,7,8,44,32,56,123,2]
...
Embedded (sorry) in this numerical representation is not only the data points themselves, but also the relationship between them. Many readers will be familiar with the old author quote, “If I had more time, I would have written a shorter letter.” This is the gist of embeddings: representing the initial data set in a condensed “summary” that maintains (most of) the integrity of the original.
There are a bunch of different ways to create embeddings, and all involve some sort of algorithm applied to the data to reduce it down to a numerical format, usually with some linear algebra going on behind the scenes. You can even use LLMs…to create embeddings for LLMs 🤯.
Storing and working with embeddings#
To bring your nicely embedded data to bear on an AI model, you’ve got two things you need to do:
- store the embedding data somewhere, and then
- retrieve it whenever you need it for a model.
Whether you’re doing fine tuning or RAG, you need relatively sophisticated ways to search through your embeddings and pull out whichever ones are relevant to the model or prompt – and kind of in real time, too. “Regular” databases don’t play nicely with embeddings on either of these storage or retrieval dimensions.
When it comes to storage, embeddings are big, kind of goofy formats that don’t fit nicely into a table (relational databases) or a document (NoSQL databases). And when it comes to retrieval, they need to be queried regularly with semi-sophisticated search algorithms, so you can’t just put them in a data warehouse (like Snowflake) or object storage (like S3).
This is why vector databases exist. They’re specially designed for this format of data, aimed towards use cases in machine learning and AI.
