What does LangChain do?

TL;DR#

LangChain helps developers build AI experiences into their products.

Think of it as all the things needed to turn language models into reliable agentic software products (first chatbots, then agents, now agent harnesses)

LangChain makes basically everything but the model itself:

• Open source frameworks (LangChain, LangGraph) to build apps with LLMs
• Observability tooling (LangSmith) to monitor + evaluate agents.
• Infrastructure (sandboxes, model router, deployment tooling) to run agents in production.

LangChain plays in a very competitive + fast moving space, and is following elders like Databricks and dbt on the long, tough road to build a company around an open source project.

In the dark ages of LLMs#

It started innocently enough.

When AI labs like OpenAI shipped their first model APIs (GPT-3), they kicked off the era of AI products. Beyond just talking to ChatGPT in OpenAI’s apps, you could use them via API – which meant that you could build your own products around them. Any developer could prompt a model and get a response back (unfortunately called a "completion" at the time).

The first wave of products built with these APIs were chatbots: think Notion’s AI where you can ask a question, and answer it with context from your workspace. This is a language chain (a lang…chain…).

But building one of these things in 2023 was really hard. An LLM is a useful but unpredictable component, like your uncle Rick at Thanksgiving.

Let’s say you want to build the first version of "Ask Technically" - a chatbot on technically.dev that will answer questions ("what’s the difference between a data lake and a data warehouse?") using the Technically archives. The Technically archives don’t exist within the brain of the model; we need to connect them with the model somehow.

It’d work something like this:

The flow would be:

• Ask a question to the model
• Query the Technically archives to return relevant documents
• Provide those docs to the LLM to generate an answer based on them
• Validate the answer format + return it to the user

If you’ve ever heard of RAG or Retrieval Augmented Generation…you’re looking at it.

In code this looks roughly like so - you define the chain, and then run the chain on an input (a starting prompt):

const chain = RunnableSequence.from([
  queryTheArchive,   // query the database
  answerPrompt,      // hand the docs to the model with instructions
  model,             // call the model
]);

await chain.invoke("What's the difference between a data lake and a data warehouse?");

Building a workflow like this around an LLM API seems simple, but presents at least 4 problems:

1. RAG is kind of hard: It’s not a set it and forget it thing - the quality of answers depends on how you “chunk” content (into paragraphs, entire posts, etc) and other configuration details.
2. Saving and reusing prompts: You need to pass the model instructions, like how it should respond (“do not admit that you’re a Belieber”). You need to save + version these instructions somewhere, so that you can measure how well they perform.
3. Putting guardrails on responses: Early LLMs were really bad at staying on task - if we wanted the model to only respond with 1-sentence answers, it wouldn’t always follow that instruction.
4. Model switching: What if you want to use Anthropic models instead of OpenAI?

It was clear that developers needed to solve for these - or else LLM applications would be at risk of behaving like Rick after a couple glasses of Chartreuse.

The original LangChain framework (alongside LlamaIndex and other frameworks) filled that need, and the people rejoiced.

One problem to the next#

What’s funny about this AI cycle is how quickly solving one problem exposes the next problem. OG LangChain lets developers build apps with LLMs, but what were they supposed to do about the rest of the software lifecycle that was also drastically changing thanks to AI?

We still needed a way to observe what’s actually happening with these user sessions, and ideally to score whether the model’s responses were good or not. We probably want a way to evaluate changes we make to the product before we ship them, too.

We basically needed new developer tooling built for LLM workloads, due to LLMs being unpredictable (or “non-deterministic”). Whereas old school software was very predictable and deterministic.

That’s what LangChain set out to do with LangSmith, which started as a simple observability tool to see what these RAG apps were actually doing. It allowed users to trace these LLM chains from input through to output.

A trace is pretty simple, it just tells you exactly what happened and how long it took. You record the user’s input, the model’s output, and call it a day.

If you have enough of these traces logged and stored somewhere, you could imagine having an LLM analyze them to suggest improvements to your LLM application, which is what LangSmith’s newer Engine product does.

This is a competitive space - observability was a pretty well-established category prior to this AI wave, and observability tools like Comet, PostHog’s self-driving mode, and many other tools all see this “self-improving LLM app” space as a huge opportunity.