How to use open-weight AI models

So you read part 1, and despite your best inclinations decided that you wanted to keep reading. You now know what open weights means, the difference between model and harness, and why open weights might be compelling. Perhaps you even want to try using some of these models yourself…well lucky you, because that’s what part 2 is about.

Logistically, when we talk about “running” models, the harness always runs on your computer (or increasingly, phone). The model itself might run on your computer, but it might run on a more powerful computer in the cloud if it's a large or sophisticated model that won't fit on your Macbook.

Whether or not you need a beefy model that runs in the cloud or a shrimpy model locally, comes down to, as most things do, what the hell you're trying to do.

Picking a model#

There are an annoyingly high number of models out there now, and that number grows every day.

Some are general purpose large language models, and some are specifically tuned for different tasks like coding or math.

Some are small, meaning their output is less "good" but can usually be run faster or on cheaper servers. Some are large, meaning their output is more "good" but they cost more to host and use and response times are longer.

The main things that contribute to a model's performance are the following:

Parameter count (7B, 14B, 70B) is the main quality lever, also more simply referred to as size. More parameters, smarter model, but also slower and hungrier for memory. For a laptop, 7B-14B is the range that actually runs. Larger models need bigger machines.

Quantization is how compressed the weights are, ergo your tradeoff between quality and speed. Think of it like an MP3 versus a FLAC file: smaller, faster, and you probably can't tell the difference in normal use. 4-bit is the default pick for local models, it roughly halves the size for a quality hit you won't notice most of the time. Push below 4-bit and the model starts being dumb, 8-bit gets you closer to full precision but doubles the size for a gain that's honestly kind of marginal.

Context window is how much text the model can hold in its head at once, your prompt plus the whole conversation so far. Bigger context allows it to chew through longer docs or remember more of a long back-and-forth, but it also costs more memory and can slow things down.

Reasoning variants show their work before answering, actual step by step thinking instead of jumping straight to a response. Good for math, logic, anything with real multi-step complexity. Bad call for casual chat, where it just slows things down for no benefit.

Tool calling means that the model is trained to know when it needs to tell the harness to use a tool (used to write word documents, search the web, hit an API, or take some other action). It’s what lets the model step outside its own weights and feel like its actually doing something.

By way of a small worked example: A fridge tracker that texts you when stuff in your fridge is going bad? It doesn't need any of the heavy stuff, so it could use a small model, 4-bit, small context, no reasoning, no tools. A significant software engineering project, or working through a math paper needs the opposite end of every one of these dials.

For the purposes of this discussion, we'll pick one at either end, a fast small model for daily tasks - think: draft this email, write a recipe with these ingredients, or format this paper APA style. And a large one, write this research paper, refactor this codebase. Our small model will be Qwen 3 1.7B (that’s one and change billion parameters, so relatively small) and for the large one lets give Kimi K3 a spin because it's new and exciting and boasts a whopping 2.8 trillion parameters.