API or local model: choosing your engine

Every AI project starts with an architecture decision: calling a model via an API (Claude, GPT, Gemini) or running an open-weight model (Llama, Mistral, Qwen, DeepSeek) on your own machines. This choice shapes cost, privacy, and quality. Here's the honest breakdown.

The API: top quality without infrastructure

• State-of-the-art quality. The world's best models remain closed or prohibitively expensive to host. Via the API, you access them in ten lines of code.
• Zero infra. No GPUs to buy, no scaling to manage: you pay per token consumed.
• The trade-offs. Your data passes through a third party (read the retention policy, enable "zero retention" options where they exist), prices and models evolve without your input, and a provider outage becomes your outage.

Local models: total control, total demands

• Privacy. Nothing leaves your machine: decisive for medical, legal, and defense work.
• Predictable cost. An amortized GPU costs the same whether it handles a thousand or a million requests.
• The trade-offs. Quality lags behind top-tier models at comparable size, and you're the one managing updates, scaling, and failures.

Quantization: fitting a giant into a normal machine

A model stores its parameters as precise numbers (16-bit). Quantization rounds them to 8, 5, or 4 bits: the model becomes 2 to 4 times lighter and faster, often with minimal quality loss. That's what lets an 8-billion-parameter model run on a decent laptop with tools like Ollama or llama.cpp. To serve many simultaneous users, you move to an optimized inference server like vLLM.

The decision rule

Start with the API: iterate fast, validate the use case, measure real costs. Switch to local (or hybrid: local for routine work, API for hard cases) only if privacy demands it or if your per-token bill sustainably exceeds the cost of a machine and its upkeep.