Everyone talks about open models and forgets the boring truth: the model is the easiest part. In 2026 the open-source AI stack is deep, mature and genuinely production-grade — if you know which layer does what. Here's the stack we actually see working at teams shipping open models, from weights to users, without paying a closed-API toll. Consider this the field manual, not the brochure.
Layer 0: hardware and the boring math
Before any software decision, do the arithmetic. One 8B model in FP8 serves comfortably on a single 24–48GB GPU at useful concurrency. A 70B-class dense model in INT4 fits on one 80GB card, tight but workable with offloading and small batches. A 400B MoE wants two to eight 80GB GPUs with fast interconnect, or a serious rental bill. Map your expected concurrency and latency SLO to a hardware shape before you fall in love with a checkpoint, and remember that tokens per second per dollar, not peak benchmark scores, decides your unit economics.
Storage and distribution matter too: pulling multi-hundred-GB weights onto fifty nodes is its own little infrastructure project. A local model registry or aggressive caching saves real money and real deploy-time pain.
Common failure modes to pre-empt
After enough deployments, the same mistakes recur. Teams skip the eval set and "just try it" for a quarter, then can't tell whether a regression came from the model, the prompt or the retrieval layer. Teams pin open weights to whatever tag was latest at launch, then get surprised by breaking tokenizer or chat-template changes in a community re-upload. Teams run one GPU at 90 percent utilization and discover that p99 latency triples under minor traffic spikes because there was never headroom. And teams adopt a beautiful framework with no exit plan, coupling their whole product to an abstraction that went stale eight months ago.
Every one of these is a process failure, not a technology failure. The open stack rewards teams that treat it like production infrastructure — versioned, measured, boring — and punishes those who treat it like a demo.
Structuring costs across the whole stack
Real deployments also spend money on the layers nobody demos. Weights are free to download; they're not free to store, replicate or move. Serving replicas are not free even when idle. Observability retention, guardrail models, re-embedding after a corpus change — each is modest alone and real in aggregate. The good news is that every layer in this stack is open, which means every layer is negotiable, replaceable and tunable. That's the structural advantage closed stacks can't match, and it compounds with scale.
One honest warning: "we self-host because it's cheaper" dies the first time you need twenty-four-seven on-call, GPU failure handling and a compliance audit. Self-hosting wins on steady-state volume; it loses on operational immaturity. Price the whole lifecycle, not the GPU hours.
Layer 1: the weights
The foundation is model choice, and the 2026 shortlist is honestly embarrassing in the best way. Llama 4 — Maverick and Scout — covers general US-origin workhorses. Qwen 3 spans multilingual breadth and small-dense efficiency. DeepSeek V3 and R1 own the cost-efficient general and reasoning slots. GLM and Kimi K2 anchor the Chinese open-wave story, and Mistral keeps mattering for European deployments with clean, predictable licensing. All of it lives on Hugging Face, which is now effectively the package registry of this era — fine-tunes, merges, quantizations and the occasional license of dubious clarity, all under one roof.
One licensing note that still bites people: "open" ranges from Apache 2.0 (Qwen, many Mistral releases) through MIT (DeepSeek V3 and R1) to Llama's custom community license with its very-large-company carve-outs. Read the license on the exact checkpoint you're pulling, not on the family name or the blog post announcing it. Ten minutes now beats one awkward call with counsel later.
Layer 2: quantization
Almost nobody outside the lab serves full-precision weights. FP8 is the sane default on Hopper and newer GPUs. AWQ and GPTQ 4-bit variants dominate when memory is tight. GGUF runs on CPUs and consumer GPUs through llama.cpp and Ollama. The quality loss at 4-bit for most instruction-tuned models is small enough that your users will never notice, and your hardware bill drops by roughly half — which on a fleet of inference nodes is the difference between "pilot" and "budget line."
A practical tip: benchmark your quantized model on your eval set, not on vibes. Most teams find INT8 indistinguishable from FP16 and INT4 within a point or two on structured tasks — enough to ship, rarely enough to regret, and occasionally enough to change which model is the "right" choice for your GPU budget.
Layer 3: serving
vLLM is the default serious-serving engine, and for good reason: continuous batching, PagedAttention, solid OpenAI-compatible APIs, and tensor parallelism that just works across multi-GPU boxes. TensorRT-LLM squeezes out more throughput at the bleeding edge if you're willing to live inside NVIDIA's ecosystem and tolerate the build friction. SGLang is the rising pick for structured generation and agent workloads heavy on prefix reuse — its radix cache and constrained decoding are genuinely lovely for JSON-emitting pipelines.
llama.cpp and Ollama cover the local and edge end — laptops, dev workstations, offline-first products. In practice, most teams end up with two serving paths: vLLM for the shared GPU fleet, Ollama for developer machines, and the same weights pulled from the same registry so evals transfer cleanly between them.
Layer 4: orchestration and agents
LangChain and LlamaIndex still exist and are still fine, but the center of gravity has shifted to lighter plumbing: direct OpenAI-compatible client calls, plus thin frameworks — or none at all. Tool-calling reliability on Qwen 3 and Llama 4 is now good enough that a surprising number of production "agents" are just a while-loop, a JSON schema and a validation function. Frameworks earn their keep on complex graphs and stateful human-in-the-loop flows; they're overkill for "extract fields from this PDF."
For RAG, Postgres with pgvector honestly handles most scale you'll ever need, with transactional consistency and one less system to operate. Qdrant and Weaviate earn their keep at the higher end — hundreds of millions of vectors, heavy filtering, distributed shards — or when you want their hybrid search out of the box rather than assembling it yourself.
Layer 5: fine-tuning
Full fine-tunes are rare on apps teams; LoRA and QLoRA are the workhorses. Unsloth has made consumer-GPU fine-tuning embarrassingly accessible — real adapters in an evening. Axolotl and Hugging Face TRL handle the more involved pipelines, and both have matured dramatically. The hard part isn't the training run; it's the dataset. Fifty perfect examples of your exact task beat fifty thousand scraped ones every single time, and most failed fine-tunes are data problems wearing a training-script costume. Budget for labeling, schema design and eval, not just for GPUs.
Layer 6: evaluation
This is where the amateurs separate from the operators. Eyeballing a few outputs doesn't scale, and "it feels better" is not a release criterion. Teams running open models in 2026 keep a frozen eval set of a few hundred real examples pulled from production traffic, score structured outputs programmatically, run LLM-as-judge for open-ended responses, and gate deploys on regression thresholds. lm-evaluation-harness covers the academic benchmarks, but your own boring eval set is the one that prevents 2 a.m. incidents. If you adopt one discipline from this entire stack, make it this one.
One detail that saves tears: version your eval set alongside the models, and rerun it on every dependency bump. vLLM upgrades, CUDA upgrades and driver changes have all silently shifted model behavior for teams that assumed inference was deterministic infrastructure.
Layer 7: observability and guardrails
Langfuse and OpenTelemetry-based tracing give you per-request visibility into tokens, latency, cost and cache hit rates — the difference between "the model is slow" and "prompts from customer X are burning 60 percent of our KV cache." Prompt-injection guards and PII filters sit at the edge of anything user-facing, and they're cheaper to add before launch than after the first incident review.
And open-weight models will happily produce confident nonsense at scale, so output validation — schemas for structured responses, refusal checks for sensitive flows, grounding checks for RAG answers — belongs in the request path, enforced in code, not in a policy document nobody reads at 2 a.m.
Layer 8: rollout and change management
The stack doesn't end at request handling; it ends when a model swap survives contact with production. The teams doing this well roll new checkpoints out the same way they roll out services: shadow mode first, where the candidate runs alongside the incumbent on live traffic and every output pair gets logged. Then a canary cohort behind a feature flag, with automatic rollback if tool-call error rate, refusal rate or p95 latency moves past a pre-agreed threshold. Then general availability, announced with a re-eval report attached, because Friday deploys without measurement are how incident channels get created.
Ownership matters as much as process. Give the open stack a named owner with authority to pin versions and schedule upgrade windows, and give that owner a small budget for GPUs and eval labeling. The failure mode of self-hosted open AI isn't the technology; it's a half-adopted shared toy with no owner, drifting silently until an audit or an outage exposes it. Treat the stack like the database tier — boring, monitored, staffed — and it will behave like one.
The honest takeaways
- Open weights plus vLLM plus a decent eval set is a complete, competitive stack today — no closed API required for a large class of products.
- The scarce skill isn't model choice; it's evaluation discipline and data quality. Models are replaceable every quarter; your eval harness is an appreciating asset.
- Self-hosting pays off at volume and for privacy-sensitive workloads; it's overkill for low-traffic side projects where an API is honestly cheaper.
- Keep one model from each camp — DeepSeek, Qwen, Llama — warm in your eval harness. The rankings genuinely shuffle every few months now.
- Vendor your dependencies: pin model versions, containerize serving, and treat upgrades as release events with re-eval gates.
The open stack isn't a purist ideology anymore. It's just... the stack. Closed APIs still earn their place for frontier reasoning at low volume, and for teams that never want to touch infrastructure. But if you haven't priced out serving Qwen 3 on your own GPUs against your actual traffic, you're doing procurement by vibes. Run the numbers, build the eval set, and let the data make the argument.