For the last two years the default answer in AI engineering was to use the OpenAI API and ship it. Costs scale linearly with that choice, and on every serious agentic project I have looked at lately the margins are going. Burn millions of tokens a day on tool-calling workflows and a third-party API stops being architecture and becomes a liability. Self-hosting LLM infrastructure has gone from a messy research project to something you can actually run in production.
Most dev teams wait for the first five-figure API bill before they think about hardware. By then they are usually pushing sensitive data that should never have left the VPC, whether that is patient records or proprietary code. The privacy case for running your own models is the stronger one. If you want to squeeze your current spend first, my guide on LLM performance hacks covers that, and it is worth doing before you go near bare metal.
Which benchmarks matter for agents
Most leaderboards are noise if you are shipping agents. What matters is whether the model invents function arguments, not whether it can recite string theory. Ignore the general MMLU number and read these instead:
- Berkeley Function Calling Leaderboard (BFCL v3): the reference for structured tool use and nested invocations.
- IFEval (Instruction Following Eval): how strictly the model sticks to formatting constraints. If your agent has to return valid JSON every single time, watch this one.
- τ-bench (Tau-bench): end-to-end competence across multi-turn simulated environments.
- SWE-bench Verified: only relevant if your agents modify code or work through GitHub issues, and essential if they do.
How far you can quantize
Quantization trades VRAM against logic degradation, so memory is only half the question. The usual mistake when self-hosting LLM nodes is going too thin. I have watched logic chains break outright at Q2 or Q3 because the long tail of specialized knowledge gets compressed out of existence.
Protip: stay at Q4_K_M (4-bit quantization) or above. Below that, structured output reliability starts to decay, and that is the one property your agent pipelines depend on. A 70B parameter model at Q4 needs roughly 42GB of VRAM. Budget for the KV cache too: long context windows can take another 15-20GB during peak generation.
Hardware: GPUs and cloud instances
An H100 is overkill for most of this. For a single-machine deployment the L40S or the A100 (80GB) is usually the right buy. Google Cloud Platform (GCP) is the only major provider currently selling single-GPU A100 instances (a2-ultragpu-1g), which makes it the cheapest sandbox for self-hosting LLM workflows.
For agentic AI experiments, spot instances cut compute costs by up to 70%. The catch is that your agent logic has to be reschedulable, so it resumes from a checkpoint when the instance gets evicted.
Open weight models worth running in 2026
The open-weight ecosystem moved fast. As of March 2026, this is what I am putting in front of clients:
- Qwen 3.5-27B: a dense hybrid transformer that punches above its weight class. It matches GPT-5 mini on SWE-bench and stays stable under tool calling.
- GLM-4.7-Flash: a 30B Mixture-of-Experts (MoE) model that activates only about 3B parameters per token, so multi-turn reasoning stays quick even at 128k context.
- GPT-OSS-20B: OpenAI’s own open-sourced model. Reliable, competitive, and it exposes configurable reasoning levels (low, medium, high).
Production deployment with vLLM
Ollama is fine for dev and test. Production runs on vLLM, which handles memory fragmentation through PagedAttention. Without that you will not hold throughput once concurrent agent requests arrive.
# Serving Qwen 3.5-27B with vLLM
vllm serve Qwen/Qwen3.5-27B-GGUF \
--dtype auto \
--quantization k_m \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--port 8000 \
--api-key your-production-key \
--enable-auto-tool-choiceThe “Phantom Claude” strategy
A codebase already locked into the Anthropic API does not need a refactor. Put LiteLLM in front as a translation proxy: it takes Anthropic-formatted requests and maps them onto your local vLLM OpenAI-compatible endpoint. The code still thinks it is talking to Claude while the tokens land on your own GPU.
Cost analysis: is it actually cheaper?
The crossover lands somewhere between 40M and 100M tokens per month. Take a mid-size team running 20 production agents at 500k tokens a day: a self-hosted A100 instance on GCP costs about $2,450 a month on committed use, and the equivalent API bill comes in north of $2,700. That comparison also ignores the latency you gain and the rate limits you stop hitting.
If this self-hosting LLM work is eating your dev hours, hand it to me. I have been wrestling with WordPress and high-performance infrastructure since the 4.x days.
Where to start
Self-hosting is a requirement now for anyone scaling privacy-first AI. Start small: one GCP machine, one A100, vLLM, systemd. Validate the agent pipeline end to end with no external API in the path, and the token tax stops looking inevitable.
