This is the existential question hanging over the inference-as-a-service space. The honest answer: for vanilla open-weight model serving, the moat is thin and getting thinner. But there are real differentiators for the ones executing well:
Where the defensibility is weak:
- "We run vLLM on rented GPUs" — Anyone can do this. Together, Fireworks, Anyscale, and a dozen others all started here. If that's still the pitch in 2026, they're in trouble.
- Open-weight model serving (Llama, Mistral, etc.) — Commodity. AWS Bedrock, Azure AI, and GCP Vertex all offer the same models with better latency (co-located with enterprise data) and no egress fees. Hard to beat the hyperscalers at their own game when you're renting their own GPUs.
Where some actually have a moat:
1. Inference optimization at the kernel level. Fireworks, in particular, rewrites CUDA kernels and builds custom serving stacks (not just vLLM wrappers). They can compress models, fuse operations, and squeeze 2–3x more throughput per GPU. That margin delta is everything when you're renting by the hour.
2. Multi-model routing / compounding. Some are building layers above raw inference — chaining models, caching semantic embeddings, routing queries to the cheapest model that can handle the task. Groq does this with deterministic hardware-software co-design. This starts to look less like "hosting" and more like an inference platform.
3. Fine-tuning and custom model deployment at scale. Together AI's real business increasingly isn't just inference — it's training and fine-tuning as a service, with LoRA adapters deployed at scale. The inference becomes sticky when a customer has 100 custom fine-tuned models deployed.
4. Latency edge. GPUs rented from CoreWeave in the same datacenter as the inference provider get lower latency than a customer going to CoreWeave themselves, because the inference provider colocated the hot model weights. Small edge, but real for chat/agent use cases.
5. Hardware-accelerated inference. Groq and Cerebras aren't renting NVIDIA GPUs at all. They built their own silicon. That's genuinely defensible — Groq's LPUs deliver token generation speeds that NVIDIA hardware literally cannot match, because they architected for inference rather than training. Cerebras similarly delivers 1,800 tokens/second on Llama 70B, which is absurd. Nobody can replicate that on rented H100s.
The likely outcome: The "me too" inference APIs (rent GPUs + vLLM + serve Llama) consolidate or die. The survivors will either own silicon (Groq, Cerebras), own the full training-to-inference pipeline (Together), or get acquired by hyperscalers for the kernel-level optimization talent. It's a classic infrastructure layer squeeze — the hyperscalers commoditize the middle, and only the hardware-advantaged or deeply integrated survive.