Self-hosting AI, without renting a GPU

Every self-hosted-LLM-vs-API cost guide currently online reaches for the same framing: rent an H100 at $2-10/hour, model your token volume, find the break-even, usually somewhere around $20,000 a month of API spend. That's a real answer for a real question, and it's also the wrong question for most people who type "self host AI" into a search bar. Most of what people actually want to self-host doesn't need a rented GPU at all. So instead of building another calculator, I installed Ollama and actually ran a real model.

What actually happened

Llama 3.1 8B, Meta's open-weight model, on ordinary consumer hardware, no cloud, no GPU rental. The download is 4.9GB. Loaded into memory it uses 5.26GB, comfortably inside a 16GB machine with room to spare. Once it's warm, a real 69-token response to an actual prompt ("why are append-only logs useful in accounting systems") took 1.6 seconds end to end, at roughly 50 tokens per second.

$ ollama pull llama3.1:8b   # 4.9GB, one time
$ ollama run llama3.1:8b "your prompt here"

# or as an API any of your own tools can call:
$ curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "...",
  "stream": false
}'

50 tokens/second on hardware you likely already own is not a frontier model's intelligence, and it doesn't need to be for a large share of what actually gets sent to an API: drafting, classifying, extracting fields, summarizing internal notes, first-pass triage before a human looks at it. For that entire category, the real cost after the one-time download is zero per request, forever, with no data leaving the machine.

This is where "hosting services" comes back in

The exact setup above doesn't need to live on your laptop. Move it to an ordinary, boring, no-GPU VPS, a Hetzner CX32 (4 vCPUs, 8GB RAM) runs about €6.80/month at current pricing, and you have an always-on private inference endpoint that any of your own apps can call over plain HTTP, the same API shown above. Worth being honest about the tradeoff: that VPS has no Apple Silicon Metal acceleration behind it, so CPU-only inference on a cheap shared vCPU will run slower than the 50 tokens/second above, sometimes considerably. The value isn't matching that speed, it's an always-on endpoint with zero per-token billing for tasks that can tolerate the pace.

One thing self-hosting does not remove

If that endpoint ends up serving more than one internal tool, the same mistake shows up that I've written about for hosted AI agents: one shared, unscoped credential for every caller. Self-hosting moves where the model runs. It does not remove the question of what each caller is actually allowed to ask it, and a self-hosted setup with no access boundaries is not meaningfully safer than an API key taped to the fridge.

The honest boundary: this is not a frontier-model replacement, and nobody should self-host their way out of a task that genuinely needs top-tier reasoning. It is a real, working, zero-marginal-cost option for the much bigger pile of tasks that don't, and the only way to know which pile a given task falls into is to actually run it once and look, the way the numbers above came from actually running it rather than modeling it.