Short answer: yes, a VPS without a GPU can run a local language model, but only small ones, and slowly. With Ollama on a Linux VPS with 6 GB RAM ($13 a month), models of about 1–4 billion parameters run well; with 10 GB ($26), 7–8 billion parameter models fit. Expect answers in seconds to minutes rather than instantly. For chat-speed answers from large models you need a GPU server or a hosted API; none of our plans have a GPU.
So when is it worth it? When the data shouldn't leave a server you control, when the job runs in the background, or when you want an AI tool that costs a fixed amount a month instead of per token.
01What Fits in How Much RAM
The model has to fit in memory along with the system. Ollama's own guidance is about 8 GB of RAM for 7B models and 16 GB for 13B models. With the usual 4-bit quantised models, a rough guide:
| Plan | RAM | Model size that fits | Examples of the class |
|---|---|---|---|
| 2 vCPU, 6 GB, 40 GB NVMe ($13) | 6 GB | 1–4 billion parameters | small Llama, Qwen, Gemma or Phi models |
| 6 vCPU, 10 GB, 40–80 GB NVMe ($26–39) | 10 GB | up to 7–8 billion | 7B/8B general and coding models |
| Windows 10 vCPU, 36 GB ($104) | 36 GB | 13B and larger | Ollama also runs on Windows |
Model names change every few months; the Ollama library lists the sizes of each model. Pick the largest version whose download size is clearly below your free RAM.
Full root SSH access · NVMe storage · DDoS protection · No setup fee
02How Fast Is It?
Speed on a CPU depends on the number of cores, memory bandwidth and model size: twice the parameters is roughly half the speed. On a few vCPUs, small models produce text at reading pace or a bit below, and a 7–8B model is noticeably slower. The first answer after loading also takes longer, while the model is read into memory.
Good fits for CPU speed:
- Summaries and classification of emails, tickets or documents in a background queue
- A private assistant for notes and drafts where waiting a bit is fine
- Agents and automations (n8n, scripts) that call the model a few times an hour
- Embeddings for search over your own files, which are cheap even on CPU
Bad fits: live chat for many users, long code generation with large models, or anything with image generation.
03Install Ollama
On Ubuntu 24.04 or Debian 12, as root:
curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama
ollama pull qwen2.5:3b # or any model from the library that fits
ollama run qwen2.5:3bThe installer sets up a systemd service called ollama that starts at boot. Type a question at the prompt; /bye leaves it.
04Use the API Safely
Ollama serves an HTTP API on 127.0.0.1:11434, so by default only the server itself can use it. Keep it that way: the API has no login, and anyone who can reach the port can use your server's CPU.
From your own computer, open a tunnel and use the API as if it ran locally:
ssh -L 11434:127.0.0.1:11434 root@your-server-ip
curl http://localhost:11434/api/generate -d '{"model":"qwen2.5:3b","prompt":"Say hi","stream":false}'If other programs on the same server use it (a bot, a script), point them at http://127.0.0.1:11434. Need it from elsewhere? Put it behind a reverse proxy with HTTPS and a password, or behind your own WireGuard VPN.
05Tips for CPU Inference
Smaller context, faster answers
Long prompts cost time on every request; keep them short.
One model at a time
Two loaded models compete for RAM; Ollama unloads idle ones after a few minutes.
Watch memory with free -h
If the server swaps, take a smaller model or a bigger plan.
Disk
each model is a few gigabytes. ollama list shows them, ollama rm deletes one.
06Is It Private?
The model runs on your server, so prompts and answers never go to an AI provider. On Rdp.Vision, the server itself is rented with an email or Telegram and paid with crypto, without ID. We don't log in to your server or look at what runs on it. More on running agents on a server: AI agents.
Yes, small models on CPU with Ollama or llama.cpp. They're slower than on a GPU but work fine for background jobs.
- run llm on vps
- ollama vps
- llm without gpu
- cpu only llm server
- self host llm
- private ai server