Self Hosted LLM: How to Run a Large Language Model on Your Own Infrastructure

A self hosted LLM is an open-weight language model (Llama, Qwen, Mistral, Gemma, Phi and others) running on hardware you control instead of behind a vendor API. In 2026 the practical rules are simple: a 1–4B-parameter model runs on a 4–8 GB VPS with no GPU, a 7–8B model needs 8 GB of RAM and patience, and anything above ~14B wants a GPU or a dedicated server. Hosting your own model wins on privacy, offline use and unlimited experimentation; it loses on raw speed and, below a certain volume, on cost.

This guide covers what a deployment actually consists of, a model-to-RAM-to-server sizing table, the honest cost comparison against API pricing, and a ten-minute Ollama setup you can run on any Ubuntu VPS.

What Self Hosting an LLM Actually Means

The term “self hosted LLM” might evoke an image of simply downloading a model and launching a chat interface, but practical deployments involve several interconnected components:

Model Files

These are the model weights used for inference. They range from compact quantized models to full-scale deployments with heavy hardware demands.

Inference Engine

This software component runs the model efficiently to generate responses and is key to the user experience.

API or Application Layer

Many deployments include endpoints for applications, scripts, or internal integrations to enable smooth interaction with the model.

User Interface

Some setups provide a direct chat interface, allowing non-developers to interact with the model easily.

Operational Infrastructure

Production environments integrate monitoring, authentication, logging, storage, and deployment management to ensure reliability and security.

Local experimentation and production deployments differ considerably. A single-user test setup on lightweight hardware is not comparable to an environment designed for multiple users or automated workflows.

Common Use Cases for Self Hosted LLMs

Self hosting is most practical when there is a clear operational need. While developers sometimes experiment with self hosting, most real-world deployments address specific technical or business challenges:

Internal Knowledge Assistants

Deploy models to help employees search internal documentation, procedures, and knowledge bases without exposing sensitive data externally.

Custom API Infrastructure

Teams can build internal applications and automation workflows that bypass third-party APIs while maintaining full infrastructure control.

Coding Assistants

Self hosting is useful for private code generation, debugging support, and other productivity tools where keeping source code internal is critical.

Offline or Restricted Environments

In air-gapped networks or regulated settings where cloud access is limited, self hosted models offer a viable solution for internal AI capabilities.

AI Experimentation and Prototyping

Developers testing open models, refining prompt workflows, or building retrieval pipelines find self hosting enables experimentation without ongoing API dependency.

Infrastructure Requirements for Self Hosted LLMs

Hosting an LLM is primarily an infrastructure decision where the model is just one part of the system. The overall performance, usability, and scalability depend on the environment that hosts the model.

CPU vs GPU Requirements

• CPU-Based Inference: Works well for smaller quantized models, testing environments, and lightweight tools where response time is less critical. This option can make early experimentation more cost effective.
• GPU Acceleration: Necessary for larger models and scenarios with heavier concurrent usage. While GPU acceleration boosts inference speeds, it comes at a higher cost.

Memory and Storage Planning

Larger models require more RAM or VRAM, though quantization can reduce these demands while potentially affecting performance. In addition, storage planning must account for:

• Model files
• Multiple model versions
• Embeddings and logs
• Container images, cache artifacts, and monitoring data

Performance and Concurrent Usage

An LLM that functions well in a personal testing scenario may struggle under heavier loads. Key performance considerations include response latency, concurrent requests, API throughput, resource contention, and background automation workloads.

Model Size → RAM → Server: The Sizing Table

Every “can I run an LLM on a VPS” question comes down to one number: the size of the quantized model file. A 4-bit (Q4_K_M) build needs roughly its file size in RAM plus 1–2 GB for the context window, the inference engine and the operating system. The download sizes below are the current Ollama library builds; the speeds are what we see on CPU-only KVM instances with no GPU, so treat them as a floor rather than a benchmark.

Two consequences follow. First, a 4 GB plan is a 1–3B-model plan: perfectly good for classification, extraction, summarising tickets or powering an internal search assistant, but not for a chat experience that feels like a hosted model. Second, speed scales with memory bandwidth more than core count, so an 8B model on an 8 GB VPS produces two to five tokens per second, which is fine for background automation (n8n workflows, nightly reports, embedding generation) and frustrating for live chat. If live chat for a team is the goal, budget for a GPU from the start.

Deployment Approaches for Self Hosted LLMs

Not every self hosted deployment is the same; the right approach depends on workload size, performance expectations, and budget. Common deployment approaches include:

Local Development Environments

Local setups allow developers to test smaller models and validate integrations without provisioning remote infrastructure. These are useful for prototyping but rarely scale for shared business use.

VPS Deployments

VPS environments suit lightweight inference, API testing, and smaller internal tools. CPU-based workloads and modest automation tasks run effectively on VPS setups, although performance ceilings may quickly become apparent with larger models or heavy concurrent usage.

Dedicated Servers

For sustained workloads or more resource-intensive deployments, dedicated hardware offers greater compute consistency, expanded memory, and stronger isolation.

GPU Cloud Infrastructure

For high-performance tasks, GPU cloud infrastructure is often the best choice. Applications such as public-facing AI services or larger internal assistants can benefit from GPU acceleration despite increased costs.

Best Tools for Self Hosting LLMs

Several tools simplify self hosted LLM deployments:

Ollama

Ollama simplifies model downloads, version management, and inference startup, making it a practical entry point for developers and teams building smaller internal tools.

vLLM

Designed for production environments, vLLM centers on inference efficiency and high throughput, making it suitable for AI-powered applications with high-performance requirements.

LocalAI

For teams that seek familiarity with OpenAI-style API compatibility while retaining full control over deployment, LocalAI can reduce the friction of migrating existing applications.

Open WebUI

A user-friendly frontend provided by Open WebUI makes the self hosted setup more accessible to non-developers who prefer a direct interface over API-only access.

Popular Open Models for Self Hosting

Choosing the right model is crucial since hardware requirements and performance vary widely. Popular models include:

Llama

A well-established option with broad compatibility for self hosted experimentation.

Mistral

Recognized for maintaining strong performance relative to model size, making it efficient without excessive resource usage.

Qwen

Favored for its multilingual performance and competitive functionality in practical applications.

Gemma

A lighter model suitable for experimentation or smaller deployments where hardware resources might be limited.

Model selection should consider workload demands, hardware constraints, language requirements, and acceptable response latency.

Challenges and Limitations

While self hosting an LLM offers significant advantages, there are operational tradeoffs:

• Hardware Costs: Inference workloads, especially those needing GPU acceleration, can be expensive.
• Performance Expectations: Private deployments may not match the highly optimized performance of commercial AI platforms, especially under heavy load.
• Quantization Tradeoffs: Reducing resource demands through quantization can affect reasoning quality or output consistency.
• Maintenance Overhead: Regular updates, dependency management, container handling, and system monitoring require ongoing effort.
• Reliability Requirements: As internal systems depend more on these deployments, uptime and stability become critical.
• Model Limitations: Self hosting controls the deployment environment, but it does not change the inherent behavior of the model, which may still produce occasional inaccuracies.

Self-Hosting vs API: The Cost Math

The most common objection on r/selfhosted is that hosting your own model is “pointless” because small-model API pricing is cheap. For low volumes that is true, so here is the arithmetic rather than the slogan. The API column uses an illustrative small-model rate of roughly $0.60 per million output tokens (check your provider’s current price sheet); the self-hosted column is the fixed VPSus plan price, which does not change with usage.

The honest summary: below roughly 50 million tokens a month, an API is cheaper and faster, and you should self host only for privacy, offline operation or the freedom to experiment without a meter running. Above that line, or whenever the prompts contain data you would not paste into a third-party service, a fixed-price server pays for itself. Remember to count the hour or two a month you will spend on updates and backups.

Run Your First Model in 10 Minutes (Ollama on Ubuntu)

Ollama is the fastest way to get from an empty VPS to a working OpenAI-compatible endpoint. On a fresh Ubuntu 22.04 or 24.04 server with at least 4 GB of RAM:

# 1. Install Ollama (binds to localhost:11434 by default)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model that fits your RAM (see the table above)
ollama pull llama3.2:3b

# 3. Talk to it from the shell
ollama run llama3.2:3b "Summarise what a VPS is in two sentences."

# 4. Call it like an API (OpenAI-compatible endpoint)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.2:3b","messages":[{"role":"user","content":"Hello"}]}'

To give non-developers a chat interface, add Open WebUI in Docker and point it at the local Ollama socket:

docker run -d --name open-webui --restart always \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Three rules keep this safe on a public server: leave Ollama bound to 127.0.0.1 (never expose port 11434), put Open WebUI behind Nginx or Caddy with HTTPS and its built-in login, and set a firewall that only opens 22, 80 and 443. The same endpoint plugs straight into automation tools: n8n’s Agent node can call your Ollama URL as a local model, which is how most of our customers turn a self-hosted n8n instance into a private AI workflow engine without per-message fees. For a team chat front end with history and user management, see our guide to self-hosted chat platforms.

Is Self Hosting an LLM Worth It?

Whether self hosting is the right choice depends on your unique requirements. For teams needing to experiment with open models, organizations requiring internal AI tools, or groups that must keep data strictly private, self hosting offers significant benefits. Conversely, for users looking for simple, hassle-free AI deployment, managed platforms may be more appropriate. Ultimately, self hosting suits circumstances where control is more important than convenience.

Build Your Self Hosted AI Environment on Flexible VPS Infrastructure

Self hosted LLM deployments need predictable compute, fast NVMe storage and full root access, which is exactly what the VPSus LLM Hosting plans are sized for. Start on KVM4 (4 vCores, 4 GB ECC RAM, 40 GB NVMe, 1 Gbps, $40/mo) for 1–4B models and internal automation, or KVM8 (8 vCores, 8 GB, 80 GB NVMe, $80/mo) when you want to run an 8B model such as Llama 3.1 or Qwen3 with room for Open WebUI alongside it.

Frequently Asked Questions

Can I run an LLM on a VPS without a GPU?

Yes. Models up to about 8B parameters run on CPU with llama.cpp-based tools such as Ollama; the limit is RAM, not the GPU. Expect 15–25 tokens per second from a 1B model, 6–12 from a 3–4B model and 2–5 from an 8B model on a few vCores, which is enough for automation and internal tools but slow for live chat.

How much RAM does a 7B or 8B model need?

About 8 GB for a 4-bit quantized build: the model file itself is 4.4–5.2 GB and the context window, the runtime and the OS take the rest. With 16 GB you can run a 12–14B model or keep two smaller models loaded at once.

Which self hosted LLM is best for a small server?

For a 4 GB server, Llama 3.2 3B, Qwen3 4B or Gemma 3 4B give the best quality per gigabyte; Phi-4-mini is strong at reasoning and code for its size. For an 8 GB server, Llama 3.1 8B and Qwen3 8B are the general-purpose defaults, with Qwen the better choice for multilingual work.

Is self hosting an LLM cheaper than using an API?

Only at volume or when privacy forces the decision. Below roughly 50 million tokens a month a small-model API usually costs less than a $40–$80 server; above that, or when prompts contain data that cannot leave your infrastructure, a fixed-price VPS is cheaper every month it runs.