Running Local LLMs in Production

You can run production-quality large language models on your own hardware. No API keys. No rate limits. No data leaving your network. The cost? A one-time GPU investment that pays for itself in 3-6 months compared to cloud API bills.

Published: June 24, 2026 Tech Stack: Ollama llama.cpp vLLM Docker NVIDIA GPU

What You’ll Learn

  • Hardware Sizing

    GPU VRAM requirements for 7B, 13B, and 70B models with Rand pricing.

  • Software Stack

    Ollama vs llama.cpp vs vLLM — when to use each.

  • Real Benchmarks

    45 tokens/second on an RTX 4090 with cost per 1M tokens in Rand.

Why Run Local LLMs?

Cloud LLM APIs are convenient, but they come with costs that compound at scale:

FactorCloud APIsSelf-Hosted
Cost at scale$0.01-0.06 per 1K tokensFixed hardware cost
Data privacyData sent to providerData stays on your network
Rate limitsPer-account quotasLimited only by GPU
LatencyNetwork round-tripLocal inference
CustomisationFixed modelsFine-tune for your use case

For South African businesses handling sensitive data — legal documents, medical records, financial information — the privacy argument alone justifies self-hosting. POPIA compliance is simpler when data never leaves your infrastructure.

Hardware Requirements

The GPU is the bottleneck. Everything else is secondary.

Model SizeGPU VRAMSystem RAMStorageGPU Options
7B params6-8 GB16 GB50 GB SSDRTX 3060, RTX 4060
13B params12-16 GB32 GB100 GB SSDRTX 3090, RTX 4090
70B params48+ GB64 GB200 GB SSDA6000, multi-GPU

Consumer vs Enterprise GPUs

Consumer GPUs (RTX series) offer the best price-performance for small-to-medium deployments. An RTX 4090 at R35,000 runs 13B parameter models comfortably. Enterprise GPUs (A100, H100) are faster but cost 5-10x more — justified only at scale.

The sweet spot for most South African businesses: an RTX 4090 (24GB VRAM) running a 13B model with FP16 quantization. Total hardware cost: under R50,000. Pays for itself in 3-6 months versus cloud API costs.

Cost-Effective Configurations

  • Starter (7B): RTX 3060 (R5,000 used) + 16GB RAM + 50GB SSD = R15,000 total
  • Standard (13B): RTX 4090 (R35,000) + 32GB RAM + 100GB SSD = R50,000 total
  • Enterprise (70B): 2x A6000 (R200,000) + 64GB RAM + 200GB SSD = R300,000 total

Software Stack

Three main options, each suited to different use cases:

Ollama

The easiest way to run local LLMs. One command to install, one command to run a model. Ollama is Docker for LLMs — it handles model downloading, quantization, and serving with a simple CLI.

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Run a model
ollama run llama3:13b

Best for: development, testing, small-scale production. Not ideal for high-throughput serving.

llama.cpp

The reference implementation for LLM inference. Runs on CPU and GPU, supports every quantization format, and is the most flexible option. The trade-off: more setup, more configuration, more things to break.

Best for: custom deployments, edge devices, CPU-only environments.

vLLM

The production serving engine. PagedAttention for efficient memory management, continuous batching for high throughput, and OpenAI-compatible API. If you need to serve 100+ requests per second, vLLM is the answer.

Best for: production serving, high-throughput workloads, multi-tenant deployments.

Deployment Architecture

Single Server Setup

For most businesses, a single GPU server is enough. The architecture is simple:

┌─────────────┐     ┌──────────────┐
│   Clients    │────▶│   vLLM /     │
│   (API)      │     │   Ollama     │
└─────────────┘     └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  GPU Server  │
                    │  (NVIDIA)    │
                    └──────────────┘

Load Balancing

Multiple GPU containers behind a load balancer. Round-robin distribution works for uniform workloads. For heterogeneous workloads (some requests need 70B, others need 7B), route by model size.

Scaling Strategies

  • Vertical scaling — Bigger GPU, more VRAM, larger models
  • Horizontal scaling — More GPU servers, load-balanced
  • Model tiering — 7B for simple tasks, 70B for complex reasoning
  • Quantization trade-offs — FP16 for quality, INT4 for throughput

Performance Benchmarks

Benchmarked on an RTX 4090 with a 13B parameter model:

MetricValue
Throughput45 tokens/second
P50 latency12ms per token
P99 latency25ms per token
Concurrent requests8 (with continuous batching)
Cost per 1M tokensR0.15 (electricity only)

Compare that to cloud APIs at $0.01-0.06 per 1K tokens. At 10 million tokens per month, self-hosting costs R1.50 in electricity versus $100-600 in API fees.

Challenges & Solutions

Model Updates

New model versions drop regularly. Updating without downtime requires a blue-green deployment: spin up the new version, test it, switch traffic, decommission the old version. Docker makes this straightforward.

Hardware Failures

GPUs fail. Hard drives die. The mitigation: keep a cold standby server with the same model loaded. When the primary fails, switch DNS or load balancer config. Recovery time: 5-10 minutes.

Security Hardening

A local LLM server is still a server. Apply standard security practices: firewall rules limiting access, TLS for API traffic, authentication for multi-tenant deployments, and regular security updates.

Conclusion

Running local LLMs in production is no longer experimental. The hardware is affordable, the software is mature, and the cost savings are real. For South African businesses, the combination of POPIA compliance, cost reduction, and performance makes self-hosted LLMs a practical choice.

If you’re evaluating self-hosted AI infrastructure, let’s talk. We help businesses deploy local LLMs that run reliably in production — from hardware selection to deployment to monitoring.

Need Help Deploying Local LLMs?

We design and deploy self-hosted AI infrastructure for South African businesses. Whether you need a single-GPU setup or a multi-node cluster, we can help you run LLMs on your own hardware.

Related Reading