Running Local LLMs: A Practical Guide for Production

Why Run Local LLMs?

Cloud LLM APIs are convenient, but they come with costs that compound at scale:

For South African businesses handling sensitive data — legal documents, medical records, financial information — the privacy argument alone justifies self-hosting. POPIA compliance is simpler when data never leaves your infrastructure.

Hardware Requirements

The GPU is the bottleneck. Everything else is secondary.

Consumer vs Enterprise GPUs

Consumer GPUs (RTX series) offer the best price-performance for small-to-medium deployments. An RTX 4090 at R35,000 runs 13B parameter models comfortably. Enterprise GPUs (A100, H100) are faster but cost 5-10x more — justified only at scale.

The sweet spot for most South African businesses: an RTX 4090 (24GB VRAM) running a 13B model with FP16 quantization. Total hardware cost: under R50,000. Pays for itself in 3-6 months versus cloud API costs.

Cost-Effective Configurations

Software Stack

Three main options, each suited to different use cases:

Ollama

The easiest way to run local LLMs. One command to install, one command to run a model. Ollama is Docker for LLMs — it handles model downloading, quantization, and serving with a simple CLI.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Run a model
ollama run llama3:13b

Best for: development, testing, small-scale production. Not ideal for high-throughput serving.

llama.cpp

The reference implementation for LLM inference. Runs on CPU and GPU, supports every quantization format, and is the most flexible option. The trade-off: more setup, more configuration, more things to break.

Best for: custom deployments, edge devices, CPU-only environments.

vLLM

The production serving engine. PagedAttention for efficient memory management, continuous batching for high throughput, and OpenAI-compatible API. If you need to serve 100+ requests per second, vLLM is the answer.

Best for: production serving, high-throughput workloads, multi-tenant deployments.

Deployment Architecture

Single Server Setup

For most businesses, a single GPU server is enough. The architecture is simple:

┌─────────────┐     ┌──────────────┐
│   Clients    │────▶│   vLLM /     │
│   (API)      │     │   Ollama     │
└─────────────┘     └──────────────┘
                           │
                           ▼
                    ┌──────────────┐
                    │  GPU Server  │
                    │  (NVIDIA)    │
                    └──────────────┘

Load Balancing

Multiple GPU containers behind a load balancer. Round-robin distribution works for uniform workloads. For heterogeneous workloads (some requests need 70B, others need 7B), route by model size.

Scaling Strategies

Performance Benchmarks

Benchmarked on an RTX 4090 with a 13B parameter model:

Compare that to cloud APIs at $0.01-0.06 per 1K tokens. At 10 million tokens per month, self-hosting costs R1.50 in electricity versus $100-600 in API fees.

Challenges & Solutions

Conclusion

Running local LLMs in production is no longer experimental. The hardware is affordable, the software is mature, and the cost savings are real. For South African businesses, the combination of POPIA compliance, cost reduction, and performance makes self-hosted LLMs a practical choice.

If you’re evaluating self-hosted AI infrastructure, let’s talk. We help businesses deploy local LLMs that run reliably in production — from hardware selection to deployment to monitoring.