A Software Engineer’s Guide to Running Quantized 70B Models on Edge Devices Using Ollama and vLLM

Running a 70 BILLION parameter model on a laptop or an on-premise server used to sound like a joke. But today, with the right quantization strategy and the right inference engine, it is actually possible. Not just barely possible either. Usably fast.

This guide is written for software engineers who already understand the basics of machine learning but want a practical, technical walkthrough of deploying large language models on resource-constrained hardware. We will cover QUANTIZATION methods, the difference between Ollama and vLLM, hardware planning, and real configuration steps you can follow right now.

Why 70B Models on Edge at All?

Is it not easier to just call a cloud API? Sure. But there are good reasons to avoid that route:

Privacy comes first. Many enterprise and healthcare workloads can not send data to third-party servers. Running locally means your data stays local.

Latency is the second reason. A local inference call can return in under a second on decent hardware. A cloud roundtrip adds unpredictable latency, especially in production.

Cost at scale matters too. API calls at high volume get expensive fast. A one-time hardware investment often pays off within months.

So yes, edge deployment of 70B models is a real engineering requirement, not just a curiosity.

What is QUANTIZATION and Why Does it Matter

A standard 70B model like LLaMA 3 70B, in full FP32 precision, would require roughly 280 GB of VRAM. That rules out every consumer device and most small servers.

QUANTIZATION reduces the precision of model weights from 32-bit or 16-bit floating point numbers to lower-bit integers. This shrinks the model size dramatically.

Here is a quick comparison:

For most practical edge deployments, Q4_K_M (a 4-bit GGUF quantization variant) gives the best balance between quality and memory usage. It is the recommended starting point before you experiment with lower bits.

Two Engines: Ollama vs vLLM

Both are popular. Both work. But they are built for different use cases. Choosing the wrong one is a common mistake.

Ollama

OLLAMA is designed for simplicity. It wraps llama.cpp underneath and gives you a clean CLI and API interface. It is ideal for:

• Local development and testing
• Single-user inference on laptops or workstations
• Models in GGUF format
• Teams without dedicated MLOps infrastructure

OLLAMA handles quantization through pre-quantized GGUF models. You do not need to quantize models yourself. You just pull and run.

ollama pull llama3:70b-instruct-q4_K_M
ollama run llama3:70b-instruct-q4_K_M

That is it. The model runs via the built-in llama.cpp backend with CPU/GPU offloading managed automatically.

vLLM

vLLM is a high-throughput inference engine built for production serving. It uses PagedAttention for efficient KV cache management and supports continuous batching, which means it can handle many concurrent requests efficiently. It is ideal for:

• Production API servers
• Multi-user or multi-request workloads
• Teams with NVIDIA GPUs and CUDA infrastructure
• Models in Hugging Face format (with GPTQ or AWQ quantization)

vLLM supports quantized models through GPTQ and AWQ formats. You load them like this:

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/LLaMA-3-70B-Instruct-GPTQ \
  --quantization gptq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

This spins up an OpenAI-compatible REST API on port 8000.

Quick Comparison Table

Hardware Planning for 70B Edge Inference

This is where many engineers underestimate the challenge. You need to think about three things: VRAM, RAM, and storage.

For a Q4_K_M quantized 70B model, the model weights alone are around 38-40 GB. With KV cache and runtime overhead, plan for at least 44-48 GB of usable VRAM.

Realistic hardware options:

• 2x NVIDIA RTX 3090 or 4090 (24 GB each): Tight but workable with Q4_K_M
• NVIDIA A6000 (48 GB): Solid single-card option for INT4
• Mac Studio / Mac Pro with M2 Ultra (192 GB unified memory): Excellent for OLLAMA with Metal backend
• NVIDIA A100 80GB: Handles FP16 directly, no quantization needed

For CPU offloading via OLLAMA, you can run a 70B Q4 model on a machine with 64 GB RAM and a 16 GB GPU. The GPU handles the upper layers and the CPU handles the rest. Throughput will be lower (around 3-6 tokens/second), but it works for non-latency-critical tasks.

Step-by-Step: Running 70B on Ollama with GPU Offloading

Lets say you have a Linux workstation with one NVIDIA GPU (16-24 GB VRAM) and 64 GB system RAM. Here is how to set it up.

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Step 2: Set GPU layer count

OLLAMA uses the OLLAMA_NUM_GPU environment variable to control how many layers to offload to GPU. For a 70B model, you might start with:

export OLLAMA_NUM_GPU=20

You can increase this until you hit VRAM limits. The rest of the layers run on CPU.

Step 3: Pull and run the model

ollama pull llama3:70b-instruct-q4_K_M
ollama run llama3:70b-instruct-q4_K_M

Step 4: Use the REST API

OLLAMA exposes a local API at http://localhost:11434. You can call it with:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3:70b-instruct-q4_K_M",
  "prompt": "Explain KV cache in transformer inference.",
  "stream": false
}'

Step-by-Step: Running 70B on vLLM with GPTQ

This setup assumes two NVIDIA GPUs (e.g., 2x RTX 4090) and CUDA 12.x.

Step 1: Install vLLM

pip install vllm

Step 2: Launch the server with tensor parallelism

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/LLaMA-3-70B-Instruct-GPTQ \
  --quantization gptq \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --dtype float16

The --tensor-parallel-size 2 flag splits the model across two GPUs using tensor parallelism. This is important for staying within VRAM limits.

Step 3: Call the API

vLLM is OpenAI API compatible, so you can use the standard SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")

response = client.chat.completions.create(
  model="TheBloke/LLaMA-3-70B-Instruct-GPTQ",
  messages=[{"role": "user", "content": "What is PagedAttention?"}]
)
print(response.choices[0].message.content)

Common Problems and How to Fix Them

Out of memory errors: Reduce the context length with --max-model-len 2048 in vLLM, or decrease OLLAMA_NUM_GPU layers in OLLAMA to offload more to CPU.

Slow token generation: For OLLAMA on CPU offload, this is expected. Consider moving to a higher-VRAM machine or a smaller quantization level (Q3_K_M).

Model quality degradation: Very low-bit quantization (INT2, Q2_K) can produce noticeably worse outputs on complex reasoning tasks. Benchmark your specific use case before going below Q4.

CUDA errors on multi-GPU setups: Make sure both GPUs are visible with nvidia-smi and that CUDA and driver versions are compatible with your vLLM installation.

When to Use Which Tool

Can you use both in the same project? Yes. A common pattern is to use OLLAMA for local development and testing, then switch to vLLM for production serving. The underlying model can be the same, though you may need to convert formats between GGUF and Hugging Face.

If you are building AI-powered video or image generation pipelines, efficient local inference can also power the prompt processing layer before sending to generation APIs. Handling text generation locally while using specialized tools for visual output is a common hybrid architecture in modern AI systems.

You might also be interested in how AI video generation models work under the hood and how GPU infrastructure powers modern generative AI, both of which connect directly to the hardware and model serving concepts discussed here.

Final Thoughts

Running 70B models on edge hardware is no longer a research experiment. With Q4_K_M quantization, the right inference engine, and careful hardware selection, it is a production-ready strategy for engineers who need privacy, low latency, or cost control.

OLLAMA gives you a simple path to start within minutes. vLLM gives you the throughput and production-readiness to scale. Understanding when to use which, and how to configure each, is a real engineering skill worth building today.

Start small. Profile your hardware. Then scale up.