Running LLMs Tempatanly in Logistik: Ollama + VPN + Air-Gapped Pelaksanaan

I've burned more GPU budget on "private cloud" demos than I care to admit. At AINNA, where we run NeuralOps infrastructure and a growing fleet of Sistem Berasingan, one of the hardest problems we solve is putting Large Bahasa Model inside logistics environments that cannot - and should not - phone home to OpenAI. This article is the technical blueprint I wish I had three years ago: Ollama + VPN + air-gapped deployment, built for real 3PL warehouses, freight brokers, and Malaysian PKS that handle customer manifests, customs forms, and supplier contracts.

1. Why Logistik Needs Tempatan LLMs

Logistik data is messy, regulated, and high velocity. A typical 3PL in Melaka processes thousands of airway bills, packing lists, and customs declarations every day. Most of those documents contain:

• Pelanggan names, IC numbers, and phone numbers (PDPA-sensitive).
• Pembekal pricing and contract terms (commercially sensitive).
• Customs HS codes, declared values, and origin data (legally sensitive).

Sending that to a public API - even with promises of "zero retention" - is a non-starter for many compliance officers I've worked with. Tempatan inference removes that risk. But it also introduces baharu ones: model management, GPU failures, context-window crashes, and VPN latency that can make a chatbot feel like it's typing on a typewriter.

We originally tried a hybrid approach: sensitive extraction on local models, summarization on Groq. It worked until a customs broker asked, "Where exactly did my Bill of Lading go?" That single audit question killed the hybrid design. Now our default architecture at AINNA is VPN-first, local-first. Awan LLMs are an explicit opt-in, not a default.

2. The Stack: Ollama, WireGuard, and Air-Gapped vLLM

2.1 Ollama as the Edge Inference Enjin

Ollama is not the fastest inference engine. It is not the most feature-complete. What it is, in my experience, is the most reliable way to get a model running on an edge box in under ten minutes. For logistics clients who need to deploy in a warehouse with no dedicated ML engineer on site, that matters more than raw throughput.

Our standard edge node is a workstation with an NVIDIA RTX A6000 (48 GB VRAM), 128 GB RAM, and a 2 TB NVMe. That box lives in the warehouse rack, behind the client's firewall, reachable only over the AINNA-managed WireGuard VPN. Ollama runs inside Docker with a restricted Modelfile tuned for structured extraction:

# /models/qwen-logistics/Modelfile
FROM qwen2.5-coder:14b
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
PARAMETER num_predict 1024
PARAMETER top_p 0.3
PARAMETER repeat_penalty 1.1
SYSTEM "You are a logistics document parser. Extract only the fields requested. Respond in valid JSON. Do not add explanations."

Why qwen2.5-coder:14b? In our benchmarks it beats llama3.1:8b on structured JSON extraction from mixed Bahasa Inggeris/Bahasa Melayu/Bahasa documents, and it fits comfortably in 48 GB with headroom for the KV cache. The 14B parameter size is the sweet spot for edge: good enough accuracy, small enough latency, cheap enough hardware.

2.2 VPN-First, Not Awan-First

Every AINNA deployment starts with a WireGuard hab-and-spoke topology. The hab is a hardened VPS or the client's own NeuralOps Awan Terurus node. The spokes are the warehouse edge boxes, finance office desktops, and any detached inventory servers. No spoke has a public IP. No spoke can reach the internet directly. Semua control-plane traffic - model pulls, telemetry, agen orchestration - flows through the hab.

# /etc/wireguard/wg0.conf on a warehouse spoke
[Interface]
PeribadiKey = 
Address = 10.200.0.5/32
ListenPort = 51820
DNS = 10.200.0.1

[Peer]
PublicKey = 
BenarkanedIPs = 10.200.0.1/32, 10.201.0.0/24
Endpoint = neuralops-hab.ainna.bond:51820
PersistentKeepalive = 25

The BenarkanedIPs line is the critical security control. It means the warehouse Ollama node can talk to the NeuralOps hab and the internal analytics subnet, but not to Google, GitHub, or a rogue mirror. When we need to update a model, we stage the GGUF or safetensors on the hab, sign it, and push it down through the VPN. The edge node never downloads from the public internet.

2.3 Air-Gapped vLLM Cluster for Heavy Workloads

Ollama is great for single-box inference. For a centralized cluster serving multiple warehouses - what we call the Lapisan Pelayan Ejen AI & LLM in NeuralOps - we use vLLM. A typical hab cluster is two A100 80GB GPUs in tensor-parallel mode, running llama-3.3-70b AWQ-quantized:

python -m vllm.entrypoints.openai.api_server \
  --model /models/Llama-3.3-70B-Instruct-AWQ-INT4 \
  --served-model-name llama-3.3-70b-logistics \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 32768 \
  --max-num-batched-tokens 8192 \
  --port 8000 \
  --api-key ${VLLM_API_KEY}

This cluster has no outbound NAT. Model weights live on an encrypted NFS mount. Logs ship to a local Loki instance, never to a SaaS. Backups go to a tape rotation. It is, by design, boring infrastructure - and boring infrastructure is what keeps auditors happy.

3. Model Pemilihan for Logistik Workloads

We do not deploy one model and hope. We run a model router in the Lapisan Ejen AI that picks the cheapest capable model for each task. Here is what actually works in production:

3.1 Pengekodan and Agentic Tugasans

• kimi-k2.7-code:cloud: We use this for complex agen reasoning when the client permits cloud usage. It is expensive but excellent at multi-step planning - e.g., "Reschedule this container shipment around a port strike and update the inventory reservations." On a cloud endpoint we see roughly $0.02 / 1K tokens depending on provider. We only route high-value decisions here.
• qwen2.5-coder: Our default on-premise code and extraction model. Handles JSON, SQL generation for the Sistem Inventori, and light report automation. Runs at ~120 ms/token on the A6000 edge node.

3.2 Penaakulan and Document Sedang parse

• deepseek-coder: Strong for long-context document understanding. We use it for customs declaration checks where the prompt includes 20+ pages of HS-code reference tables. Watch out: the 236B variant needs serious VRAM. We usually run the 33B distilled version locally.
• llama-3.3-70b: The "safe choice" for mixed Bahasa Inggeris/Bahasa conversations in the Papan Pemuka Ejen AI. Quantized to AWQ INT4, it fits on two A100s and gives acceptable quality for chat and classification. Not the cheapest to run locally - power and cooling are real costs - but lebih murah than leaking data.

Kos reality check: A 1,000-token call to a cloud API might cost $0.02 on Groq or $0.03+ on OpenAI. The same call on local Ollama costs $0 in API fees - but you already paid RM 50,000 for the server, RM 800/bulan for power, and my team's time to keep it patched. Tempatan is not free. It is a capex-and-compliance trade, not a price hack.

4. AINNA's VPN-First, Hub-and-Spoke vLLM Seni Bina

Here is the pattern we repeat across Malaysian logistics deployments. I call it "hab-and-spoke vLLM."

1. Hub: One or more GPU servers in the client's data center or a dedicated NeuralOps Awan AI/Ejen node. Runs vLLM, the model registry, the agen orchestrator, and the central vector database.
2. Spokes: Gudang edge nodes running Ollama for low-latency extraction and offline fallback. Each spoke connects to the hab via WireGuard. Spokes can operate autonomously if the VPN drops.
3. Control plane: AINNA NeuralOps manages WireGuard keys, Docker image updates, database backups, and 24/7 monitoring. We do not manage the client's network - we secure the overlay.

Client applications talk to the models through a thin OpenAI-compatible client. The base URL points to a VPN address, not a public endpoint:

import openai
import os

client = openai.OpenAI(
    base_url="http://10.201.0.5:11434/v1",
    api_key="ollama",
    timeout=30.0
)

resp = client.chat.completions.create(
    model="qwen2.5-coder:14b",
    messages=[
        {"role": "system", "content": "Extract consignee, HS code, and declared value as JSON."},
        {"role": "user", "content": doc_text[:6000]}
    ],
    max_tokens=512,
    temperature=0.0,
    response_format={"type": "json_object"}
)
print(resp.choices[0].message.content)

That base_url is reachable only inside the WireGuard tunnel. If an attacker compromises the warehouse LAN, they still cannot reach the model server unless they also have the WireGuard private key and the hab authorizes their IP. Defense in depth is not a slogan; it is the only reason I sleep on Friday nights.

5. Token Optimization and Kependaman Tuning

Tempatan inference saves money only if you are not wasting tokens. I have seen warehouses burn 30% of their GPU capacity on bloated prompts and oversized context windows.

5.1 Konteks Window Budgeting

Do not set num_ctx = 128k because the model supports it. KV cache scales quadratically with context length in attention, and on consumer GPUs that translates directly to slower generation and higher VRAM use. For airway bill extraction, we cap context at 8,192 tokens. For long-form customs analysis, we chunk documents into 4,000-token windows and run parallel inference, then merge results.

5.2 Kelompoking and KV Cache

vLLM's PagedAttention is the reason we use it for the hab. We configure max-num-batched-tokens and max-num-seqs based on observed traffic, not defaults. For a 3PL with 40 concurrent agens, we typically run:

--max-num-seqs 64
--max-num-batched-tokens 16384
--gpu-memory-utilization 0.90

That pushes the A100s to ~85% utilization without OOM crashes. Ollama does not support continuous batching the same way, so for edge nodes we limit concurrent requests in the reverse proxy (usually Nginx or Caddy) and queue the rest.

5.3 Quantization Dagangan-offs

We quantize almost everything that runs locally. AWQ INT4 and Q4_K_M are standard. The quality drop is measurable on coding tasks but rarely on logistics extraction, where the output schema constrains the model heavily. My rule of thumb: if the task is classification or structured extraction, quantize aggressively. If the task is open-ended reasoning or customer-facing chat, keep more bits.

6. Produkion Failures and What Actually Works

I will not pretend this has been smooth. Here are three failures that shaped our current architecture.

6.1 The 32K Konteks Crash

We deployed llama-3.3-70b with max-model-len 32768 and told the finance team they could dump entire annual ledgers into the Papan Pemuka Kewangan. It worked in testing. In production, the first user pasted a 28,000-token spreadsheet and the vLLM process was killed by the OOM killer. Root cause: long context plus high batch size exceeded VRAM. Fix: cap per-request context to 16K, add input length validation in the API gateway, and expose a token counter in the UI so users see the cost before they submit.

6.2 VPN Kependaman Surprises

A client in Johor routed all traffic through a hab in Kuala Lumpur because "the cloud is in KL." The round-trip added 35 ms. That does not sound like much, but LLM inference is a sequential token stream. For a 500-token response, 35 ms per token becomes 17.5 seconds of perceived latency. We moved the hab closer - same city, same ISP - and latency dropped below 4 ms per token. Lesson: place inference where the users are, not where the cloud provider is.

6.3 Model Drift on Air-Gapped Nodes

Edge nodes that never touch the internet also never get silent security patches. We once had a spoke running an outdated Ollama version with a known container escape. Because it was air-gapped, the CVE was theoretical - until a technician plugged in a USB drive. Now every detached system has an offline patch schedule: signed update packages staged on the hab, pushed over VPN, installed during maintenance windows, and verified by AINNA's monitoring layer before the node rejoins the mesh.

7. Keselamatan, PDPA, and Malaysian PKS Reality

Malaysia's Personal Data Perlindungan Act 2010 (PDPA) does not explicitly ban cloud AI, but it requires clear purpose limitation, consent, and security safeguards. In practice, that means a DPO asking uncomfortable questions about data residency. Tempatan LLMs give a clean answer: data never leaves the premises.

We layer the usual NeuralOps controls on top:

• Penyulitan SSL/TLS inside the VPN, because encryption should not stop at the tunnel.
• Pengasingan Docker for Ollama and the Papan Pemuka Ejen AI.
• Pencatatan Log Menyeluruh to a local SIEM, not a foreign SaaS.
• Sandaran Automatik of model configs, fine-tunes, and vector stores.
• Pemantauan 24/7 with alerts for GPU temperature, OOM events, and unauthorized API calls.

For a Melaka PKS running a single 3PL warehouse, the jumlah cost of a local Ollama deployment - server, switch, UPS, VPN setup, and managed support - is roughly RM 45,000–65,000 first year. That is not cheap. But a single PDPA fine or a leaked supplier contract can cost far more. We do not sell local LLMs as a cost saver. We sell them as a risk reducer.

8. Bottom Line

Running LLMs locally in logistics is not a hobbyist project. It is an infrastructure decision with real trade-offs in cost, latency, security, and model quality. My production-tested recipe is:

• Use Ollama on warehouse edge nodes for fast, offline, structured extraction.
• Use vLLM on a central GPU hab for heavy reasoning and multi-tenant agens.
• Sambung everything with a WireGuard VPN hab-and-spoke; never expose model APIs to the public internet.
• Keep the cluster air-gapped by default, with signed, staged updates.
• Pick models pragmatically: qwen2.5-coder for edge coding/extraction, deepseek-coder for long documents, llama-3.3-70b for general chat, and cloud models like kimi-k2.7-code:cloud only when the business case justifies the risk.
• Optimize tokens, context windows, and batch sizes like you optimize SQL queries - because wasted tokens are wasted GPU dollars.

At AINNA, this pattern powers our Sistem Berasingan, Sistem E-dagangs, Sistem Inventoris, and the Papan Pemuka Ejen AIs we ship to logistics clients across Malaysia. It is not the flashiest architecture. It is the one that still works at 2 AM when a customs officer asks for an audit trail and the internet is down.

That is why I build local-first.