Abstract.
I download a vLLM Docker image that supports AMD, and the ROCm GPU driver that runs on the Strix Halo platform (AI Max+ 395 APU). I connect the running Docker container to local GGUF AI models for a fully self-contained, no-Python-wrapper LLM service. Every step -- pulling the ROCm Docker image, configuring iGPU passthrough, installing AMD SMI inside the container, mapping local model directories, and starting the server with tool-calling -- is performed directly via Docker CLI and the vLLM serve command.
** Attributions: **
***TinyComputers.io ↗, vLLM Documentation v0.7.3 ↗. ***
An Introduction.
vLLM provides high-throughput LLM serving with PagedAttention, continuous batching, and an OpenAI-compatible API. On AMD ROCm hardware -- specifically the Strix Halo integrated iGPU (gfx1151) found in the AI Max+ 395 APU -- running vLLM inside Docker using the official ROCm image enables local inference with full tool-calling support. No Python wrappers are required; I use Docker CLI and the vLLM serve command directly.
My target platform is the AMD Strix Halo system (Ryzen AI MAX+ 395 APU) with 128 GB of unified memory. Unlike discrete GPUs where VRAM is the hard limit, the unified memory architecture lets the CPU and iGPU share the same memory pool, allowing larger models to run on a single chip.
The Big Picture.
There are several options for running LLMs locally: llama.cpp, Ollama, and vLLM being the most popular. I choose vLLM for a specific reason: tool-calling support. vLLM implements the OpenAI-compatible API with proper function calling, which means coding assistants can use tools like file reading, code execution, and search. This is critical for getting a capable coding assistant rather than just a chat interface.
vLLM also offers excellent performance through continuous batching, PagedAttention for efficient memory management, and support for a wide range of models. The trade-off is that it is more resource-intensive than llama.cpp, but when I have the VRAM, the capabilities are worth it.
NOTE: I bypass Python wrappers entirely in this workflow. I deal directly with Docker and the vLLM CLI -- no pip installs, no Python SDKs, no library management on the host side.
Prerequisites.
My setup uses:
An AMD iGPU supported by ROCm (Strix Halo gfx1151).
ROCm drivers and llama.cpp.
Docker is installed and functional.
I have a Docker virtual environment (VENV) activated (e.g., source ~/env/docker/bin/activate).
My system has at least 16 GB of system RAM (more for larger models).
I have downloaded GGUF AI models stored under /home/brian/Downloads/Models.
Verifying ROCm Installation.
I confirm my ROCm installation sees the iGPU:
rocminfo | grep gfx1151
Expected output:
Name: gfx1151
Uuid: GPU-XX
Device Type: GPU
Pulling the ROCm vLLM Docker Image.
The ROCm team maintains Docker images that bundle the full ROCm stack, PyTorch, and vLLM with all dependencies. I pull from Docker Hub:
docker pull rocm/vllm:latest
NOTE: The image is several GB in size as it includes the full ROCm stack. The rocm/vllm:latest tag points to the latest stable build. Alternatively, rocm/vllm-dev:nightly provides a nightly development build.
Starting the Container with iGPU Passthrough.
The container must be started with explicit iGPU device access and shared memory configuration. vLLM uses PyTorch, which relies on shared memory for inter-process communication during tensor-parallel inference.
I start the container with iGPU access and port forwarding:
docker run -d \
--name vllm-dev \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
rocm/vllm-dev:nightly \
tail -f /dev/null
The important flags:
| Flag |
Purpose |
--device=/dev/kfd |
Kernel Fusion Driver -- gives the container access to the AMD GPU compute interface |
--device=/dev/dri |
Direct Rendering Interface -- grants GPU rendering access |
--group-add video |
Adds the video group for GPU permission requirements |
--cap-add=SYS_PTRACE |
Required for ptrace system calls used by ROCm |
--security-opt seccomp=unconfined |
Disables seccomp profile restrictions for ROCm |
-p 8000:8000 |
Exposes the vLLM API port on the host |
-v ~/.cache/huggingface:/root/.cache/huggingface |
Persists downloaded HuggingFace models between container restarts |
tail -f /dev/null |
Keeps the container alive for subsequent exec commands |
WARNING: I must use either --ipc=host or --shm-size flags when running vLLM in Docker directly. The example above uses tail -f /dev/null to keep the container alive; when running vLLM directly, I add --ipc=host to the Docker run command.
My Personal Settings
docker run -d \
--name vllm-dev \
--device=/dev/kfd \
--device=/dev/dri \
--group-add video \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v /home/brian/Downloads/Models:/models \
rocm/vllm-dev:nightly \
tail -f /dev/null
The additional -v /home/brian/Downloads/Models:/models mount maps my local GGUF model directory into the container at /models. This lets vLLM load models from a local path rather than downloading from HuggingFace Hub.
Installing AMD SMI (Critical Step).
Before starting the vLLM server, the AMD SMI (System Management Interface) Python package must be installed inside the container. This is required for vLLM to detect the ROCm platform correctly. Without this step, vLLM fails with an "UnspecifiedPlatform" error.
docker exec -it vllm-dev bash -c 'pip install /opt/rocm/share/amd_smi'
Verification:
docker exec vllm-dev python3 -c "import amdsmi; print('AMD SMI installed successfully')"
NOTE: The AMD SMI package is bundled inside the ROCm Docker image at /opt/rocm/share/amd_smi. The pip install command installs it from the local source path. No external download is needed.
Starting the vLLM Server.
Once the container is running and AMD SMI is installed, I start the vLLM server with tool-calling support. The vllm serve command launches the OpenAI-compatible HTTP server.
Standard (HuggingFace Model).
This command downloads the model from HuggingFace Hub on first run:
docker exec -d vllm-dev bash -c 'vllm serve Qwen/Qwen2.5-7B-Instruct \
--max-model-len 32768 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--host 0.0.0.0 \
--port 8000 > /tmp/vllm.log 2>&1'
My Personal Settings
docker exec -d vllm-dev bash -c 'vllm serve /models/11-hermes.gguf \
--max-model-len 32768 \
--load-format gguf \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--host 0.0.0.0 \
--port 8000 > /tmp/vllm.log 2>&1'
Argument breakdown:
| Argument |
Purpose |
vllm serve <model> |
Start the OpenAI-compatible HTTP server with the specified model |
--max-model-len 32768 |
Maximum context length in tokens. Coding assistants need long contexts for system prompts and code |
--load-format gguf |
Use the GGUF model loader for quantized GGUF files |
--enable-auto-tool-choice |
Enable automatic function-calling / tool-use support |
--tool-call-parser hermes |
Use the Hermes format for tool calls (compatible with Qwen, Hermes, and Nous models) |
--host 0.0.0.0 |
Bind to all interfaces so the server is reachable from the host |
--port 8000 |
Listen on port 8000 (must match the Docker publish port) |
Monitoring Server Startup.
The first startup takes time as vLLM downloads model weights, compiles ROCm graphs, and warms up. I monitor progress:
docker exec vllm-dev tail -f /tmp/vllm.log
Expected log sequence:
Loading model shards / GGUF file.
Compiling ROCm graphs (capturing iGPU kernels).
"Uvicorn running on http://0.0.0.0:8000" -- server ready.
Server Arguments Reference.
Key engine arguments available to vllm serve (from the vLLM docs):
| Argument |
Description |
--model |
Name or path of the HuggingFace model to use. Accepts local paths |
--task |
Task type: auto, generate, embedding, embed, classify, score, reward, transcription |
--tokenizer |
Name or path of the tokenizer |
--revision |
Model revision / commit hash |
--download-dir |
Directory to download models to (overrides HF_HUB cache) |
--load-format |
Format: auto, pt, safetensors, npcache, dummy, gguf, bitsandbytes, etc. |
--dtype |
Data type: auto, half, float16, bfloat16, float, float32 |
--kv-cache-dtype |
KV cache dtype: auto, fp8, fp8_e5m2, fp8_e4m3 |
--max-model-len |
Maximum sequence length the model can handle |
--quantization |
Quantization method: awq, fp8, gguf, gptq, bitsandbytes, etc. |
--tensor-parallel-size |
Number of GPU workers for tensor parallelism |
--pipeline-parallel-size |
Number of workers for pipeline parallelism |
--host |
Bind address (default: localhost) |
--port |
Listen port (default: 8000) |
--enable-auto-tool-choice |
Enable automatic tool choice for function calling |
--tool-call-parser |
Tool call parser format (e.g., hermes) |
--api-key |
API key for server authentication |
Verifying the Server.
Once the server reports "Uvicorn running", I test the API endpoint from the host:
curl http://localhost:8000/v1/models
Expected JSON response showing the loaded model with its maximum context length.
Client Usage (OpenAI Python Client).
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed", # vLLM doesn't require an API key by default
)
completion = client.chat.completions.create(
model="/models/11-hermes.gguf",
messages=[
{"role": "user", "content": "Write a Python function to sort a list"}
],
)
print(completion.choices[0].message.content)
Client Usage (curl).
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/models/11-hermes.gguf",
"messages": [
{"role": "user", "content": "Hello, explain vLLM briefly"}
]
}'
Local Models Reference.
The following GGUF models are available at /home/brian/Downloads/Models:
| File Name |
Size |
Description |
01-qwen2.5.gguf |
1.26 GB |
Qwen 2.5 small model |
02-gemma-4-31b.gguf |
21.9 GB |
Gemma 4 31B parameter model |
03-glm4.7.gguf |
21.7 GB |
GLM-4 7B model |
04-diffgemma.gguf |
19.1 GB |
Diffusion Gemma model |
05-gpt.gguf |
11.7 GB |
GPT model |
06-gemma-4-12b.gguf |
8.6 GB |
Gemma 4 12B parameter model |
07-qwen3.6.gguf |
20.4 GB |
Qwen 3 6B model |
08-qwen2.5-fp16.gguf |
3.56 GB |
Qwen 2.5 FP16 precision |
09-qwen3-coder.gguf |
36.0 GB |
Qwen 3 Coder model |
10-llama-3.3.gguf |
49.9 GB |
Llama 3.3 70B model |
11-hermes.gguf |
38.4 GB |
Hermes model with tool-calling support |
12-deepseek-r1.gguf |
50.0 GB |
DeepSeek R1 reasoning model |
13-paddle.gguf |
341 MB |
Paddle OCR / vision model |
14-orinth.gguf |
36.9 GB |
Orinth model |
NOTE: The --load-format gguf flag must be used when loading GGUF files. vLLM's --load-format accepts values including auto, pt, safetensors, gguf, bitsandbytes, and others.
Troubleshooting.
When vLLM fails with an UnspecifiedPlatform error, I install AMD SMI:
docker exec -it vllm-dev bash -c 'pip install /opt/rocm/share/amd_smi'
Container Lacks iGPU Access.
I verify the container sees the iGPU:
docker exec vllm-dev rocminfo | grep gfx
If no iGPU is visible, I check that --device=/dev/kfd and --device=/dev/dri are included in docker run.
Model Not Found.
When vLLM reports "model not found" for a local GGUF file, I verify the mount path and file permissions:
docker exec vllm-dev ls -la /models/
docker exec vllm-dev ls -la /models/11-hermes.gguf
I make sure the mount path in docker run -v matches the path used in vllm serve.
Shared Memory Issues.
When vLLM crashes with shared memory errors, I add --ipc=host or --shm-size=16g to the docker run command.
The Results.
This workflow produces a fully functional OpenAI-compatible LLM serving endpoint running inside a Docker container on AMD ROCm hardware. The server supports:
Full OpenAI Chat Completions and Models API.
Automatic function calling / tool use via --enable-auto-tool-choice and --tool-call-parser hermes.
Long context windows up to 32,768 tokens (configurable via --max-model-len).
Local GGUF model loading from a mounted host directory -- no HuggingFace downloads required.
No Python wrappers on the host side; all operations use Docker CLI and vllm serve directly.
In Conclusion.
I use Docker CLI commands and the vLLM service directly to (1/5) run vLLM on (2/5) AMD hardware using (3/5) the ROCm driver via (4/5) Docker, (5/5) without needing Python wrappers. The critical steps I take are: pulling the ROCm image, starting the container with iGPU passthrough, installing AMD SMI inside the container, mounting local model directories, and launching vllm serve with tool-calling flags. The result is a production-grade OpenAI-compatible LLM service endpoint that supports function calling, long contexts, and local GGUF models on AMD Strix Halo hardware.
Until next time: Be safe, be kind, be awesome, kia kaha!!
Document Details.
The following information is the metadata for this post.
#vLLM #ROCm #AMD #Docker #LocalAI #LLM-Inference #Tool-Calling #Strix-Halo #GGUF #OpenAI-Compatible
SEO Title (60 Characters).
Installing vLLM for AMD ROCm via Docker on Strix Halo
SEO Description (150 Characters).
Step-by-step guide to installing vLLM for AMD ROCm on Strix Halo hardware using Docker. Covers ROCm image pull, iGPU passthrough, AMD SMI installation, and server startup with tool-calling.
Permalink
https://solodev.app/installing-vllm-onto-evo-x2
Version
v0.5.2
