Installing vLLM onto EVO-X2.

Abstract.

I download a vLLM Docker image that supports AMD, and the ROCm GPU driver that runs on the Strix Halo platform (AI Max+ 395 APU). I connect the running Docker container to local GGUF AI models for a fully self-contained, no-Python-wrapper LLM service. Every step -- pulling the ROCm Docker image, configuring iGPU passthrough, installing AMD SMI inside the container, mapping local model directories, and starting the server with tool-calling -- is performed directly via Docker CLI and the vLLM serve command.

** Attributions: **

***TinyComputers.io ↗, vLLM Documentation v0.7.3 ↗. ***

An Introduction.

vLLM provides high-throughput LLM serving with PagedAttention, continuous batching, and an OpenAI-compatible API. On AMD ROCm hardware -- specifically the Strix Halo integrated iGPU (gfx1151) found in the AI Max+ 395 APU -- running vLLM inside Docker using the official ROCm image enables local inference with full tool-calling support. No Python wrappers are required; I use Docker CLI and the vLLM serve command directly.

My target platform is the AMD Strix Halo system (Ryzen AI MAX+ 395 APU) with 128 GB of unified memory. Unlike discrete GPUs where VRAM is the hard limit, the unified memory architecture lets the CPU and iGPU share the same memory pool, allowing larger models to run on a single chip.

The Big Picture.

There are several options for running LLMs locally: llama.cpp, Ollama, and vLLM being the most popular. I choose vLLM for a specific reason: tool-calling support. vLLM implements the OpenAI-compatible API with proper function calling, which means coding assistants can use tools like file reading, code execution, and search. This is critical for getting a capable coding assistant rather than just a chat interface.

vLLM also offers excellent performance through continuous batching, PagedAttention for efficient memory management, and support for a wide range of models. The trade-off is that it is more resource-intensive than llama.cpp, but when I have the VRAM, the capabilities are worth it.

NOTE: I bypass Python wrappers entirely in this workflow. I deal directly with Docker and the vLLM CLI -- no pip installs, no Python SDKs, no library management on the host side.

Prerequisites.

My setup uses:

• An AMD iGPU supported by ROCm (Strix Halo gfx1151).

• ROCm drivers and llama.cpp.

• Docker is installed and functional.

• I have a Docker virtual environment (VENV) activated (e.g., source ~/env/docker/bin/activate).

• My system has at least 16 GB of system RAM (more for larger models).

• I have downloaded GGUF AI models stored under /home/brian/Downloads/Models.

Verifying ROCm Installation.

I confirm my ROCm installation sees the iGPU:

rocminfo | grep gfx1151

Expected output:

Name:                    gfx1151
Uuid:                    GPU-XX
Device Type:             GPU

Pulling the ROCm vLLM Docker Image.

The ROCm team maintains Docker images that bundle the full ROCm stack, PyTorch, and vLLM with all dependencies. I pull from Docker Hub:

docker pull rocm/vllm:latest

NOTE: The image is several GB in size as it includes the full ROCm stack. The rocm/vllm:latest tag points to the latest stable build. Alternatively, rocm/vllm-dev:nightly provides a nightly development build.

Starting the Container with iGPU Passthrough.

The container must be started with explicit iGPU device access and shared memory configuration. vLLM uses PyTorch, which relies on shared memory for inter-process communication during tensor-parallel inference.

I start the container with iGPU access and port forwarding:

docker run -d \
  --name vllm-dev \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  rocm/vllm-dev:nightly \
  tail -f /dev/null

The important flags:

WARNING: I must use either --ipc=host or --shm-size flags when running vLLM in Docker directly. The example above uses tail -f /dev/null to keep the container alive; when running vLLM directly, I add --ipc=host to the Docker run command.

docker run -d \
  --name vllm-dev \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /home/brian/Downloads/Models:/models \
  rocm/vllm-dev:nightly \
  tail -f /dev/null

Installing AMD SMI (Critical Step).

Before starting the vLLM server, the AMD SMI (System Management Interface) Python package must be installed inside the container. This is required for vLLM to detect the ROCm platform correctly. Without this step, vLLM fails with an "UnspecifiedPlatform" error.

docker exec -it vllm-dev bash -c 'pip install /opt/rocm/share/amd_smi'

Verification:

docker exec vllm-dev python3 -c "import amdsmi; print('AMD SMI installed successfully')"

NOTE: The AMD SMI package is bundled inside the ROCm Docker image at /opt/rocm/share/amd_smi. The pip install command installs it from the local source path. No external download is needed.

Starting the vLLM Server.

Once the container is running and AMD SMI is installed, I start the vLLM server with tool-calling support. The vllm serve command launches the OpenAI-compatible HTTP server.

Standard (HuggingFace Model).

This command downloads the model from HuggingFace Hub on first run:

docker exec -d vllm-dev bash -c 'vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --host 0.0.0.0 \
  --port 8000 > /tmp/vllm.log 2>&1'

docker exec -d vllm-dev bash -c 'vllm serve /models/11-hermes.gguf \
  --max-model-len 32768 \
  --load-format gguf \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --host 0.0.0.0 \
  --port 8000 > /tmp/vllm.log 2>&1'

Argument breakdown:

Monitoring Server Startup.

The first startup takes time as vLLM downloads model weights, compiles ROCm graphs, and warms up. I monitor progress:

docker exec vllm-dev tail -f /tmp/vllm.log

Expected log sequence:

1. Loading model shards / GGUF file.

2. Compiling ROCm graphs (capturing iGPU kernels).

3. "Uvicorn running on http://0.0.0.0:8000" -- server ready.

Server Arguments Reference.

Key engine arguments available to vllm serve (from the vLLM docs):

Verifying the Server.

Once the server reports "Uvicorn running", I test the API endpoint from the host:

curl http://localhost:8000/v1/models

Expected JSON response showing the loaded model with its maximum context length.

Client Usage (OpenAI Python Client).

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",  # vLLM doesn't require an API key by default
)

completion = client.chat.completions.create(
    model="/models/11-hermes.gguf",
    messages=[
        {"role": "user", "content": "Write a Python function to sort a list"}
    ],
)

print(completion.choices[0].message.content)

Client Usage (curl).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/models/11-hermes.gguf",
    "messages": [
      {"role": "user", "content": "Hello, explain vLLM briefly"}
    ]
  }'

Local Models Reference.

The following GGUF models are available at /home/brian/Downloads/Models:

NOTE: The --load-format gguf flag must be used when loading GGUF files. vLLM's --load-format accepts values including auto, pt, safetensors, gguf, bitsandbytes, and others.

Troubleshooting.

"UnspecifiedPlatform" Error.

When vLLM fails with an UnspecifiedPlatform error, I install AMD SMI:

docker exec -it vllm-dev bash -c 'pip install /opt/rocm/share/amd_smi'

Container Lacks iGPU Access.

I verify the container sees the iGPU:

docker exec vllm-dev rocminfo | grep gfx

If no iGPU is visible, I check that --device=/dev/kfd and --device=/dev/dri are included in docker run.

Model Not Found.

When vLLM reports "model not found" for a local GGUF file, I verify the mount path and file permissions:

docker exec vllm-dev ls -la /models/
docker exec vllm-dev ls -la /models/11-hermes.gguf

I make sure the mount path in docker run -v matches the path used in vllm serve.

Shared Memory Issues.

When vLLM crashes with shared memory errors, I add --ipc=host or --shm-size=16g to the docker run command.

The Results.

This workflow produces a fully functional OpenAI-compatible LLM serving endpoint running inside a Docker container on AMD ROCm hardware. The server supports:

• Full OpenAI Chat Completions and Models API.

• Automatic function calling / tool use via --enable-auto-tool-choice and --tool-call-parser hermes.

• Long context windows up to 32,768 tokens (configurable via --max-model-len).

• Local GGUF model loading from a mounted host directory -- no HuggingFace downloads required.

• No Python wrappers on the host side; all operations use Docker CLI and vllm serve directly.

In Conclusion.

I use Docker CLI commands and the vLLM service directly to (1/5) run vLLM on (2/5) AMD hardware using (3/5) the ROCm driver via (4/5) Docker, (5/5) without needing Python wrappers. The critical steps I take are: pulling the ROCm image, starting the container with iGPU passthrough, installing AMD SMI inside the container, mounting local model directories, and launching vllm serve with tool-calling flags. The result is a production-grade OpenAI-compatible LLM service endpoint that supports function calling, long contexts, and local GGUF models on AMD Strix Halo hardware.

Until next time: Be safe, be kind, be awesome, kia kaha!!

Document Details.

The following information is the metadata for this post.

Hash Tags.

#vLLM #ROCm #AMD #Docker #LocalAI #LLM-Inference #Tool-Calling #Strix-Halo #GGUF #OpenAI-Compatible

SEO Title (60 Characters).

Installing vLLM for AMD ROCm via Docker on Strix Halo

SEO Description (150 Characters).

Step-by-step guide to installing vLLM for AMD ROCm on Strix Halo hardware using Docker. Covers ROCm image pull, iGPU passthrough, AMD SMI installation, and server startup with tool-calling.

Permalink

https://solodev.app/installing-vllm-onto-evo-x2

Version

v0.5.2