Node-based Stable Diffusion UI with AMD MI60 GPU acceleration via ROCm.

Overview#

ComfyUI runs as a containerized service using a custom ROCm 5.7 image built for the MI60’s gfx906 architecture. It can run standalone on both GPUs or alongside vLLM in the image-chat configuration.

Quick Start#

# Build the container image
cd comfyui
make build

# Start ComfyUI (standalone, both GPUs)
make up

# Or use the image-chat config (vLLM on GPU0, ComfyUI on GPU1)
curl -X POST -H "Content-Type: application/json" \
  -d '{"config":"image-chat"}' http://localhost:9100/switch

Access the UI at http://localhost:8188.

Dynamic switching between GPU configurations based on workload needs.

Overview#

Rather than running a single fixed configuration, I dynamically switch between GPU configurations. A Python service manages the transitions via HTTP API.

Available Configurations#

Configuration files are stored as compose YAML files.

Prometheus metrics and Grafana dashboards for GPU health and state monitoring.

Prometheus Metrics#

The gpu-state-service exposes Prometheus metrics on port 9100:

# GPU state and configuration
gpu_config_info{config="big-chat",model="llama-3.3-70b"} 1
gpu_state_info{state="ready"} 1
gpu_ready 1
gpu_config_switches_total 5
gpu_config_uptime_seconds 3600

# Temperature monitoring (from separate AMD GPU exporter on port 9101)
amd_gpu_temperature_junction_celsius{gpu="0"} 72
amd_gpu_temperature_junction_celsius{gpu="1"} 75
amd_gpu_utilization_percent{gpu="0"} 85
amd_gpu_power_watts{gpu="0"} 180

Prometheus Endpoints#

Temperature Alerts#

The service monitors junction temperatures and sends ntfy notifications:

How to get an AMD Instinct MI60 running for AI workloads, including cooling solutions, BIOS settings, and fan control.

Introduction#

The AMD Instinct MI60 is a powerful server GPU featuring 32 GB of HBM2 VRAM and PCIe 3.0 connectivity. It remains a good choice for budget-conscious AI developers looking for high VRAM capacity at a fraction of modern GPU prices. With proper setup, the MI60 can handle local LLM inference, Whisper transcription, Stable Diffusion, and other workloads.

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

1. Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.

2. PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.

Every query to Claude Code means sending my source code to Anthropic’s servers. For proprietary codebases, that’s a non-starter. With vLLM and LiteLLM, I can point Claude Code at my own hardware - keeping my code on my network while maintaining the same workflow.

The Architecture#

The trick is that Claude Code expects the Anthropic Messages API, but local inference servers speak OpenAI’s API format. LiteLLM bridges this gap. It accepts Anthropic-formatted requests and translates them to OpenAI format for my local vLLM instance.

Two AMD MI60s from eBay cost me about $1,000 total and gave me 64GB of VRAM. That’s enough to run Llama 3.3 70B at home with a 32K context window.

When I started looking into running large language models locally, the obvious limiting factor was VRAM. Consumer GPUs top out at 24GB, and that’s on an RTX 4090 at the high end. I wanted to run 70B parameter models locally, on hardware I own.