Running local AI inference on AMD Instinct MI60 GPUs - a budget-friendly path to 70B parameter models at home.

Overview#

The AMD Instinct MI60 is a datacenter GPU from 2018 that remains surprisingly capable for local AI workloads. With 32GB of HBM2 memory per card, a dual-MI60 setup provides 64GB of VRAM, enough to run 70B parameter models with quantization.

This section documents my setup: hardware modifications for cooling, software configuration for ROCm and vLLM, and the workflows that make it practical for daily use.

Node-based Stable Diffusion UI with AMD MI60 GPU acceleration via ROCm.

Overview#

ComfyUI runs as a containerized service using a custom ROCm 5.7 image built for the MI60’s gfx906 architecture. It can run standalone on both GPUs or alongside vLLM in the image-chat configuration.

Quick Start#

# Build the container image
cd comfyui
make build

# Start ComfyUI (standalone, both GPUs)
make up

# Or use the image-chat config (vLLM on GPU0, ComfyUI on GPU1)
curl -X POST -H "Content-Type: application/json" \
  -d '{"config":"image-chat"}' http://localhost:9100/switch

Access the UI at http://localhost:8188.

Dynamic switching between GPU configurations based on workload needs.

Overview#

Rather than running a single fixed configuration, I dynamically switch between GPU configurations. A Python service manages the transitions via HTTP API.

Available Configurations#

Configuration files are stored as compose YAML files.

Prometheus metrics and Grafana dashboards for GPU health and state monitoring.

Prometheus Metrics#

The gpu-state-service exposes Prometheus metrics on port 9100:

# GPU state and configuration
gpu_config_info{config="big-chat",model="llama-3.3-70b"} 1
gpu_state_info{state="ready"} 1
gpu_ready 1
gpu_config_switches_total 5
gpu_config_uptime_seconds 3600

# Temperature monitoring (from separate AMD GPU exporter on port 9101)
amd_gpu_temperature_junction_celsius{gpu="0"} 72
amd_gpu_temperature_junction_celsius{gpu="1"} 75
amd_gpu_utilization_percent{gpu="0"} 85
amd_gpu_power_watts{gpu="0"} 180

Prometheus Endpoints#

Temperature Alerts#

The service monitors junction temperatures and sends ntfy notifications:

How to get an AMD Instinct MI60 running for AI workloads, including cooling solutions, BIOS settings, and fan control.

Introduction#

The AMD Instinct MI60 is a powerful server GPU featuring 32 GB of HBM2 VRAM and PCIe 3.0 connectivity. It remains a good choice for budget-conscious AI developers looking for high VRAM capacity at a fraction of modern GPU prices. With proper setup, the MI60 can handle local LLM inference, Whisper transcription, Stable Diffusion, and other workloads.

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

1. Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.

2. PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.