GPU on Hullabalooing

ComfyUI on MI60

Wed, 21 Jan 2026 00:00:00 -0700

Node-based Stable Diffusion UI with AMD MI60 GPU acceleration via ROCm.

Overview#

ComfyUI runs as a containerized service using a custom ROCm 5.7 image built for the MI60’s gfx906 architecture. It can run standalone on both GPUs or alongside vLLM in the image-chat configuration.

Quick Start#

# Build the container image
cd comfyui
make build

# Start ComfyUI (standalone, both GPUs)
make up

# Or use the image-chat config (vLLM on GPU0, ComfyUI on GPU1)
curl -X POST -H "Content-Type: application/json" \
  -d '{"config":"image-chat"}' http://localhost:9100/switch

Access the UI at http://localhost:8188.

GPU Configuration Management

Wed, 21 Jan 2026 00:00:00 -0700

Dynamic switching between GPU configurations based on workload needs.

Overview#

Rather than running a single fixed configuration, I dynamically switch between GPU configurations. A Python service manages the transitions via HTTP API.

Available Configurations#

Config	Model	Use Case
`big-chat`	Llama 3.3 70B (TP=2)	Maximum quality chat
`coder`	Qwen3 32B (TP=2)	Coding tasks with thinking mode
`dual-chat`	Qwen3 8B + 14B	Two models, different tasks
`image-chat`	Qwen 7B + ComfyUI	Text + image generation

Configuration files are stored as compose YAML files.

GPU Metrics and Monitoring

Wed, 21 Jan 2026 00:00:00 -0700

Prometheus metrics and Grafana dashboards for GPU health and state monitoring.

Prometheus Metrics#

The gpu-state-service exposes Prometheus metrics on port 9100:

# GPU state and configuration
gpu_config_info{config="big-chat",model="llama-3.3-70b"} 1
gpu_state_info{state="ready"} 1
gpu_ready 1
gpu_config_switches_total 5
gpu_config_uptime_seconds 3600

# Temperature monitoring (from separate AMD GPU exporter on port 9101)
amd_gpu_temperature_junction_celsius{gpu="0"} 72
amd_gpu_temperature_junction_celsius{gpu="1"} 75
amd_gpu_utilization_percent{gpu="0"} 85
amd_gpu_power_watts{gpu="0"} 180

Prometheus Endpoints#

Port	Service	Metrics
9100	gpu-state-service	Config state, switches, uptime
9101	AMD GPU exporter	Temperature, utilization, power
9102	NVIDIA GPU exporter	RTX 2080 metrics (if present)

Temperature Alerts#

The service monitors junction temperatures and sends ntfy notifications:

MI60 Hardware Setup

Wed, 21 Jan 2026 00:00:00 -0700

How to get an AMD Instinct MI60 running for AI workloads, including cooling solutions, BIOS settings, and fan control.

Introduction#

The AMD Instinct MI60 is a powerful server GPU featuring 32 GB of HBM2 VRAM and PCIe 3.0 connectivity. It remains a good choice for budget-conscious AI developers looking for high VRAM capacity at a fraction of modern GPU prices. With proper setup, the MI60 can handle local LLM inference, Whisper transcription, Stable Diffusion, and other workloads.

vLLM Inference on MI60

Wed, 21 Jan 2026 00:00:00 -0700

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.
PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.

Running Claude Code with Local LLMs via vLLM and LiteLLM

Tue, 20 Jan 2026 00:00:00 +0000

Every query to Claude Code means sending my source code to Anthropic’s servers. For proprietary codebases, that’s a non-starter. With vLLM and LiteLLM, I can point Claude Code at my own hardware - keeping my code on my network while maintaining the same workflow.

The Architecture#

The trick is that Claude Code expects the Anthropic Messages API, but local inference servers speak OpenAI’s API format. LiteLLM bridges this gap. It accepts Anthropic-formatted requests and translates them to OpenAI format for my local vLLM instance.

An Affordable AI Server

Sat, 17 Jan 2026 00:00:00 -0700

Two AMD MI60s from eBay cost me about $1,000 total and gave me 64GB of VRAM. That’s enough to run Llama 3.3 70B at home with a 32K context window.

When I started looking into running large language models locally, the obvious limiting factor was VRAM. Consumer GPUs top out at 24GB, and that’s on an RTX 4090 at the high end. I wanted to run 70B parameter models locally, on hardware I own.