Vllm on Hullabalooing

I Pushed Local LLMs Harder. Here's What Two Models Actually Did.

Mon, 02 Mar 2026 00:00:00 -0700

In Part 1 of this series, I set up Claude Code against local LLMs on dual MI60 GPUs and watched it scaffold a Flask application from scratch. Small tasks worked. Complex ones did not. I ended with three ideas I wanted to test: running a dense model, trying Claude Code’s agent teams feature, and building a persistent memory layer for coding sessions.

I started the experiment that mattered most: giving a local LLM a project with real scope and seeing what happened. I ran the same project against two different models. The results were instructive, and not in the direction I expected.

I Ran Claude Code on Local LLMs for a Month. Here's What Worked for Me.

Mon, 16 Feb 2026 00:00:00 +0000

I have been running local LLMs on dual AMD MI60 GPUs for over a year, and recently pointed Claude Code at them to see how close I could get to frontier-quality agentic coding on my own hardware. I wanted to know: how close can local models actually get to Anthropic’s Claude for building real software?

This is the first post in a series documenting that experiment. Future posts will cover specific agentic coding sessions, hardware and model setup, and the tools I am building to close the gap.

vLLM Inference on MI60

Wed, 21 Jan 2026 00:00:00 -0700

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.
PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.

Running Claude Code with Local LLMs via vLLM and LiteLLM

Tue, 20 Jan 2026 00:00:00 +0000

Every query to Claude Code means sending my source code to Anthropic’s servers. For proprietary codebases, that’s a non-starter. With vLLM and LiteLLM, I can point Claude Code at my own hardware - keeping my code on my network while maintaining the same workflow.

The Architecture#

The trick is that Claude Code expects the Anthropic Messages API, but local inference servers speak OpenAI’s API format. LiteLLM bridges this gap. It accepts Anthropic-formatted requests and translates them to OpenAI format for my local vLLM instance.