In Part 1 of this series, I set up Claude Code against local LLMs on dual MI60 GPUs and watched it scaffold a Flask application from scratch. Small tasks worked. Complex ones did not. I ended with three ideas I wanted to test: running a dense model, trying Claude Code’s agent teams feature, and building a persistent memory layer for coding sessions.

I started the experiment that mattered most: giving a local LLM a project with real scope and seeing what happened. I ran the same project against two different models. The results were instructive, and not in the direction I expected.

I have been running local LLMs on dual AMD MI60 GPUs for over a year, and recently pointed Claude Code at them to see how close I could get to frontier-quality agentic coding on my own hardware. I wanted to know: how close can local models actually get to Anthropic’s Claude for building real software?

This is the first post in a series documenting that experiment. Future posts will cover specific agentic coding sessions, hardware and model setup, and the tools I am building to close the gap.

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

1. Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.

2. PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.

Every query to Claude Code means sending my source code to Anthropic’s servers. For proprietary codebases, that’s a non-starter. With vLLM and LiteLLM, I can point Claude Code at my own hardware - keeping my code on my network while maintaining the same workflow.

The Architecture#

The trick is that Claude Code expects the Anthropic Messages API, but local inference servers speak OpenAI’s API format. LiteLLM bridges this gap. It accepts Anthropic-formatted requests and translates them to OpenAI format for my local vLLM instance.