For most of the history of software, an application existed because enough people wanted it to pay for the time it took to build. A tool that only one person needed usually didn’t get written, unless that person happened to be a programmer with a lot of spare evenings. Everyone else adapted to whatever the market offered.

That has changed. With modern LLM coding tools, a person with a clear picture of the application they want can build it, even if they will be its only user. I think of it as custom software on demand: not a product with a market, but a tool made for one person’s way of working. I’ve been building software this way for most of this year, and I think it is one of the more important things these tools make possible.

A recent paper from Tsinghua and Renmin University caught my attention: Task-Decoupled Planning for Long-Horizon Agents. The core insight resonated with a problem I’ve been wrestling with on a Java code modernization project at work, and I think the pattern deserves more attention.

The Entangled Context Problem#

Most LLM agent architectures fall into two camps: step-wise planning (reactive, one step at a time) or one-shot planning (generate the whole plan upfront). Both share a fatal flaw: they maintain a monolithic reasoning context that spans the entire task history.

Production inference using vLLM with tensor parallelism across dual MI60 GPUs.

Why vLLM?#

After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, vLLM emerged as the best choice:

1. Tensor Parallelism: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn’t fit on a single GPU.

2. PagedAttention: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.

Two AMD MI60s from eBay cost me about $1,000 total and gave me 64GB of VRAM. That’s enough to run Llama 3.3 70B at home with a 32K context window.

When I started looking into running large language models locally, the obvious limiting factor was VRAM. Consumer GPUs top out at 24GB, and that’s on an RTX 4090 at the high end. I wanted to run 70B parameter models locally, on hardware I own.