https://cruver.ai/tags/inference/ Recent content in Inference on Hullabalooing Hugo en Wed, 21 Jan 2026 00:00:00 -0700 https://cruver.ai/gpu-ai/posts/vllm-inference-mi60/ Wed, 21 Jan 2026 00:00:00 -0700 https://cruver.ai/gpu-ai/posts/vllm-inference-mi60/ <p>Production inference using vLLM with tensor parallelism across dual MI60 GPUs.</p> <h2 id="why-vllm">Why vLLM?<a class="anchor" href="#why-vllm">#</a></h2> <p>After evaluating Ollama, llama.cpp, and vLLM for MI60 inference, <strong>vLLM</strong> emerged as the best choice:</p> <ol> <li> <p><strong>Tensor Parallelism</strong>: Native support for splitting large models across multiple GPUs. With dual MI60s (64GB total), I can run 70B parameter models that wouldn&rsquo;t fit on a single GPU.</p> </li> <li> <p><strong>PagedAttention</strong>: More efficient memory management, allowing higher GPU utilization (90%) without OOM errors.</p>