https://cruver.ai/tags/prometheus/ Recent content in Prometheus on Hullabalooing Hugo en Wed, 21 Jan 2026 00:00:00 -0700 https://cruver.ai/gpu-ai/posts/gpu-metrics-monitoring/ Wed, 21 Jan 2026 00:00:00 -0700 https://cruver.ai/gpu-ai/posts/gpu-metrics-monitoring/ <p>Prometheus metrics and Grafana dashboards for GPU health and state monitoring.</p> <h2 id="prometheus-metrics">Prometheus Metrics<a class="anchor" href="#prometheus-metrics">#</a></h2> <p>The <code>gpu-state-service</code> exposes Prometheus metrics on port 9100:</p> <div class="highlight"><pre tabindex="0" style="color:#e2e4e5;background-color:#282a36;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-text" data-lang="text"><span style="display:flex;"><span># GPU state and configuration </span></span><span style="display:flex;"><span>gpu_config_info{config=&#34;big-chat&#34;,model=&#34;llama-3.3-70b&#34;} 1 </span></span><span style="display:flex;"><span>gpu_state_info{state=&#34;ready&#34;} 1 </span></span><span style="display:flex;"><span>gpu_ready 1 </span></span><span style="display:flex;"><span>gpu_config_switches_total 5 </span></span><span style="display:flex;"><span>gpu_config_uptime_seconds 3600 </span></span><span style="display:flex;"><span> </span></span><span style="display:flex;"><span># Temperature monitoring (from separate AMD GPU exporter on port 9101) </span></span><span style="display:flex;"><span>amd_gpu_temperature_junction_celsius{gpu=&#34;0&#34;} 72 </span></span><span style="display:flex;"><span>amd_gpu_temperature_junction_celsius{gpu=&#34;1&#34;} 75 </span></span><span style="display:flex;"><span>amd_gpu_utilization_percent{gpu=&#34;0&#34;} 85 </span></span><span style="display:flex;"><span>amd_gpu_power_watts{gpu=&#34;0&#34;} 180</span></span></code></pre></div><h2 id="prometheus-endpoints">Prometheus Endpoints<a class="anchor" href="#prometheus-endpoints">#</a></h2> <table> <thead> <tr> <th>Port</th> <th>Service</th> <th>Metrics</th> </tr> </thead> <tbody> <tr> <td>9100</td> <td>gpu-state-service</td> <td>Config state, switches, uptime</td> </tr> <tr> <td>9101</td> <td>AMD GPU exporter</td> <td>Temperature, utilization, power</td> </tr> <tr> <td>9102</td> <td>NVIDIA GPU exporter</td> <td>RTX 2080 metrics (if present)</td> </tr> </tbody> </table> <h2 id="temperature-alerts">Temperature Alerts<a class="anchor" href="#temperature-alerts">#</a></h2> <p>The service monitors junction temperatures and sends ntfy notifications:</p>