Prometheus metrics and Grafana dashboards for GPU health and state monitoring.
Prometheus Metrics#
The gpu-state-service exposes Prometheus metrics on port 9100:
# GPU state and configuration
gpu_config_info{config="big-chat",model="llama-3.3-70b"} 1
gpu_state_info{state="ready"} 1
gpu_ready 1
gpu_config_switches_total 5
gpu_config_uptime_seconds 3600
# Temperature monitoring (from separate AMD GPU exporter on port 9101)
amd_gpu_temperature_junction_celsius{gpu="0"} 72
amd_gpu_temperature_junction_celsius{gpu="1"} 75
amd_gpu_utilization_percent{gpu="0"} 85
amd_gpu_power_watts{gpu="0"} 180Prometheus Endpoints#
| Port | Service | Metrics |
|---|---|---|
| 9100 | gpu-state-service | Config state, switches, uptime |
| 9101 | AMD GPU exporter | Temperature, utilization, power |
| 9102 | NVIDIA GPU exporter | RTX 2080 metrics (if present) |
Temperature Alerts#
The service monitors junction temperatures and sends ntfy notifications: