> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlit.io/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Monitoring for LLM Inference

> OpenTelemetry-native GPU monitoring for LLM inference on NVIDIA, AMD, and Intel GPUs — utilization, memory, power, and temperature.

The **OpenTelemetry GPU Collector** is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) - with no Python dependencies, no DCGM daemon, and no vendor-specific agents.

It is fully configured via standard OpenTelemetry environment variables and follows the [OTel semantic conventions for hardware metrics](https://opentelemetry.io/docs/specs/semconv/hardware/gpu/). Use [collector modes](/latest/gpu-collector/modes) (`OTEL_GPU_COLLECTOR_MODE`, default `all`) to turn optional features on by use case.

Resource identity models this process as a **GPU metrics agent** (host + accelerator telemetry), not an application service: `telemetry.sdk.*`, `telemetry.distro.name=opentelemetry-gpu-collector`, `telemetry.distro.version`, plus host/k8s/cloud attributes. Set `OTEL_SERVICE_NAME` only if your backend requires `service.name`.

## Goals

* **OpenTelemetry-native** - uses standard `OTEL_*` env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend
* **Cross-vendor GPU support** - NVIDIA (NVML), AMD, and Intel on Linux and Windows from a single binary
* **OTel semantic conventions** - `hw.gpu.*` metric names, `hw.id` / `hw.name` / `hw.vendor` attributes per spec
* **Zero dependencies** - no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics
* **Resilient** - continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s

## What it collects

<CardGroup cols={3}>
  <Card title="GPU Hardware Telemetry" icon="microchip">
    Utilization, memory, temperature, power, energy, clocks, PCIe/interconnect throughput, throttle, ECC/XID/RAS — NVIDIA, AMD, and Intel on Linux and Windows
  </Card>

  <Card title="Process + workload attribution" icon="user">
    Per-process VRAM/util plus cmdline, OS state (zombie), owner, uptime, and LLM framework labels (`vllm`, `ollama`, …). Kubernetes pod name/namespace via PodResources
  </Card>

  <Card title="eBPF CUDA Tracing" icon="code">
    Kernel launch counts, grid/block sizes, graph replays, memory allocations, and memory copies via uprobes on `libcudart.so` (runtime) and `libcuda.so` (driver) — covers llama.cpp/Ollama as well as PyTorch (on by default on Linux; discovers libs from `/proc` with host PID — no CUDA mount)
  </Card>
</CardGroup>

## GPU vendor support

| Vendor | Linux | Windows |
| - | - | - |
| **NVIDIA** | NVML (+ MIG, NVLink, XID events) | NVML (`nvml.dll`, NVLink) + PDH process attribution (MIG not available on Windows) |
| **AMD** | sysfs/hwmon/DRM + `libamd_smi` (PCIe/XGMI/throttle/RAS when present) | DXGI + PDH (incl. encode/decode) + ADL (util/temp/clocks/fan) |
| **Intel** | DRM/hwmon + Level Zero Sysman (util/VRAM/power/media) | DXGI + PDH (incl. encode/decode) + Level Zero Sysman |

## Platform support

| Feature | Linux | macOS | Windows |
| - | :-: | :-: | :-: |
| System metrics (CPU, memory, disk, network) | Yes | Yes | Yes |
| Process metrics (CPU, memory, threads, FDs) | Yes | Yes | Yes |
| GPU device + process metrics | Yes | - | Yes |
| eBPF CUDA tracing / occupancy | Yes | - | - |

## How it works

```
Host Metrics (all platforms via gopsutil)
    +-- CPU utilization, memory, disk I/O, filesystem, network
    +-- Process: self CPU, memory, threads, FDs, Go runtime

GPU Metrics (Linux + Windows)
    +-- Linux: PCI scan → NVML / sysfs / DRM (+ eBPF CUDA, default on)
    +-- Windows: NVML.dll / DXGI + PDH (no eBPF)

Export
    +-- OTel SDK --> OTLP gRPC/HTTP --> your OTel collector / backend
```

<Steps>
  <Step title="Discovers GPUs">
    On Linux, the collector scans `/sys/bus/pci/devices/` for GPU class codes. On Windows, it loads `nvml.dll` when present and enumerates adapters via DXGI.
  </Step>

  <Step title="Initialises vendor backend">
    Each detected GPU is handed to its vendor-specific backend. NVIDIA uses NVML. Linux AMD/Intel read sysfs/hwmon; Windows AMD/Intel use DXGI memory totals and PDH for utilization and per-process attribution.
  </Step>

  <Step title="Registers OTel instruments">
    Observable gauge and counter instruments are registered with the OTel SDK meter. On each collection tick, the SDK calls back into the collector to read fresh values from each GPU.
  </Step>

  <Step title="Exports via OTLP">
    Metrics are exported via OTLP to any compatible backend - OpenLIT, Grafana, Datadog, New Relic, or a standard OTel Collector.
  </Step>
</Steps>

***

<CardGroup cols={2}>
  <Card title="Quickstart" href="/latest/gpu-collector/quickstart" icon="bolt">
    Get the collector running in under 5 minutes with Docker
  </Card>

  <Card title="Configuration" href="/latest/gpu-collector/configuration" icon="sliders">
    Full reference for all environment variables
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.