GPU Performance Monitoring

Simple GPU monitoring setup for AI workloads. Track NVIDIA and AMD GPU usage, temperature, and costs with zero code changes using OpenTelemetry.

Get started

OpenLIT SDK

Collect and send GPU performance metrics directly from your application to an OpenTelemetry endpoint.

OpenTelemetry GPU Collector

Install the OpenTelemetry GPU Collector as a Docker container to collect and send GPU performance metrics to an OpenTelemetry endpoint.

Using the SDK

1. Deploy OpenLIT
2. Install OpenLIT

   Open your command line or terminal and run:

   pip install openlit
   
3. Initialize OpenLIT in your Application

   Not sure which method to choose? Check out Instrumentation Methods to understand the differences.

   Zero-code instrumentation

   Via CLI arguments

   # Start GPU monitoring instantly
   openlit-instrument --collect-system-metrics python your_app.py
   
   # With custom settings
   openlit-instrument \
     --otlp-endpoint http://127.0.0.1:4318 \
     --service-name my-gpu-app \
     --environment production \
     --collect-system-metrics \
     python your_app.py
   

   Via Environment Variables

   Set environment variables:

   export OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"
   export OPENLIT_COLLECT_SYSTEM_METRICS=true
   export OTEL_SERVICE_NAME=my-gpu-app
   
   # Run your application
   openlit-instrument python your_app.py
   

   Manual instrumentation

   You can set up OpenLIT in your application using either function arguments directly in your code or by using environment variables.

   Parameters

   Add the following two lines to your application code:

   import openlit
   
   openlit.init(
     otlp_endpoint="http:127.0.0.1:4318", 
     collect_system_metrics=True  # This enables GPU monitoring
   )
   

   Replace:

   1. YOUR_OTEL_ENDPOINT with the URL of your OpenTelemetry backend, such as http://127.0.0.1:4318 if you are using OpenLIT and a local OTel Collector.

   Note: collect_system_metrics=True replaces the deprecated collect_gpu_stats=True

   Environment Variables

   Configure your OTLP endpoint using environment variables:

   export OTEL_EXPORTER_OTLP_ENDPOINT=""http://127.0.0.1:4318"
   export OPENLIT_COLLECT_SYSTEM_METRICS=true
   

   Add the following two lines to your application code:

   import openlit
   
   openlit.init(collect_system_metrics=True)
   

   Replace:

   1. YOUR_OTEL_ENDPOINT with the URL of your OpenTelemetry backend, such as http://127.0.0.1:4318 if you are using OpenLIT and a local OTel Collector.

   To send metrics to other Observability tools, refer to the supported destinations.

   For more advanced configurations and application use cases, visit the OpenLIT Python repository.

Using the Collector

1. Deploy OpenLIT
2. Pull `otel-gpu-collector` Docker Image

   You can quickly start using the OTel GPU Collector by pulling the Docker image:

   docker pull ghcr.io/openlit/otel-gpu-collector:latest
   
3. Run `otel-gpu-collector` Docker container

   You can quickly start using the OTel GPU Collector by pulling the Docker image: Here’s a quick example showing how to run the container with the required environment variables:

   docker run --gpus all \
       -e OTEL_SERVICE_NAME='chatbot' \
       -e OTEL_RESOURCE_ATTRIBUTES='deployment.environment=staging' \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318" \
       ghcr.io/openlit/otel-gpu-collector:latest
   

   For more advanced configurations of the collector, visit the OTel GPU Collector repository.

   Note: If you’ve deployed OpenLIT using Docker Compose, make sure to use the host’s IP address or add OTel GPU Collector to the Docker Compose:

   Docker Compose: Add the following config under `services`

   otel-gpu-collector:
     image: ghcr.io/openlit/otel-gpu-collector:latest
     environment:
       OTEL_SERVICE_NAME: 'chatbot'
       OTEL_RESOURCE_ATTRIBUTES: 'deployment.environment=staging'
       OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4318"
     device_requests:
     - driver: nvidia
       count: all
       capabilities: [gpu]
     depends_on:
     - otel-collector
     restart: always
   

   Host IP: Use the Host IP to connect to OTel Collector

   OTEL_EXPORTER_OTLP_ENDPOINT="http://192.168.10.15:4318"
   

---

Configuration

Configure the OpenLIT SDK according to you requirements.

Integrations

60+ AI integrations with automatic instrumentation and performance tracking

Destinations

Send elemetry to Datadog, Grafana, New Relic, and other observability stacks