NVIDIA Data Center GPU Manager (DCGM)

This document describes how to configure your Google Kubernetes Engine deployment so that you can use Google Cloud Managed Service for Prometheus to collect metrics from NVIDIA Data Center GPU Manager. This document shows you how to do the following:

  • Set up the exporter for DCGM to report metrics.

These instructions apply only if you are using managed collection with Managed Service for Prometheus. If you are using self-deployed collection, then see the source repository for DCGM Exporter for installation information.

These instructions are provided as an example and are expected to work in most Kubernetes environments. For information about a managed DCGM offering, see Collect and view DCGM metrics.

If you are having trouble installing an application or exporter due to restrictive security or organizational policies, then we recommend you consult open-source documentation for support.

For information about NVIDIA Data Center GPU Manager, see NVIDIA DCGM.

Prerequisites

To collect metrics from DCGM by using Managed Service for Prometheus and managed collection, your deployment must meet the following requirements:

  • Your cluster must be running Google Kubernetes Engine version 1.28.15-gke.2475000 or later.
  • You must be running Managed Service for Prometheus with managed collection enabled. For more information, see Get started with managed collection.

  • Verify that you have sufficient quota for NVIDIA GPUs.

  • To enumerate GPU nodes in your GKE cluster and their GPU types in the relevant cluster, run the following command:

    kubectl get nodes -l cloud.google.com/gke-accelerator -o jsonpath='{range .items[*]}{@.metadata.name}{" "}{@.metadata.labels.cloud\.google\.com/gke-accelerator}{"\n"}{end}'

  • Note that you might have to install a compatible NVIDIA GPU driver on the nodes if automatic installation was disabled or not supported for your GKE version. To verify that the NVIDIA GPU device plugin is running, run the following command:

    kubectl get pods -n kube-system | grep nvidia-gpu-device-plugin

Install the DCGM exporter

We recommend that you install the DCGM exporter, DCGM-Exporter, by using the following config:

# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-dcgm
  namespace: gmp-public
  labels:
    app: nvidia-dcgm
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm
  updateStrategy:
    type: RollingUpdate
  template:
    metadata: