Get started with managed collection

This document describes how to set up Google Cloud Managed Service for Prometheus with managed collection. The setup is a minimal example of working ingestion, using a Prometheus deployment that monitors an example application and stores collected metrics in Monarch.

This document shows you how to do the following:

• Set up your environment and command-line tools.
• Set up managed collection for your cluster.
• Configure a resource for target scraping and metric ingestion.
• Migrate existing prometheus-operator custom resources.

We recommend that you use managed collection; it reduces the complexity of deploying, scaling, sharding, configuring, and maintaining the collectors. Managed collection is supported for GKE and all other Kubernetes environments.

Managed collection runs Prometheus-based collectors as a Daemonset and ensures scalability by only scraping targets on colocated nodes. You configure the collectors with lightweight custom resources to scrape exporters using pull collection, then the collectors push the scraped data to the central datastore Monarch. Google Cloud never directly accesses your cluster to pull or scrape metric data; your collectors push data to Google Cloud. For more information about managed and self-deployed data collection, see Data collection with Managed Service for Prometheus and Ingestion and querying with managed and self-deployed collection.

Before you begin

This section describes the configuration needed for the tasks described in this document.

Set up projects and tools

To use Google Cloud Managed Service for Prometheus, you need the following resources:

• A Google Cloud project with the Cloud Monitoring API enabled.

  • If you don't have a Google Cloud project, then do the following:

    1. In the Google Cloud console, go to New Project:

       Create a New Project

    2. In the Project Name field, enter a name for your project and then click Create.

    3. Go to Billing:

       Go to Billing

    4. Select the project you just created if it isn't already selected at the top of the page.

    5. You are prompted to choose an existing payments profile or to create a new one.

    The Monitoring API is enabled by default for new projects.

  • If you already have a Google Cloud project, then ensure that the Monitoring API is enabled:

    1. Go to APIs & services:

       Go to APIs & services

    2. Select your project.

    3. Click Enable APIs and Services.

    4. Search for "Monitoring".

    5. In the search results, click through to "Cloud Monitoring API".

    6. If "API enabled" is not displayed, then click the Enable button.

• A Kubernetes cluster. If you do not have a Kubernetes cluster, then follow the instructions in the Quickstart for GKE.

You also need the following command-line tools:

• gcloud
• kubectl

The gcloud and kubectl tools are part of the Google Cloud CLI. For information about installing them, see Managing Google Cloud CLI components. To see the gcloud CLI components you have installed, run the following command:

gcloud components list

Configure your environment

To avoid repeatedly entering your project ID or cluster name, perform the following configuration:

• Configure the command-line tools as follows:

  • Configure the gcloud CLI to refer to the ID of your Google Cloud project:

    gcloud config set project PROJECT_ID
    

  • If running on GKE, use gcloud CLI to set your cluster:

    gcloud container clusters get-credentials CLUSTER_NAME --location LOCATION --project PROJECT_ID
    

  • Otherwise, use the kubectl CLI to set your cluster:

    kubectl config set-cluster CLUSTER_NAME
    

  For more information about these tools, see the following:

  • gcloud CLI overview
  • kubectl commands

Set up a namespace

Create the NAMESPACE_NAME Kubernetes namespace for resources you create as part of the example application. We recommend using the namespace name gmp-test when using this documentation to configure an example Prometheus setup.

Create the namespace by running the following:

kubectl create ns NAMESPACE_NAME

Set up managed collection

You can use managed collection on both GKE and non-GKE Kubernetes clusters.

After managed collection is enabled, the in-cluster components will be running but no metrics are generated yet. PodMonitoring or ClusterPodMonitoring resources are needed by these components to correctly scrape the metrics endpoints. You must either deploy these resources with valid metrics endpoints or enable one of the managed metrics packages, for example, Kube state metrics, built into GKE. For troubleshooting information, see Ingestion-side problems.

Enabling managed collection installs the following components in your cluster:

• The gmp-operator Deployment, which deploys the Kubernetes operator for Managed Service for Prometheus.
• The rule-evaluator Deployment, which is used to configure and run alerting and recording rules.
• The collector DaemonSet, which horizontally scales collection by scraping metrics only from pods running on the same node as each collector.
• The alertmanager StatefulSet, which is configured to send triggered alerts to your preferred notification channels.

For reference documentation about the Managed Service for Prometheus operator, see the manifests page.

Enable managed collection: GKE

Managed collection is enabled by default for the following:

• GKE Autopilot clusters running GKE version 1.25 or greater.

• GKE Standard clusters running GKE version 1.27 or greater. You can override this default when creating the cluster; see Disable managed collection.

If you are running in a GKE environment that does not enable managed collection by default, then see Enable managed collection manually.

Managed collection on GKE is automatically upgraded when new in-cluster component versions are released.

Managed collection on GKE uses permissions granted to the default Compute Engine service account. If you have a policy that modifies the standard permissions on the default node service account, you might need to add the Monitoring Metric Writer role to continue.

Enable managed collection manually

If you are running in a GKE environment that does not enable managed collection by default, then you can enable managed collection by using the following:

• The Managed Prometheus Bulk Cluster Enablement dashboard in Cloud Monitoring.
• The Kubernetes Engine page in the Google Cloud console.
• The Google Cloud CLI. To use the gcloud CLI, you must be running GKE version 1.21.4-gke.300 or newer.

• Terraform for Google Kubernetes Engine. To use Terraform to enable Managed Service for Prometheus, you must be running GKE version 1.21.4-gke.300 or newer.

gcloud config set project PROJECT_ID

gcloud container clusters create CLUSTER_NAME --zone ZONE --enable-managed-prometheus

Disable managed collection

If you want to disable managed collection on your clusters, then you can use one of the following methods:

gcloud config set project PROJECT_ID

gcloud container clusters create CLUSTER_NAME --zone ZONE --no-enable-managed-prometheus

Enable managed collection: non-GKE Kubernetes

If you are running in a non-GKE environment, then you can enable managed collection using the following:

• The kubectl CLI.

• VMware or bare metal on-premises deployments running version 1.12 or newer.

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/manifests/setup.yaml

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/manifests/operator.yaml

Deploy the example application

The example application emits the example_requests_total counter metric and the example_random_numbers histogram metric (among others) on its metrics port. The manifest for the application defines three replicas.

To deploy the example application, run the following command:

kubectl -n NAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/examples/example-app.yaml

Configure a PodMonitoring resource

To ingest the metric data emitted by the example application, Managed Service for Prometheus uses target scraping. Target scraping and metrics ingestion are configured using Kubernetes custom resources. The managed service uses PodMonitoring custom resources (CRs).

A PodMonitoring CR scrapes targets only in the namespace the CR is deployed in. To scrape targets in multiple namespaces, deploy the same PodMonitoring CR in each namespace. You can verify the PodMonitoring resource is installed in the intended namespace by running kubectl get podmonitoring -A.

For reference documentation about all the Managed Service for Prometheus CRs, see the prometheus-engine/doc/api reference.

The following manifest defines a PodMonitoring resource, prom-example, in the NAMESPACE_NAME namespace. The resource uses a Kubernetes label selector to find all pods in the namespace that have the label app.kubernetes.io/name with the value prom-example. The matching pods are scraped on a port named metrics, every 30 seconds, on the /metrics HTTP path.

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: prom-example
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prom-example
  endpoints:
  - port: metrics
    interval: 30s

To apply this resource, run the following command:

kubectl -n NAMESPACE_NAME apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.17.2/examples/pod-monitoring.yaml

Your managed collector is now scraping the matching pods. You can view the status of your scrape target by enabling the target status feature.

To configure horizontal collection that applies to a range of pods across all namespaces, use the ClusterPodMonitoring resource. The ClusterPodMonitoring resource provides the same interface as the PodMonitoring resource but does not limit discovered pods to a given namespace.

If you are running on GKE, then you can do the following:

• To query the metrics ingested by the example application using PromQL in Cloud Monitoring, see Query using Cloud Monitoring.
• To query the metrics ingested by the example application using Grafana, see Query using Grafana or any Prometheus API consumer.
• To learn about filtering exported metrics and adapting your prom-operator resources, see Additional topics for managed collection.

If you are running outside of GKE, then you need to create a service account and authorize it to write your metric data, as described in the following section.

Provide credentials explicitly

When running on GKE, the collecting Prometheus server automatically retrieves credentials from the environment based on the node's service account. In non-GKE Kubernetes clusters, credentials must be explicitly provided through the OperatorConfig resource in the gmp-public namespace.

1. Set the context to your target project:

   gcloud config set project PROJECT_ID
   

2. Create a service account:

   gcloud iam service-accounts create gmp-test-sa
   

3. Grant the required permissions to the service account:

   gcloud projects add-iam-policy-binding PROJECT_ID\
     --member=serviceAccount:gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com \
     --role=roles/monitoring.metricWriter
   

4. Create and download a key for the service account:

   gcloud iam service-accounts keys create gmp-test-sa-key.json \
     --iam-account=gmp-test-sa@PROJECT_ID.iam.gserviceaccount.com
   

5. Add the key file as a secret to your non-GKE cluster:

   kubectl -n gmp-public create secret generic gmp-test-sa \
     --from-file=key.json=gmp-test-sa-key.json
   

6. Open the OperatorConfig resource for editing:

   kubectl -n gmp-public edit operatorconfig config
   

   1. Add the text shown in bold to the resource:

      apiVersion: monitoring.googleapis.com/v1
      kind: OperatorConfig
      metadata:
        namespace: gmp-public
        name: config
      collection:
        credentials:
          name: gmp-test-sa
          key: key.json
      

      Make sure you also add these credentials to the rules section so that managed rule evaluation works.

   2. Save the file and close the editor. After the change is applied, the pods are re-created and start authenticating to the metric backend with the given service account.

   Additional topics for managed collection

   This section describes how to do the following:

   • Enable the target status feature for easier debugging.
   • Configure target scraping using Terraform.
   • Filter the data you export to the managed service.
   • Scrape Kubelet and cAdvisor metrics.
   • Convert your existing prom-operator resources for use with the managed service.
   • Run managed collection outside of GKE.

   Enabling the target status feature

   Managed Service for Prometheus provides a way to check whether your targets are being properly discovered and scraped by the collectors. This target status report is meant to be a tool for debugging acute problems. We strongly recommend only enabling this feature to investigate immediate issues. Leaving target status reporting on in large clusters might cause the operator to run out of memory and crash loop.

   You can check the status of your targets in your PodMonitoring or ClusterPodMonitoring resources by setting the features.targetStatus.enabled value within the OperatorConfig resource to true, as shown in the following:

   apiVersion: monitoring.googleapis.com/v1
   kind: OperatorConfig
   metadata:
     namespace: gmp-public
     name: config
   features:
     targetStatus:
       enabled: true
   

   After a few seconds, the Status.Endpoint Statuses field appears on every valid PodMonitoring or ClusterPodMonitoring resource, when configured.

   If you have a PodMonitoring resource with the name prom-example in the NAMESPACE_NAME namespace, then you can check the status by running the following command:

   kubectl -n NAMESPACE_NAME describe podmonitorings/prom-example
   

   The output looks like the following:

   API Version:  monitoring.googleapis.com/v1
   Kind:         PodMonitoring
   ...
   Status:
     Conditions:
       ...
       Status:                True
       Type:                  ConfigurationCreateSuccess
     Endpoint Statuses:
       Active Targets:       3
       Collectors Fraction:  1
       Last Update Time:     2023-08-02T12:24:26Z
       Name:                 PodMonitoring/custom/prom-example/metrics
       Sample Groups:
         Count:  3
         Sample Targets:
           Health:  up
           Labels:
             Cluster:                     CLUSTER_NAME
             Container:                   prom-example
             Instance:                    prom-example-589ddf7f7f-hcnpt:metrics
             Job:                         prom-example
             Location:                    REGION
             Namespace:                   NAMESPACE_NAME
             Pod:                         prom-example-589ddf7f7f-hcnpt
             project_id:                  PROJECT_ID
           Last Scrape Duration Seconds:  0.020206416
           Health:                        up
           Labels:
             ...
           Last Scrape Duration Seconds:  0.054189485
           Health: