Apache Flink

The Apache Flink integration collects client, jobmanager and taskmanager logs and parses them into a JSON payload. The result includes fields for source, level, and message.

For more information about Flink, see the Apache Flink documentation.

Prerequisites

To collect Flink telemetry, you must install the Ops Agent:

  • For metrics, install version 2.18.1 or higher.
  • For logs, install version 2.17.0 or higher.

This integration supports Flink versions 1.12.5, 1.13.6, and 1.14.4.

Configure the Ops Agent for Flink

Following the guide to Configure the Ops Agent, add the required elements to collect telemetry from Flink instances, and restart the agent.

Example configuration

The following commands create the configuration to collect and ingest telemetry for Flink:

# Configures Ops Agent to collect telemetry from the app. You must restart the agent for the configuration to take effect.

set -e

# Check if the file exists
if [ ! -f /etc/google-cloud-ops-agent/config.yaml ]; then
  # Create the file if it doesn't exist.
  sudo mkdir -p /etc/google-cloud-ops-agent
  sudo touch /etc/google-cloud-ops-agent/config.yaml
fi

# Create a back up of the existing file so existing configurations are not lost.
sudo cp /etc/google-cloud-ops-agent/config.yaml /etc/google-cloud-ops-agent/config.yaml.bak

# Configure the Ops Agent.
sudo tee /etc/google-cloud-ops-agent/config.yaml > /dev/null << EOF
metrics:
  receivers:
    flink:
      type: flink
  service:
    pipelines:
      flink:
        receivers:
          - flink
logging:
  receivers:
    flink:
      type: flink
  service:
    pipelines:
      flink:
        receivers:
          - flink
EOF

For these changes to take effect, you must restart the Ops Agent:

Linux

  1. To restart the agent, run the following command on your instance: 
    sudo systemctl restart google-cloud-ops-agent
  2. To confirm that the agent restarted, run the following command and verify that the components "Metrics Agent" and "Logging Agent" started: 
    sudo systemctl status "google-cloud-ops-agent*"

Windows

  1. Connect to your instance using RDP or a similar tool and login to Windows.
  2. Open a PowerShell terminal with administrator privileges by right-clicking the PowerShell icon and selecting Run as Administrator
  3. To restart the agent, run the following PowerShell command: 
    Restart-Service google-cloud-ops-agent -Force
  4. To confirm that the agent restarted, run the following command and verify that the components "Metrics Agent" and "Logging Agent" started: 
    Get-Service google-cloud-ops-agent*

To ingest logs from Flink, you must create a receiver for the logs that Flink produces and then create a pipeline for the new receiver.

To configure a receiver for your flink logs, specify the following fields:

Field Default Description
exclude_paths A list of filesystem path patterns to exclude from the set matched by include_paths.
include_paths [/opt/flink/log/flink-*-standalonesession-*.log, /opt/flink/log/flink-*-taskexecutor-*.log, /opt/flink/log/flink-*-client-*.log] A list of filesystem paths to read by tailing each file. A wild card (*) can be used in the paths.
record_log_file_path false If set to true, then the path to the specific file from which the log record was obtained appears in the output log entry as the value of the agent.googleapis.com/log_file_path label. When using a wildcard, only the path of the file from which the record was obtained is recorded.