Fine-tune and scale reinforcement learning with verl on GKE

This tutorial shows you how to orchestrate a distributed training environment for reinforcement learning on Google Kubernetes Engine (GKE). You use Ray and the verl (Volcano Engine Reinforcement Learning) framework to set up a distributed training environment to fine-tune a Qwen2.5-32B-Instruct model on the GSM8K dataset.

This tutorial focuses on the Group Relative Policy Optimization (GRPO) training pipeline on GKE with Ray and verl. GRPO is a reinforcement learning algorithm designed to improve a model's reasoning ability. This memory-efficient algorithm simplifies the reinforcement learning (RL) process by eliminating the Critic, or value model, and using a relative group-based calculation instead.

This tutorial is a good starting point if you need to set up a distributed training environment where data, model weights, and the training engine are decoupled for efficiency.

This tutorial supports the following GPU architectures:

  • Intel or AMD-based GPU nodes: Set up and scale using NVIDIA B200 or H200 GPUs, using GKE Dynamic Resource Allocation (DRA) for Autopilot path.
  • Arm-based A4X (GB200) nodes: Set up and scale using NVIDIA GB200 Grace Blackwell Superchips, using GKE Dynamic Resource Allocation (DRA) and Multi-Node NVLink (IMEX).

Background

The following sections provide a brief overview of the concepts used in this tutorial.

Reinforcement learning (RL)

RL teaches models through experience, exploration, and feedback rather than static imitation. Although pre-training teaches a model what to say, reinforcement learning from human feedback (RLHF) teaches it how to be helpful, safe, and logical. RL serves as the bridge between a base model and a fine-tuned model for a specialized use case.

For more information, see What is reinforcement learning?

Group Relative Policy Optimization (GRPO)

GRPO, an algorithm popularized by DeepSeek, offers a memory-efficient alternative to Proximal Policy Optimization (PPO) for LLM alignment by removing the Critic model. Instead of a Critic network, GRPO generates a group of responses for the same prompt and uses the average reward of that group as the baseline.

For more information, see GRPO.

Volcano Engine Reinforcement Learning (verl)

verl is a high-performance framework designed to handle the complex memory and compute patterns of LLM-based RL.

For more information, see verl.

Objectives

This tutorial shows you how set up reinforcement learning on GKE with verl, by completing the following steps:

  1. Set up a GKE cluster with A4X (GB200 Superchips), A4 (B200 GPUs), or A3 Ultra (H200 GPUs).
  2. Configure KubeRay to manage a distributed Ray cluster.
  3. Use Cloud Storage FUSE to mount a Cloud Storage bucket across all nodes.
  4. Run a GRPO training job using verl to align the Qwen2.5-32B-Instruct model with the GSM8K dataset.

Before you begin

  • Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  • Install the Google Cloud CLI.

  • If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  • To initialize the gcloud CLI, run the following command: 

    gcloud init

  • Create or select a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the