As of April 20th, 2026, BigLake is now called Lakehouse. BigLake metastore is now called the Lakehouse runtime catalog. Lakehouse APIs, client libraries, CLI commands, and IAM names remain unchanged and still reference BigLake.

Set up Spark and Hive with the Lakehouse runtime catalog

This document explains how to set up and configure Apache Spark and Apache Hive to use the Lakehouse runtime catalog. You learn how to create an Apache Hive catalog, configure your Spark sessions to connect to the metastore, and run workloads to create tables that you can query directly in BigQuery.

Before you begin

Read About Hive Catalogs in Lakehouse runtime catalog to understand how Spark connects to the Lakehouse runtime catalog.
Review Supported storage formats and data types.
Review Limitations and considerations.
Verify that billing is enabled for your Google Cloud project.
Enable the Lakehouse, Managed Service for Apache Spark APIs.
Roles required to enable APIs
To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.
Enable the APIs

Required roles

To get the permissions that you need to use Lakehouse runtime catalog, ask your administrator to grant you the following IAM roles on your project:

Create a Managed Service for Apache Spark cluster: Dataproc Editor (roles/dataproc.editor)
Cluster service account:
- Dataproc Worker (roles/dataproc.worker)
- Storage Object User (roles/storage.objectUser)
- BigLake Editor (roles/biglake.editor)
- Service Usage Consumer (roles/serviceusage.serviceUsageConsumer)
Write access to all Lakehouse runtime catalog resources: BigLake Editor (roles/biglake.editor)
Read-only access to all Lakehouse runtime catalog resources: BigLake Viewer (roles/biglake.viewer)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For instructions, see Granting a single role.

General workflow

To use the Lakehouse runtime catalog with Spark and Hive, you follow this general workflow:

Create a Lakehouse Hive catalog.
Configure your Spark session by using your preferred tool (such as Managed Service for Apache Spark or BigQuery Studio).
Perform database and table operations within your Spark session.
Submit batch workloads to Managed Service for Apache Spark and query the resulting tables directly from BigQuery.

Create a Lakehouse Hive catalog

To use the Lakehouse runtime catalog with Spark and Hive, you must first create a Hive catalog.

A Lakehouse Hive catalog is a collection of Hive databases. Before you run Spark jobs, create a catalog to register it with Lakehouse Metastore. The catalog has a name and a Cloud Storage location where the Hive data resides.

Console

In the Google Cloud console, open the Lakehouse page.

Go to Lakehouse
Click Create catalog.
Select Lakehouse runtime catalog.
For Catalog type, select Hive Metastore.
In the Select a Cloud Storage bucket field, enter the name of the Cloud Storage bucket to use with your catalog. Alternatively, click Browse to choose an existing bucket or create a new one.
For Catalog ID, give a name to your Lakehouse Hive catalog.
For Primary location, specify the same region as your bucket.
Click Create.

gcloud

To create a Hive catalog, run the following command:

gcloud beta biglake hive catalogs create LAKEHOUSE_CATALOG_ID \
    --project=PROJECT_ID \
    --location-uri="gs://GCS_WAREHOUSE_PATH" \
    --primary-location=REGION \
    --description="DESCRIPTION"

Replace the following:

LAKEHOUSE_CATALOG_ID: the Hive catalog name.
GCS_WAREHOUSE_PATH: the Cloud Storage path that stores your Hive warehouse.
PROJECT_ID: your Google Cloud project ID.
REGION: the primary region of the metastore. For single-region buckets, it should match the bucket region. For dual-region or multi-region buckets, it should be one of the constituent regions and where the primary replica of the metastore is intended. The other region becomes the secondary replica.