Manage a repository

This document shows you how to do the following in Dataform:

• Configure Dataform workflow settings.
• Manage the Dataform core package.
• Version control your code.

Before you begin

1. Select or create a repository.

2. Optional: Connect your repository to a third-party Git repository.

3. Create and initialize a development workspace in your repository.

Required roles

To get the permissions that you need to complete the tasks in this document, ask your administrator to grant you the following IAM roles:

• Configure Dataform settings and manage the location of the Dataform core package: Dataform Admin (roles/dataform.admin) on repositories.
• Update the Dataform core package and use version control in Dataform: Dataform Editor (roles/dataform.editor) on workspaces.

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Configure Dataform workflow settings

This section shows you how to edit the Dataform workflow processing settings for a specific repository.

You might want to edit the settings file to rename the schemas or add custom compilation variables to your repository.

About repository settings

For all repositories, a custom service account is required for workflow execution and strict act-as mode is enforced. You can't use the default Dataform service agent to run workflows.

Each Dataform repository contains a unique workflow settings file. The file contains the Google Cloud project ID and the schema in which Dataform publishes assets in BigQuery. Dataform uses default settings that you can override to best suit your needs by editing the settings file.

As of Dataform core 3.0.0, workflow settings are stored in the workflow_settings.yaml file by default. In earlier versions of Dataform core, workflow settings are stored in the dataform.json file. Both of these configuration files must be in the root directory of your repository. The Dataform core 3.0 workflow_settings.yaml file is backward compatible with the dataform.json file. You can continue to use the dataform.json file to store workflow settings. As a best practice, you should migrate your repository workflow settings to the workflow_settings.yaml format for future compatibility.

About workflow_settings.yaml

The workflow_settings.yaml file, introduced in Dataform core 3.0, stores Dataform workflow settings in the YAML format.

The following code sample shows a sample workflow_settings.yaml file:

  defaultProject: my-gcp-project-id
  defaultDataset: dataform
  defaultLocation: australia-southeast2
  defaultAssertionDataset: dataform_assertions

In the preceding code sample, the key-value pairs are described as follows:

• defaultProject: your BigQuery Google Cloud project ID.
• defaultDataset: the BigQuery dataset in which Dataform creates assets, called dataform by default.

• defaultLocation (optional): your default BigQuery dataset location. Dataform uses this location to process your code and store the results. This processing location must match the location of your BigQuery datasets. However, it doesn't need to match the Dataform repository location.

  If you don't set the defaultLocation parameter, Dataform determines the location based on the datasets that your SQL query references. This works as follows:

  • If your query references datasets from the same location, Dataform uses that location.
  • If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see Cross-region dataset replication.
  • If your query doesn't reference any datasets, the default location for Dataform is the US multi-region. To choose a different location, set the default location. Alternatively, use the @@location system variable in your query. For more information, see Specify locations.

• defaultAssertionDataset: the BigQuery dataset in which Dataform creates views with assertion results, called dataform_assertions by default.

For more information about workflow_settings.yaml properties, see WorkflowSettings in GitHub.

You can access the properties defined in workflow_settings.yaml in your Dataform code as properties of the dataform.projectConfig object.

The following mappings from the workflow_settings.yaml options to the code-accessible dataform.projectConfig options apply:

• defaultProject => defaultDatabase
• defaultDataset => defaultSchema
• defaultAssertionDataset => assertionSchema
• projectSuffix => databaseSuffix
• datasetSuffix => schemaSuffix
• namePrefix => tablePrefix

The following code sample shows the dataform.projectConfig object referenced in a SELECT statement in a view:

  config { type: "view" }
  SELECT ${when(
    !dataform.projectConfig.tablePrefix,
    "table prefix is set!",
    "table prefix is not set!"
  )}

About dataform.json

The dataform.json file stores Dataform workflow settings in the JSON format.

The following code sample shows a sample dataform.json file:

  {
    "warehouse": "bigquery",
    "defaultDatabase": "my-gcp-project-id",
    "defaultSchema": "dataform",
    "defaultLocation": "australia-southeast2",
    "assertionSchema": "dataform_assertions"
  }

In the preceding code sample, the key-value pairs are described as follows:

• warehouse: a pointer to BigQuery, where Dataform creates assets.
• defaultDatabase: your BigQuery Google Cloud project ID.
• defaultSchema: the BigQuery dataset in which Dataform creates assets.

• defaultLocation (optional): your default BigQuery dataset location. Dataform uses this location to process your code and store the results. This processing location must match the location of your BigQuery datasets. However, it doesn't need to match the Dataform repository location.

  If you don't set the defaultLocation parameter, Dataform determines the location based on the datasets that your SQL query references. This works as follows:

  • If your query references datasets from the same location, Dataform uses that location.
  • If your query references datasets from two or more different locations, an error occurs. For details about this limitation, see Cross-region dataset replication.
  • If your query doesn't reference any datasets, the default location for Dataform is the US multi-region. To choose a different location, set the default location. Alternatively, use the @@location system variable in your query. For more information, see Specify locations.

• assertionSchema: the BigQuery dataset in which Dataform creates views with assertion results, called dataform_assertions by default.

You can access the properties defined in the dataform.json file in your project code as properties of the dataform.projectConfig object.

Configure schema names

To configure schema names, you need to edit the defaultDataset and defaultAssertionSchema properties in the workflow_settings.yaml file, or the defaultSchema and assertionSchema properties in the dataform.json file.

To configure the name of a schema, follow these steps:

  ...
  defaultDataset: mytables
  ...

{
  ...
  "defaultSchema": "mytables",
  ...
}

Create custom compilation variables

Compilation variables contain values that you can modify with compilation overrides in a release configuration or in a Dataform API request.

After you define a compilation variable in workflow_settings.yaml and add it to selected tables, you can modify its value in a release configuration or Dataform API compilation overrides to execute tables conditionally.

For more information about executing tables conditionally by using compilation variables, see Introduction to code lifecycle in Dataform.

To create a compilation variable that you can use across a repository, follow these steps:

...
vars:
  myVariableName: myVariableValue
...

defaultProject: default_bigquery_database
defaultLocation: us-west1
defaultDataset: dataform_data,
vars:
executionSetting: dev

{
  ...
  "vars": {
    "myVariableName": "myVariableValue"
  },
  ...
}

{
"warehouse": "bigquery",
"defaultSchema": "dataform_data",
"defaultDatabase": "default_bigquery_database".
"defaultLocation":"us-west-1",
"vars": {
"executionSetting":"dev"
}
}

Add a compilation variable to a table

To add a compilation variable to a SQLX table definition file, follow these steps:

1. Go to your Dataform development workspace.
2. In the Files pane, select a SQLX table definition file.

3. In the file, enter a when clause in the following format:

   ${when(dataform.projectConfig.vars.VARIABLE === "SET_VALUE", "CONDITION")}
   

   Replace the following:

   • VARIABLE: the name of your variable—for example executionSetting
   • SET_VALUE: a value for the variable—for example, staging
   • CONDITION: a condition for execution of the table

The following code sample shows a table definition SQLX file with a when clause and the executionSetting variable that executes 10% of data in the staging execution setting:

  select
    *
  from ${ref("data")}
  ${when(
    dataform.projectConfig.vars.executionSetting === "staging",
    "where mod(farm_fingerprint(id) / 10) = 0",
  )}

The following code sample shows a view definition SQLX file with a when clause and the myVariableName variable:

  config { type: "view" }
  SELECT ${when(
    dataform.projectConfig.vars.myVariableName === "myVariableValue",
    "myVariableName is set to myVariableValue!",
    "myVariableName is not set to myVariableValue!"
  )}

Configure workflow-level defaults for Iceberg tables

You can provide workflow-level defaults for Apache Iceberg managed tables in the workflow_settings.yaml file. You can set default values for the bucket name, table folder root, and table folder subpath.

Dataform uses these default values to construct the storage URI for an Iceberg managed table if you don't set these values in the config block of the table definition file.

To configure workflow-level defaults for the tables, add the defaultIcebergConfig block with the following fields to your workflow_settings.yaml file:

• bucketName: Optional. The default name for the Cloud Storage bucket.
• tableFolderRoot: Optional. The default name for the first-level folder inside the Cloud Storage bucket.
• tableFolderSubpath: Optional. The default name for a sub-folder within the tableFolderRoot.

The following code sample shows a workflow_settings.yaml file with workflow-level defaults for Iceberg managed tables:

defaultProject: "defaultProject"
defaultDataset: "defaultDataset"
defaultLocation: "us-central1"
defaultIcebergConfig:
  bucketName: "wsDefaultBucketName"
  tableFolderRoot: "wsDefaultTableFolderRoot"
  tableFolderSubpath: "wsDefaultTableFolderSubpath"

You can access these fields in your Dataform code as properties of the ProjectConfig object:

dataform.ProjectConfig.defaultIcebergConfig.bucketName
dataform.ProjectConfig.defaultIcebergConfig.tableFolderRoot
dataform.ProjectConfig.defaultIcebergConfig.tableFolderSubpath

For more information, see Create Apache Iceberg managed tables and the iceberg block properties.

Migrate workflow settings to workflow_settings.yaml

To ensure your workflow settings file is compatible with future Dataform core framework versions, you should migrate your workflow settings from the dataform.json file to the workflow_settings.yaml file.

The workflow_settings.yaml file replaces the dataform.json file.

If Dataform core is the only dependency package in your repository, then the workflow_settings.yaml file also replaces the package.json file. For more information about replacing the package.json file with the workflow_settings.yaml file, see Manage the Dataform core package.

The following table shows mapping of workflow settings properties from the dataform.json file to the workflow_settings.yaml file:

To migrate workflow settings to workflow_settings.yaml, follow these steps:

1. In the Google Cloud console, go to the Dataform page.

   Go to Dataform

2. Select a repository, and then select a workspace.

3. In the Files pane, click addAdd, and then click Create file.

4. In the Add a file path field, enter workflow_settings.yaml.

5. Click Create file.

6. In the workflow_settings.yaml file, add the settings from your dataform.json file, mapped to the YAML format.

7. In the Files pane, click the More menu next to dataform.json, and then click Delete.

8. To confirm deletion of dataform.json, click Delete.

The following code sample shows workflow settings defined in a dataform.json file:

{
  "warehouse": "bigquery",
  "defaultDatabase": "dataform-demos",
  "defaultLocation": "US",
  "defaultSchema": "dataform",
  "assertionSchema": "dataform_assertions"
  "vars": {
    "environmentName": "development"
  }
}

The following code sample shows the preceding dataform.json file converted to workflow_settings.yaml:

defaultProject: dataform-demos