Connect a Git repository and run a workflow

Console

1. Go to the BigQuery Dataform page.

   Go to Dataform

2. Click add Create repository.

3. On the Create repository page, do the following:

   1. In the Repository ID field, enter quickstart-repo.
   2. In the Region list, select a region—for example, europe-west4.
   3. In the Service account list, select a custom service account for the repository.
   4. Click Create.
   5. Click Go to repositories.

You have successfully created a Dataform repository. Next, you can connect the Dataform repository to a remote Git repository.

API

To create a repository, use the projects.locations.repositories.create method. Run the API request with the following information:

• Endpoint: POST https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories
• Query parameter: repositoryId=REPOSITORY_ID

Alternatively, in your terminal, run the following curl command:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"serviceAccount": "SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com"}' \
  "https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories?repositoryId=REPOSITORY_ID"

Replace the following:

• PROJECT_ID: the unique identifier of the Google Cloud project where you want to create the Dataform repository.
• LOCATION: the Google Cloud region where you want to create the repository—for example, europe-west4.
• REPOSITORY_ID: the unique identifier for your new Dataform repository—for example, quickstart-repo.
• SERVICE_ACCOUNT_NAME: the ID of the custom service account created to run BigQuery jobs.

You have successfully created a Dataform repository. Next, you can connect the Dataform repository to a remote Git repository.

Console

1. Go to the Secret Manager page.

   Go to Secret Manager

2. Click add Create secret.

3. In the Name field, enter dataform-git-token.

4. In the Secret value field, enter your GitHub personal access token (PAT).

   For instructions on how to create a PAT, see Managing your personal access tokens.

   We recommend setting an expiration date for your token according to your organization's security policies.

5. Click Create secret.

6. On the secret details page, click the Permissions tab, and then click Grant access.

7. In the New principals field, enter your Dataform service agent: service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com.

   Replace PROJECT_NUMBER with your Google Cloud project number. For details on finding your project number, see Find the project name, number, and ID.

8. In the Select a role field, select Secret Manager > Secret Manager Secret Accessor.

9. Click Save.

10. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

11. Click quickstart-repo.

12. On the repository page, click Settings > Connect with Git.

13. In the Link to remote repository pane, select HTTPS.

14. In the Remote Git repository URL field, enter https://github.com/dataform-co/dataform-example-project-bigquery.git.

15. In the Default remote branch name field, enter master.

16. In the Secret menu, select dataform-git-token.

17. Click Link.

You have successfully connected your Dataform repository to a remote Git repository and granted the necessary permissions. Next, you can create and initialize a development workspace.

API

1. To store your Git personal access token, create a secret in Secret Manager with the projects.secrets.create method. Run the API request with the following information:

   • Endpoint: POST https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets
   • Query parameter: secretId=dataform-git-token

   • Body:

     {
       "replication": {
         "automatic": {}
       }
     }
     

   Alternatively, in your terminal, run the following curl command:

   curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
       "replication": {
         "automatic": {}
       }
     }' \
     "https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets?secretId=dataform-git-token"
   

2. Add a version to the secret containing your GitHub personal access token (PAT). For instructions on how to create a PAT, see Managing your personal access tokens. We recommend setting an expiration date for your token according to your organization's security policies.

   To add a secret version, use the projects.secrets.addVersion method. Run the API request with the following information:

   • Endpoint: POST https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets/dataform-git-token:addVersion

   • Body:

     {
       "payload": {
         "data": "GITHUB_PAT"
       }
     }
     

   Alternatively, in your terminal, run the following curl command:

   curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
       "payload": {
         "data": "$(echo -n "GITHUB_PAT" | base64)"
       }
     }' \
     "https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets/dataform-git-token:addVersion"
   

   Replace GITHUB_PAT with your GitHub personal access token. The curl command automatically converts your PAT to a Base64-encoded string before sending the request.

3. To let Dataform access the secret, grant the Secret Manager Secret Accessor role (roles/secretmanager.secretAccessor) to the Dataform service agent. To grant the role, select one of the following options:

   gcloud

   Run the gcloud secrets add-iam-policy-binding command:

   gcloud secrets add-iam-policy-binding dataform-git-token \
       --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com" \
       --role="roles/secretmanager.secretAccessor"
   

   Replace PROJECT_NUMBER with your Google Cloud project number. For details on finding your project number, see Find the project name, number, and ID.

   Secret Manager API

   Use the projects.secrets.setIamPolicy method. Run the API request with the following information:

   • Endpoint: POST https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets/dataform-git-token:setIamPolicy

   • Body:

     {
       "policy": {
         "bindings": [
           {
             "role": "roles/secretmanager.secretAccessor",
             "members": [
               "serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com"
             ]
           }
         ]
       }
     }
     

   Alternatively, in your terminal, run the following curl command:

   curl -X POST \
   -H "Authorization: Bearer $(gcloud auth print-access-token)" \
   -H "Content-Type: application/json" \
   -d '{
     "policy": {
       "bindings": [
         {
           "role": "roles/secretmanager.secretAccessor",
           "members": [
             "serviceAccount:service-PROJECT_NUMBER@gcp-sa-dataform.iam.gserviceaccount.com"
           ]
         }
       ]
     }
   }' \
   "https://secretmanager.googleapis.com/v1/projects/PROJECT_ID/secrets/dataform-git-token:setIamPolicy"
   

   Replace PROJECT_NUMBER with your Google Cloud project number. For details on finding your project number, see Find the project name, number, and ID.

4. To connect your repository to a remote Git repository, use the projects.locations.repositories.patch method. Run the API request with the following information:

   • Endpoint: PATCH https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID
   • Query parameter: updateMask=gitRemoteSettings

   Alternatively, in your terminal, run the following curl command:

   curl -X PATCH \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
       "gitRemoteSettings": {
         "url": "https://github.com/dataform-co/dataform-example-project-bigquery.git",
         "defaultBranch": "master",
         "authenticationTokenSecretVersion": "projects/PROJECT_ID/secrets/dataform-git-token/versions/1"
       }
     }' \
     "https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID?updateMask=gitRemoteSettings"
   

You have successfully connected your Dataform repository to a remote Git repository and granted the necessary permissions. Next, you can create and initialize a development workspace.

Console

1. Go to the BigQuery Dataform page.

   Go to Dataform

2. Click quickstart-repo.

3. In your repository, go to the Development Workspaces tab.

4. Click add Create development workspace.

5. In the Workspace ID field, enter dev-workspace.

6. Click Create.

7. On the Development Workspaces tab, select the dev-workspace workspace.

You have successfully created and initialized a development workspace. Next, you can configure the workflow settings.

API

To create a workspace, use the projects.locations.repositories.workspaces.create method. Run the API request with the following information:

• Endpoint: POST https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces
• Query parameter: workspaceId=WORKSPACE_ID

Alternatively, in your terminal, run the following curl command:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d "{}" \
  "https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces?workspaceId=WORKSPACE_ID"

Replace WORKSPACE_ID with the unique identifier for your new Dataform development workspace—for example, dev-workspace.

You have successfully created and initialized a development workspace. Next, you can configure the workflow settings.

Console

1. Go to the BigQuery Dataform page.

   Go to Dataform

2. Click quickstart-repo.

3. In your repository, go to the Development Workspaces tab, and then click dev-workspace.

4. In the Files pane, select workflow_settings.yaml.

5. In the file, replace the value of defaultProject with your project ID.

   The file is automatically saved.

You have successfully updated your workflow settings. Next, you can add a new source declaration to your project.

API

1. Create a local file named workflow_settings.yaml and paste the following configuration into the file:

   defaultProject: PROJECT_ID
   defaultDataset: dataform
   dataformCoreVersion: CORE_VERSION
   

   Replace CORE_VERSION with the latest stable (non-beta) version of Dataform core—for example, 3.0.43. You can find the latest version listed in Releases.

2. In your terminal, encode the file content into a Base64 string:

   base64 -w 0 workflow_settings.yaml
   

3. Copy the resulting output string to use in the SETTINGS_DEFINITION placeholder should you decide to use the alternative curl command later in these steps.

4. To update your workflow settings, use the projects.locations.repositories.workspaces.writeFile method. Run the API request with the following information:

   • Endpoint: POST https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces/WORKSPACE_ID:writeFile

   Alternatively, in your terminal, run the following curl command:

   curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
       "path": "workflow_settings.yaml",
       "contents": "SETTINGS_DEFINITION"
     }' \
     "https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces/WORKSPACE_ID:writeFile"
   

   Replace SETTINGS_DEFINITION with the YAML file's content as a Base64-encoded string.

You have successfully updated your workflow settings. Next, you can add a new source declaration to your project.

Console

1. Go to the BigQuery Dataform page.

   Go to Dataform

2. Click quickstart-repo.

3. In your repository, go to the Development Workspaces tab, and then click dev-workspace.

4. In the Files pane, select the definitions folder.

5. Click more_vert More file actions > Create file.

6. In the Add a file path field, enter definitions/sources/tags.sqlx.

7. Click Create file.

8. In the SQL editor for the new definitions/sources/tags.sqlx file, paste the following code:

   config {
     type: "declaration",
     database: "bigquery-public-data",
     schema: "stackoverflow",
     name: "tags"
   }
   

You have successfully created a source declaration. Next, you can add a new view to your project.

API

1. Create a local file named tags.sqlx.

2. Paste the following code into the tags.sqlx file:

   config {
     type: "declaration",
     database: "bigquery-public-data",
     schema: "stackoverflow",
     name: "tags"
   }
   

3. In your terminal, encode the file content into a single continuous string:

   base64 -w 0 tags.sqlx
   

4. Copy the resulting output string to use in the SOURCE_DEFINITION placeholder should you decide to use the alternative curl command later in these steps.

5. To create a source declaration file in your workspace, use the projects.locations.repositories.workspaces.writeFile method. Run the API request with the following information:

   • Endpoint: POST https://dataform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/repositories/REPOSITORY_ID/workspaces/WORKSPACE_ID:writeFile

   Alternatively, in your terminal, run the following