Dataform overview

This document introduces you to Dataform concepts and processes.

Dataform is a service for data analysts to develop, test, control versions, and schedule complex workflows for data transformation in BigQuery.

Dataform lets you manage data transformation in the Extraction, Loading, and Transformation (ELT) process for data integration. After raw data is extracted from source systems and loaded into BigQuery, Dataform helps you to transform it into a well-defined, tested, and documented suite of data tables.

Dataform lets you perform the following data transformation actions:

  • Develop and run workflows for data transformation.
  • Collaborate with team members on workflow development through Git.
  • Manage a large number of tables and their dependencies.
  • Declare source data and manage table dependencies.
  • View a visualization of the dependency tree of your workflow.
  • Manage data with SQL code in a central repository.
  • Reuse code with JavaScript.
  • Test data correctness with quality tests on source and output tables.
  • Version control SQL code.
  • Document data tables inside SQL code.

Data transformation processes in Dataform

The data transformation workflow for Dataform is as follows:

  1. Dataform lets you create repositories to manage your code.
  2. Dataform lets you create workspaces for development.
  3. Dataform lets you develop workflows in a development workspace.
  4. Dataform compiles Dataform core into SQL.
  5. Dataform runs the dependency tree.

Dataform lets you create repositories to manage your code

In a Dataform repository, you use Dataform core, an extension of SQL, to write SQLX files in which you define your workflow. Dataform repositories support version control. You can link a Dataform repository to a third-party Git provider.

Dataform lets you create workspaces for development

You can create development workspaces inside a Dataform repository for Dataform core development. In a development workspace, you can make changes to the repository, compile, test, and push them to the main repository through Git.

Dataform lets you develop Dataform core in a development workspace

In a development workspace, you can define and document tables, their dependencies, and transformation logic to build your workflow. You can also configure actions in JavaScript.

Dataform compiles Dataform core

During compilation, Dataform performs the following tasks:

  • Compiles Dataform core into a workflow of Standard SQL.
  • Adds boilerplate SQL statements, such as CREATE TABLE or INSERT, to the code inline with your query configuration.
  • Transpiles (compiles source-to-source) JavaScript into SQL.
  • Resolves dependencies and checks for errors including missing or circular dependencies.
  • Builds the dependency tree of all actions to be run in BigQuery.

Dataform compilation is hermetic to verify compilation consistency, meaning that the same code compiles to the same SQL compilation result every time. Dataform compiles your code in a sandbox environment with no internet access. No additional actions, such as calling external APIs, are available during compilation.

To debug in real-time, you can inspect the compiled workflow of your project in an interactive graph in your development workspace.

Dataform runs the dependency tree

In BigQuery, Dataform performs the following tasks:

  • Runs SQL commands, following the order of the dependency tree.
  • Runs assertion queries against your tables and views to check data correctness.
  • Runs other SQL operations that you defined.
  • Updates metadata in Knowledge Catalog (Preview).

After the execution, you can use your tables and views for all your analytics purposes.

You can view logs to see what tables were created, if assertions passed or failed, how long each action took to complete, and other information. You can also view the exact SQL code that was run in BigQuery.

Dataform features

With Dataform, you can develop and deploy tables, incremental tables, or views to BigQuery. Dataform offers a web environment for the following activities:

  • Workflow development
  • Connection with GitHub, GitLab, Azure DevOps Services, and Bitbucket
  • Continuous integration and continuous deployment
  • Workflow execution

The following sections describe the main features of Dataform.

Repositories

Each Dataform project is stored in a repository. A Dataform repository houses a collection of JSON configuration files, SQLX files, and JavaScript files.

Dataform repositories contain the following types of files:

  • Config files

    Config JSON or SQLX files let you configure your workflows. They contain general configuration, execution schedules, or schema for creating new tables and views.

  • Definitions

    Definitions are SQLX and JavaScript files that define new tables, views, and additional SQL operations to run in BigQuery.

  • Includes

    Includes are JavaScript files where you can define variables and functions to use in your project.

Each Dataform repository must be connected to a custom service account. You select a custom service account when you create a repository. You can edit the service account later.

Version control

Dataform uses the Git version control system to maintain a record of each change made to project files and to manage file versions.

Each Dataform repository can manage its own Git repository, or be connected to a remote third-party Git repository. You can connect a Dataform repository to a GitHub, GitLab, Azure DevOps Services, or Bitbucket repository.

Users version control their workflow code inside Dataform workspaces. In a Dataform workspace, you can pull changes from the repository, commit all or selected changes, and push them to Git branches of the repository.

Workflow development

In Dataform, you make changes to files and directories inside a development workspace. A development workspace is a virtual, editable copy of the contents of a Git repository. Dataform preserves the state of files in your development workspace between sessions.

In a development workspace, you can develop workflow actions by using Dataform core with SQLX and JavaScript, or exclusively with JavaScript. You can automatically format your Dataform core or JavaScript code.

Each element of a Dataform workflow, such as a table or assertion, corresponds to an action that Dataform performs in BigQuery. For example, a table definition file is an action of creating or updating the table in BigQuery.

In a Dataform workspace, you can develop the following workflow actions: