Use JavaScript packages

This document shows you how to do the following:

Install a JavaScript package in a Dataform repository.
Authenticate a private NPM package to enable its installation in a repository.
Create a custom JavaScript package that you can use to develop workflows.

Before you begin

In the Google Cloud console, go to the Dataform page.

Go to Dataform
Do one or both of the following:
1. To install a package in a repository or authenticate a private NPM package to enable its installation, follow these steps:
  1. Select or create a repository.
  2. Select or create a development workspace.
  3. Optional: To install a private package, authenticate the private package.
  4. If your repository doesn't contain a package.json file, create package.json and move the Dataform core package.
2. To create a package, follow these steps:
  1. Create a Dataform repository that's dedicated to your package. Match the repository name to the name of your package.
  2. Connect the repository to a third-party Git repository that will host your package.
  3. Create and initialize a workspace in the Dataform repository.
Ensure that you have the necessary permissions to complete the tasks in this document.

Required roles

To get the permissions that you need to complete the tasks in this document, ask your administrator to grant you the following IAM roles:

Dataform Editor (roles/dataform.editor) on workspaces and repositories
Dataform Admin (roles/dataform.admin) on repositories

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Install a package

This section shows you how to install a JavaScript package and import it to a JavaScript file and a SQLX file so that you can use the package to develop workflows in Dataform.

To use a package in Dataform, you need to install it in your repository.

You can install the following types of packages in Dataform:

Published public NPM packages
Non-published public NPM packages
Authenticated private NPM packages

Then, to use the package in a JavaScript or SQLX file, you need to import selected contents of the package to the file. You can also import a whole package to a JavaScript or SQLX file instead of its selected contents.

To prevent issues with package installation in your production environment, we recommend that you do the following:

Explicitly specify the package version in package.json, for example, 3.0.0. Don't use other dependencies options of package.json, for example, >version.
Test new package versions in a non-production environment. For more information about configuring different workflow lifecycle environments, see Best practices for the workflow lifecycle.

Add a package as a dependency

To install a package inside a Dataform repository, you need to add it as a dependency in the package.json file:

In your workspace, in the Files pane, select package.json.
Add the package to the dependencies block:
1. Add a published public NPM package in the following format:
```
"PACKAGE-NAME": "PACKAGE-VERSION"
```
  Replace the following:
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-VERSION with the latest version of the published public NPM package. To prevent issues with package installation, explicitly specify the version, for example, 3.0.0.
2. Add a non-published public NPM package in the following format:
```
"PACKAGE-NAME": "PACKAGE-URL"
```
  Replace the following:
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-URL with the tar.gz URL of the third-party package repository, for example https://github.com/user/sample-package-repository/archive/master.tar.gz.
3. Add an authenticated private NPM package in the following format:
```
"REGISTRY-SCOPE/PACKAGE-NAME": "PACKAGE-URL"
```
  Replace the following:
  - REGISTRY-SCOPE with the name of the package. REGISTRY-SCOPE must match the registry scope defined in the .npmrc file in your repository.
  - PACKAGE-NAME with the name of the package.
  - PACKAGE-URL with the tar.gz URL of the package repository, for example https://github.com/user/sample-package-repository/archive/master.tar.gz.
Click Install packages.
Commit and push your changes.

The following code sample shows the public open-source Slowly changing dimensions package package added to the .package.json file:

 ```json
 {
   "name": "repository-name",
   "dependencies": {
     "@dataform/core": "2.0.3",
     "dataform-scd": "https://github.com/dataform-co/dataform-scd/archive/0.3.tar.gz"
   }
 }
 ```

Import a package function or constant to a JavaScript file in Dataform

To use a function or a constant from a package inside a JavaScript file in Dataform, you need to first import it to the file.

To import a function or a constant from a package to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .js file in which you want to use the package.
In the file, import a function or a constant in the following format:
```
const { EXPORT-NAME } = require("PACKAGE-NAME");
```
1. Replace EXPORT-NAME with the name of the function or constant that you want to use, declared in module.exports in the package index.js file.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the postoffice package imported and used in a JavaScript file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

const { getDomain } = require("postoffice");
getDomain();

Import a whole package to a JavaScript file in Dataform

To import the whole package to a JavaScript file instead of importing selected functions or constants to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .js file in which you want to use the package.
In the file, import the package in the following format:
```
const CONSTANT-NAME = require("PACKAGE-NAME");
```
1. Replace CONSTANT-NAME with a name for the constant.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the imported postoffice package used in a JavaScript file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

const postoffice = require("postoffice");
postoffice.getDomain();

Import a package function or constant to a SQLX file in Dataform

To use a function or a constant from a package inside a SQLX file, you need to first import it to the file.

To import a function or a constant from a package to a SQLX file, follow these steps:

In your workspace, in the Files pane, select a .sqlx file in which you want to use the package.
In the file, enter the following js block:
```
js {
  const { EXPORT-NAME } = require("PACKAGE-NAME");
}
```
1. Replace EXPORT-NAME with the name of the function or constant that you want to use, declared in module.exports in the package index.js file.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the getDomain function from the postoffice package imported in a js block and used in a SELECT statement in a SQLX file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

config {
    type: "table",
}

js {
  const { getDomain } = require("postoffice");
}

SELECT ${getDomain("email")} as test

Import a whole package to a SQLX file in Dataform

To import the whole package to a SQLX file instead of importing selected functions or constants to a JavaScript file, follow these steps:

In your workspace, in the Files pane, select a .sqlx file in which you want to use the package.
In the file, import the package in the following format:
```
js {
  const CONSTANT-NAME = require("PACKAGE-NAME");
}
```
1. Replace CONSTANT-NAME with a name for the constant.
2. Replace PACKAGE-NAME with the name of the package that you want to use.
Commit and push your changes.

The following code sample shows the postoffice package imported in a js block and its getDomain function used in a SELECT statement in a SQLX file:

/*
 * Contents of postoffice index.js:
 * module.exports = { getDomain };
 */

config {
    type: "table",
}

js {
  const postoffice = require("postoffice");
}

SELECT ${

Use JavaScript packages Stay organized with collections Save and categorize content based on your preferences.