Databricks Utilities (dbutils) reference

This page contains reference for Databricks Utilities (dbutils). The utilities provide modules with commands that enable you to work with your Databricks environment from notebooks. For example, you can manage files and object storage, and work with secrets. dbutils are available in Python, R, and Scala notebooks.

note

dbutils only supports compute environments that use DBFS.

Utility modules

The following table lists the Databricks Utilities modules, which you can retrieve using dbutils.help().

Module	Description
credentials	Utilities for interacting with credentials within notebooks
data	Utilities for understanding and interacting with datasets (EXPERIMENTAL)
fs	Utilities for accessing the Databricks file system (DBFS)
jobs	Utilities for leveraging job features
library	Deprecated. Utilities for managing session-scoped libraries
meta	Utilities to hook into the compiler (EXPERIMENTAL)
notebook	Utilities for managing the control flow of notebooks (EXPERIMENTAL)
preview	Utilities in preview
secrets	Utilities for leveraging secrets within notebooks
widgets	Utilities for parameterizing notebooks.
api	Utilities for managing application builds

Module	Description
credentials	Utilities for interacting with credentials within notebooks
data	Utilities for understanding and interacting with datasets (EXPERIMENTAL)
fs	Utilities for accessing the Databricks file system (DBFS)
jobs	Utilities for leveraging job features
library	Deprecated. Utilities for managing session-scoped libraries
meta	Utilities to hook into the compiler (EXPERIMENTAL)
notebook	Utilities for managing the control flow of notebooks (EXPERIMENTAL)
preview	Utilities in preview
secrets	Utilities for leveraging secrets within notebooks
widgets	Utilities for parameterizing notebooks.
api	Utilities for managing application builds

Command help

To list commands for a utility module along with a short description of each command, append .help() after the name of the utility module. The following example lists available commands for the notebook utility:

dbutils.notebook.help()

Output

The notebook module.

exit(value: String): void -> This method lets you exit a notebook with a value
run(path: String, timeoutSeconds: int, arguments: Map): String -> This method runs a notebook and returns its exit value

To output help for a command, run dbutils.<utility-name>.help("<command-name>"). The following example displays help for the file system utilities copy command, dbutils.fs.cp:

dbutils.fs.help("cp")

Output

/**
* Copies a file or directory, possibly across FileSystems.
*
* Example: cp("/mnt/my-folder/a", "dbfs:/a/b")
*
* @param from FileSystem URI of the source file or directory
* @param to FileSystem URI of the destination file or directory
* @param recurse if true, all files and directories will be recursively copied
* @return true if all files were successfully copied
*/
cp(from: java.lang.String, to: java.lang.String, recurse: boolean = false): boolean

Credentials utility (dbutils.credentials)

The credentials utility module contains commands to interact with credentials within notebooks. This utility is usable only on clusters with credential passthrough enabled.

The following table lists the available commands for this utility, which you can retrieve using dbutils.credentials.help().

Command	Description
assumeRole	Sets the role ARN to assume when looking for credentials to authenticate with S3.
getServiceCredentialsProvider	Returns a service credentials provider for the given service credential.
showCurrentRole	Shows the currently set role.
showRoles	Shows the set of possible assumed roles.

Command	Description
assumeRole	Sets the role ARN to assume when looking for credentials to authenticate with S3.
getServiceCredentialsProvider	Returns a service credentials provider for the given service credential.
showCurrentRole	Shows the currently set role.
showRoles	Shows the set of possible assumed roles.

assumeRole command (dbutils.credentials.assumeRole)

assumeRole(role: String): boolean

Sets the Amazon Resource Name (ARN) for the AWS Identity and Access Management (IAM) role to assume when looking for credentials to authenticate with Amazon S3. After you run this command, you can run S3 access commands, such as sc.textFile("s3a://my-bucket/my-file.csv") to access an object.

To display complete help for this command, run:

dbutils.credentials.help("assumeRole")

Example

Python

Python

dbutils.credentials.assumeRole("arn:aws:iam::123456789012:roles/my-role")

# Out[1]: True

R

R

dbutils.credentials.assumeRole("arn:aws:iam::123456789012:roles/my-role")

# TRUE

Scala

Scala

dbutils.credentials.assumeRole("arn:aws:iam::123456789012:roles/my-role")

// res0: Boolean = true

getServiceCredentialsProvider command (dbutils.credentials.getServiceCredentialsProvider)

getServiceCredentialsProvider(credentialName: String): Object

Returns a service credentials provider for the given service credential. The return object type is specific to the cloud provider.

To display complete help for this command, run:

dbutils.credentials.help("getServiceCredentialsProvider")

Example

Python

Python

dbutils.credentials.getServiceCredentialsProvider("my-credential")

R

This utility is not supported in R.

Scala

Scala

dbutils.credentials.getServiceCredentialsProvider("my-credential")

showCurrentRole command (dbutils.credentials.showCurrentRole)

showCurrentRole: List

Lists the currently set AWS Identity and Access Management (IAM) role.

To display complete help for this command, run:

dbutils.credentials.help("showCurrentRole")

Example

Python

Python

dbutils.credentials.showCurrentRole()

# Out[1]: ['arn:aws:iam::123456789012:role/my-role-a']

R

R

dbutils.credentials.showCurrentRole()

# [[1]]
# [1] "arn:aws:iam::123456789012:role/my-role-a"

Scala

Scala

dbutils.credentials.showCurrentRole()

// res0: java.util.List[String] = [arn:aws:iam::123456789012:role/my-role-a]

showRoles command (dbutils.credentials.showRoles)

showRoles: List

Lists the set of possible assumed AWS Identity and Access Management (IAM) roles.

To display complete help for this command, run:

dbutils.credentials.help("showRoles")

Example

Python

Python

dbutils.credentials.showRoles()

# Out[1]: ['arn:aws:iam::123456789012:role/my-role-a', 'arn:aws:iam::123456789012:role/my-role-b']

R

R

dbutils.credentials.showRoles()

# [[1]]
# [1] "arn:aws:iam::123456789012:role/my-role-a"
#
# [[2]]
# [1] "arn:aws:iam::123456789012:role/my-role-b"

Scala

Scala

dbutils.credentials.showRoles()

// res0: java.util.List[String] = [arn:aws:iam::123456789012:role/my-role-a, arn:aws:iam::123456789012:role/my-role-b]

Data utility (dbutils.data)

Preview

This feature is in Public Preview.

note

Available in Databricks Runtime 9.0 and above.

The data utility module contains commands to understand and interact with datasets.

The following table lists the available commands for this utility, which you can retrieve using dbutils.data.help().

Command	Description
summarize	Summarize a Spark DataFrame and visualize the statistics to get quick insights

Command	Description
summarize	Summarize a Spark DataFrame and visualize the statistics to get quick insights

summarize command (dbutils.data.summarize)

note

This feature is in Public Preview.

summarize(df: Object, precise: boolean): void

Calculates and displays summary statistics of an Apache Spark DataFrame or pandas DataFrame. This command is available for Python, Scala and R.

important

This command analyzes the complete contents of the DataFrame. Running this command for very large DataFrames can be very expensive.

To display complete help for this command, run:

dbutils.data.help("summarize")

In Databricks Runtime 10.4 LTS and above, you can use the additional precise parameter to adjust the precision of the computed statistics.

• When precise is set to false (the default), some returned statistics include approximations to reduce run time.
  • The number of distinct values for categorical columns may have ~5% relative error for high-cardinality columns.
  • The frequent value counts may have an error of up to 0.01% when the number of distinct values is greater than 10000.
  • The histograms and percentile estimates may have an error of up to 0.01% relative to the total number of rows.
• When precise is set to true, the statistics are computed with higher precision. All statistics except for the histograms and percentiles for numeric columns are now exact.
  • The histograms and percentile estimates may have an error of up to 0.0001% relative to the total number of rows.

The tooltip at the top of the data summary output indicates the mode of the current run.

Example

This example displays summary statistics for an Apache Spark DataFrame with approximations enabled by default. To see the results, run this command in a notebook. This example is based on Sample datasets.

Python

Python

df = spark.read.format('csv').load(
  '/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv',
  header=True,
  inferSchema=True
)
dbutils.data.summarize(df)

R

R

df <- read.df("/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv", source = "csv", header="true", inferSchema = "true")
dbutils.data.summarize(df)

Scala

Scala

val df = spark.read.format("csv")
  .option("inferSchema", "true")
  .option("header", "true")
  .load("/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv")
dbutils.data.summarize(df)

The visualization uses SI notation to concisely render numerical values smaller than 0.01 or larger than 10000. As an example, the numerical value 1.25e-15 will be rendered as 1.25f. One exception: the visualization uses “B” for 1.0e9 (giga) instead of “G”.

File system utility (dbutils.fs)

The file system utility module contains commands to access What is DBFS?. To access workspace files, use shell commands such as %sh ls, as there are some limitations when using dbutils.fs commands with workspace files.

warning

The Python implementation of all dbutils.fs methods uses snake_case rather than camelCase for keyword formatting.

For example, dbutils.fs.help() displays the option extraConfigs for dbutils.fs.mount(). However, in Python you would use the keyword extra_configs.

The following table lists the available commands for this utility, which you can retrieve using dbutils.fs.help().

Command	Description
cp	Copies a file or directory, possibly across FileSystems
head	Returns up to the first 'max_bytes' bytes of the given file as a String encoded in UTF-8
ls	Lists the contents of a directory
mkdirs	Creates the given directory if it does not exist, also creating any necessary parent directories
mount	Mounts the given source directory into DBFS at the given mount point
mounts	Displays information about what is mounted within DBFS
mv	Moves a file or directory, possibly across FileSystems
put	Writes the given String out to a file, encoded in UTF-8
refreshMounts	Forces all machines in this cluster to refresh their mount cache, ensuring they receive the most recent information
rm	Removes a file or directory
unmount	Deletes a DBFS mount point
updateMount	Similar to mount(), but updates an existing mount point instead of creating a new one

Command	Description
cp	Copies a file or directory, possibly across FileSystems
head	Returns up to the first 'max_bytes' bytes of the given file as a String encoded in UTF-8
ls	Lists the contents of a directory
mkdirs	Creates the given directory if it does not exist, also creating any necessary parent directories
mount	Mounts the given source directory into DBFS at the given mount point
mounts	Displays information about what is mounted within DBFS
mv	Moves a file or directory, possibly across FileSystems
put	Writes the given String out to a file, encoded in UTF-8
refreshMounts	Forces all machines in this cluster to refresh their mount cache, ensuring they receive the most recent information
rm	Removes a file or directory
unmount	Deletes a DBFS mount point
updateMount	Similar to mount(), but updates an existing mount point instead of creating a new one

tip

In notebooks, you can use the %fs magic command to access DBFS. For example, %fs ls /Volumes/main/default/my-volume/ is the same as dbutils.fs.ls("/Volumes/main/default/my-volume/"). See magic commands.

cp command (dbutils.fs.cp)

cp(from: String, to: String, recurse: boolean = false): boolean

Copies a file or directory, possibly across filesystems.

To display complete help for this command, run:

dbutils.fs.help("cp")

Example

This example copies the file named data.csv from /Volumes/main/default/my-volume/ to new-data.csv in the same volume.

Python

Python

dbutils.fs.cp("/Volumes/main/default/my-volume/data.csv", "/Volumes/main/default/my-volume/new-data.csv")

# Out[4]: True

R

R

dbutils.fs.cp("/Volumes/main/default/my-volume/data.csv", "/Volumes/main/default/my-volume/new-data.csv")

# [1] TRUE

Scala

Scala

dbutils.fs.cp("/Volumes/main/default/my-volume/data.csv", "/Volumes/main/default/my-volume/new-data.csv")

// res3: Boolean = true

head command (dbutils.fs.head)

head(file: String, max_bytes: int = 65536): String

Returns up to the specified maximum number of bytes in the given file. The bytes are returned as a UTF-8 encoded string.

To display complete help for this command, run:

dbutils.fs.help("head")

Example

This example displays the first 25 bytes of the file data.csv located in /Volumes/main/default/my-volume/.

Python

Python

dbutils.fs.head("/Volumes/main/default/my-volume/data.csv", 25)

# [Truncated to first 25 bytes]
# Out[12]: 'Year,First Name,County,Se'

R

R

dbutils.fs.head("/Volumes/main/default/my-volume/data.csv", 25)

# [1] "Year,First Name,County,Se"

Scala

Scala

dbutils.fs.head("/Volumes/main/default/my-volume/data.csv", 25)

// [Truncated to first 25 bytes]
// res4: String =
// "Year,First Name,County,Se"

ls command (dbutils.fs.ls)

ls(dir: String): Seq

Lists the contents of a directory.

To display complete help for this command, run:

dbutils.fs.help("ls")

Example

This example displays information about the contents of /Volumes/main/default/my-volume/. The modificationTime field is available in Databricks Runtime 10.4 LTS and above. In R, modificationTime is returned as a string.

Python

Python

dbutils.fs.ls("/Volumes/main/default/my-volume/")

# Out[13]: [FileInfo(path='dbfs:/Volumes/main/default/my-volume/data.csv', name='data.csv', size=2258987, modificationTime=1711357839000)]

R

R

dbutils.fs.ls("/Volumes/main/default/my-volume/")

# For prettier results from dbutils.fs.ls(<dir>), please use `%fs ls <dir>`

# [[1]]
# [[1]]$path
# [1] "/Volumes/main/default/my-volume/data.csv"

# [[1]]$name
# [1] "data.csv"

# [[1]]$size
# [1] 2258987

# [[1]]$isDir
# [1] FALSE

# [[1]]$isFile
# [1] TRUE

# [[1]]$modificationTime
# [1] "1711357839000"

Scala

Scala

dbutils.fs.ls("/tmp")

// res6: Seq[com.databricks.backend.daemon.dbutils.FileInfo] = WrappedArray(FileInfo(/Volumes/main/default/my-volume/data.csv, 2258987, 1711357839000))

mkdirs command (dbutils.fs.mkdirs)

mkdirs(dir: String): boolean

Creates the given directory if it does not exist. Also creates any necessary parent directories.

To display complete help for this command, run:

dbutils.fs.help("mkdirs")

Example

This example creates the directory my-data within /Volumes/main/default/my-volume/.

Python

Python

dbutils.fs.mkdirs("/Volumes/main/default/my-volume/my-data")

# Out[15]: True

R

R

dbutils.fs.mkdirs("/Volumes/main/default/my-volume/my-data")

# [1] TRUE

Scala

Scala

dbutils.fs.mkdirs("/Volumes/main/default/my-volume/my-data")

// res7: Boolean = true

mount command (dbutils.fs.mount)

mount(source: String, mountPoint: String, encryptionType: String = "", owner: String = null, extraConfigs: Map = Map.empty[String, String]): boolean

Mounts the specified source directory into DBFS at the specified mount point.

Serverless compatibility

Databricks recommends moving away from dbutils.fs.mount as it is not compatible with Databricks serverless compute architecture. Use Unity Catalog external location with external volumes instead.

To display complete help for this command, run:

dbutils.fs.help("mount")

Example

Python

Python

aws_bucket_name = "my-bucket"
mount_name = "s3-my-bucket"

dbutils.fs.mount("s3a://%s" % aws_bucket_name, "/mnt/%s" % mount_name)

Scala

Scala

val AwsBucketName = "my-bucket"
val MountName = "s3-my-bucket"

dbutils.fs.mount(s"s3a://$AwsBucketName", s"/mnt/$MountName")

For additional code examples, see Connect to Amazon S3.

mounts command (dbutils.fs.mounts)

mounts: Seq

Displays information about what is currently mounted within DBFS.

Serverless compatibility

Databricks recommends moving away from dbutils.fs.mounts as it is not compatible with Databricks serverless compute architecture.

To display complete help for this command, run:

dbutils.fs.help("mounts")

Example

warning

Call dbutils.fs.refreshMounts() on all other running clusters to propagate the new mount. See refreshMounts command (dbutils.fs.refreshMounts).

Python

Python

dbutils.fs.mounts()

# Out[11]: [MountInfo(mountPoint='/mnt/databricks-results', source='databricks-results', encryptionType='sse-s3')]

Scala

Scala