Skip to main content

Job

View as Markdown

GetJobResponse object

Job was retrieved successfully.

next_page_tokenstring

A token that can be used to list the next page of array properties.

Example: CAAos-uriYcxMN7_rt_v7B4=

job_idint64

The canonical identifier for this job.

Example: 11223344

creator_user_namestring

The creator user name. This field won’t be included in the response if the user has already been deleted.

Example: user.name@databricks.com

run_as_user_namestring

The email of an active workspace user or the application ID of a service principal that the job runs as. This value can be changed by setting the run_as field when creating or updating a job.

By default, run_as_user_name is based on the current job settings and is set to the creator of the job if job access control is disabled or to the user with the is_owner permission if job access control is enabled.

Example: user.name@databricks.com

settingsobject

Settings for this job and all of its runs. These settings can be updated using the resetJob method.

Show child attributesHide child attributes
namestring<= 4096 characters

An optional name for the job. The maximum length is 4096 bytes in UTF-8 encoding.

Default: Untitled

Example: A multitask job

descriptionstring<= 27700 characters

An optional description for the job. The maximum length is 27700 characters in UTF-8 encoding.

Example: This job contain multiple tasks that are required to produce the weekly shark sightings report.

email_notificationsobject

An optional set of email addresses that is notified when runs of this job begin or complete as well as when this job is deleted.

Show child attributesHide child attributes
on_startarray of string

A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_successarray of string

A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_failurearray of string

A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string

A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.

Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview

A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.

Example: user.name@databricks.com

on_maintenance_startarray of stringGA

A list of email addresses to notify when platform-initiated maintenance starts for a continuous job.

Example: user.name@databricks.com

on_maintenance_completearray of stringGA

A list of email addresses to notify when platform-initiated maintenance completes for a continuous job.

Example: user.name@databricks.com

no_alert_for_skipped_runsbooleanDeprecated

If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.

Default: false

Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this job begin or complete.

Show child attributesHide child attributes
on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes
idstring
on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes
idstring
on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes
idstring
on_duration_warning_threshold_exceededarray of object

An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

Show child attributesHide child attributes
idstring
on_streaming_backlog_exceededarray of objectPublic Preview

An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

Show child attributesHide child attributes
idstring
on_maintenance_startarray of objectGA

An optional list of system notification IDs to call when platform-initiated maintenance starts for a continuous job. A maximum of 3 destinations can be specified for the on_maintenance_start property.

Show child attributesHide child attributes
idstring
on_maintenance_completearray of objectGA

An optional list of system notification IDs to call when platform-initiated maintenance completes for a continuous job. A maximum of 3 destinations can be specified for the on_maintenance_complete property.

Show child attributesHide child attributes
idstring
notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this job.

Show child attributesHide child attributes
no_alert_for_skipped_runsboolean

If true, do not send notifications to recipients specified in on_failure if the run is skipped.

Default: false

Example: false

no_alert_for_canceled_runsboolean

If true, do not send notifications to recipients specified in on_failure if the run is canceled.

Default: false

Example: false

alert_on_last_attemptboolean

If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.

Default: false

Example: false

timeout_secondsint32

An optional timeout applied to each run of this job. A value of 0 means no timeout.

Default: 0

Example: 86400

healthobject
Show child attributesHide child attributes
rulesarray of object
Show child attributesHide child attributes
metricstring

Specifies the health metric that is being evaluated for a particular health rule.

  • RUN_DURATION_SECONDS: Expected total time for a run in seconds.
  • STREAMING_BACKLOG_BYTES: An estimate of the maximum bytes of data waiting to be consumed across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_RECORDS: An estimate of the maximum offset lag across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_SECONDS: An estimate of the maximum consumer delay across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_FILES: An estimate of the maximum number of outstanding files across all streams. This metric is in Public Preview.

Values:

  • RUN_DURATION_SECONDS
  • STREAMING_BACKLOG_BYTES
  • STREAMING_BACKLOG_RECORDS
  • STREAMING_BACKLOG_SECONDS
  • STREAMING_BACKLOG_FILES

Example: RUN_DURATION_SECONDS

opstring

Specifies the operator used to compare the health metric value with the specified threshold.

Values:

  • GREATER_THAN

Example: GREATER_THAN

valueint64

Specifies the threshold value that the health metric should obey to satisfy the health rule.

Example: 10

scheduleobject

An optional periodic schedule for this job. The default behavior is that the job only runs when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes
quartz_cron_expressionstring

A Cron expression using Quartz syntax that describes the schedule for a job. See Cron Trigger for details. This field is required.

Example: 20 30 * * * ?

timezone_idstring

A Java timezone ID. The schedule for a job is resolved with respect to this timezone. See Java TimeZone for details. This field is required.

Example: Europe/London

pause_statusstring

Indicate whether this schedule is paused or not.

Default: UNPAUSED

Values:

  • UNPAUSED
  • PAUSED
triggerobject

A configuration to trigger a run when certain conditions are met. The default behavior is that the job runs only when triggered by clicking “Run Now” in the Jobs UI or sending an API request to runNow.

Show child attributesHide child attributes
pause_statusstring

Whether this trigger is paused or not.

Default: UNPAUSED

Values:

  • UNPAUSED
  • PAUSED
file_arrivalobject

File arrival trigger settings.

Show child attributesHide child attributes
urlstring

URL to be monitored for file arrivals. The path must point to the root or a subpath of the external location.

min_time_between_triggers_secondsint32

If set, the trigger starts a run only after the specified amount of time passed since the last time the trigger fired. The minimum allowed value is 60 seconds

wait_after_last_change_secondsint32

If set, the trigger starts a run only after no file activity has occurred for the specified amount of time. This makes it possible to wait for a batch of incoming files to arrive before triggering a run. The minimum allowed value is 60 seconds.

periodicobject

Periodic trigger settings.

Show child attributesHide child attributes
intervalint32

The interval at which the trigger should run.

unitstring

The unit of time for the interval.

Values:

  • TIME_UNIT_UNSPECIFIED
  • HOURS
  • DAYS
  • WEEKS
  • MINUTES
table_updateobject
Show child attributesHide child attributes
table_namesarray of string

A list of tables to monitor for changes. The table name must be in the format catalog_name.schema_name.table_name.

min_time_between_triggers_secondsint32

If set, the trigger starts a run only after the specified amount of time has passed since the last time the trigger fired. The minimum allowed value is 60 seconds.

wait_after_last_change_secondsint32

If set, the trigger starts a run only after no table updates have occurred for the specified time and can be used to wait for a series of table updates before triggering a run. The minimum allowed value is 60 seconds.

conditionstring

The table(s) condition based on which to trigger a job run.

Values:

  • ANY_UPDATED
  • ALL_UPDATED

Example: ALL_UPDATED

continuousobject

An optional continuous property for this job. The continuous property will ensure that there is always one run executing. Only one of schedule and continuous can be used.

Pipelines started by a continuous job also run continuously, regardless of their own pipeline mode setting.

Show child attributesHide child attributes
pause_statusstring

Indicate whether the continuous execution of the job is paused or not. Defaults to UNPAUSED.

Default: UNPAUSED

Values:

  • UNPAUSED
  • PAUSED
task_retry_modestringGA

Indicate whether the continuous job is applying task level retries or not. Defaults to NEVER.

Values:

  • NEVER
  • ON_FAILURE
maintenance_windowobjectGA

Defines when platform-initiated maintenance may run for this job. If unspecified, maintenance may run at any time.

Show child attributesHide child attributes
start_hourint32

An integer between 0 and 23 denoting the start hour for the maintenance window in the 24-hour day. Platform-initiated maintenance is triggered only within a one-hour window starting at this hour. This field is required.

day_of_weekstring

The day of week on which maintenance is allowed to happen. This field is required.

Values:

  • DAY_OF_WEEK_UNSPECIFIED
  • MONDAY
  • TUESDAY
  • WEDNESDAY
  • THURSDAY
  • FRIDAY
  • SATURDAY
  • SUNDAY
timezone_idstring

A Java timezone ID. The maintenance window is resolved with respect to this timezone. See Java TimeZone for details. This field is required.

Example: Europe/London

max_concurrent_runsint32

An optional maximum allowed number of concurrent runs of the job. Set this value if you want to be able to execute multiple runs of the same job concurrently. This is useful for example if you trigger your job on a frequent schedule and want to allow consecutive runs to overlap with each other, or if you want to trigger multiple runs which differ by their input parameters. This setting affects only new runs. For example, suppose the job’s concurrency is 4 and there are 4 concurrent active runs. Then setting the concurrency to 3 won’t kill any of the active runs. However, from then on, new runs are skipped unless there are fewer than 3 active runs. This value cannot exceed 1000. Setting this value to 0 causes all new runs to be skipped.

Default: 1

Example: 10

tasksarray of object

A list of task specifications to be executed by this job. It supports up to 1000 elements in write endpoints (jobs/create, jobs/reset, jobs/update, jobs/submit). Read endpoints return only 100 tasks. If more than 100 tasks are available, you can paginate through them using jobs/get. Use the next_page_token field at the object root to determine if more results are available.

Show child attributesHide child attributes
task_keystring[ 1 .. 100 ] characters^[\\w\\-\\_]+$

A unique name for the task. This field is used to refer to this task from other tasks. This field is required and must be unique within its parent job. On Update or Reset, this field is used to reference the tasks to be updated or reset.

Example: Task_Key

depends_onarray of object

An optional array of objects specifying the dependency graph of the task. All tasks specified in this field must complete before executing this task. The task will run only if the run_if condition is true. The key is task_key, and the value is the name assigned to the dependent task.

Show child attributesHide child attributes
task_keystring[ 1 .. 100 ] characters^[\\w\\-\\_]+$

The name of the task this task depends on.

Example: Task_Key

outcomestring

Can only be specified on condition task dependencies. The outcome of the dependent task that must be met for this task to run.

run_ifstring

An optional value specifying the condition determining whether the task is run once its dependencies have been completed.

  • ALL_SUCCESS: All dependencies have executed and succeeded
  • AT_LEAST_ONE_SUCCESS: At least one dependency has succeeded
  • NONE_FAILED: None of the dependencies have failed and at least one was executed
  • ALL_DONE: All dependencies have been completed
  • AT_LEAST_ONE_FAILED: At least one dependency failed
  • ALL_FAILED: ALl dependencies have failed

Default: ALL_SUCCESS

Values:

  • ALL_SUCCESS
  • ALL_DONE
  • NONE_FAILED
  • AT_LEAST_ONE_SUCCESS
  • ALL_FAILED
  • AT_LEAST_ONE_FAILED

Example: ALL_SUCCESS

timeout_secondsint32

An optional timeout applied to each run of this job task. A value of 0 means no timeout.

Default: 0

Example: 86400

healthobject
Show child attributesHide child attributes
rulesarray of object
Show child attributesHide child attributes
metricstring

Specifies the health metric that is being evaluated for a particular health rule.

  • RUN_DURATION_SECONDS: Expected total time for a run in seconds.
  • STREAMING_BACKLOG_BYTES: An estimate of the maximum bytes of data waiting to be consumed across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_RECORDS: An estimate of the maximum offset lag across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_SECONDS: An estimate of the maximum consumer delay across all streams. This metric is in Public Preview.
  • STREAMING_BACKLOG_FILES: An estimate of the maximum number of outstanding files across all streams. This metric is in Public Preview.

Values:

  • RUN_DURATION_SECONDS
  • STREAMING_BACKLOG_BYTES
  • STREAMING_BACKLOG_RECORDS
  • STREAMING_BACKLOG_SECONDS
  • STREAMING_BACKLOG_FILES

Example: RUN_DURATION_SECONDS

opstring

Specifies the operator used to compare the health metric value with the specified threshold.

Values:

  • GREATER_THAN

Example: GREATER_THAN

valueint64

Specifies the threshold value that the health metric should obey to satisfy the health rule.

Example: 10

email_notificationsobject

An optional set of email addresses that is notified when runs of this task begin or complete as well as when this task is deleted. The default behavior is to not send any emails.

Show child attributesHide child attributes
on_startarray of string

A list of email addresses to be notified when a run begins. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_successarray of string

A list of email addresses to be notified when a run successfully completes. A run is considered to have completed successfully if it ends with a TERMINATED life_cycle_state and a SUCCESS result_state. If not specified on job creation, reset, or update, the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_failurearray of string

A list of email addresses to be notified when a run unsuccessfully completes. A run is considered to have completed unsuccessfully if it ends with an INTERNAL_ERROR life_cycle_state or a FAILED, or TIMED_OUT result_state. If this is not specified on job creation, reset, or update the list is empty, and notifications are not sent.

Example: user.name@databricks.com

on_duration_warning_threshold_exceededarray of string

A list of email addresses to be notified when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. If no rule for the RUN_DURATION_SECONDS metric is specified in the health field for the job, notifications are not sent.

Example: user.name@databricks.com

on_streaming_backlog_exceededarray of stringPublic Preview

A list of email addresses to notify when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes.

Example: user.name@databricks.com

on_maintenance_startarray of stringGA

A list of email addresses to notify when platform-initiated maintenance starts for a continuous job.

Example: user.name@databricks.com

on_maintenance_completearray of stringGA

A list of email addresses to notify when platform-initiated maintenance completes for a continuous job.

Example: user.name@databricks.com

no_alert_for_skipped_runsbooleanDeprecated

If true, do not send email to recipients specified in on_failure if the run is skipped. This field is deprecated. Please use the notification_settings.no_alert_for_skipped_runs field.

Default: false

Example: false

notification_settingsobject

Optional notification settings that are used when sending notifications to each of the email_notifications and webhook_notifications for this task.

Show child attributesHide child attributes
no_alert_for_skipped_runsboolean

If true, do not send notifications to recipients specified in on_failure if the run is skipped.

Default: false

Example: false

no_alert_for_canceled_runsboolean

If true, do not send notifications to recipients specified in on_failure if the run is canceled.

Default: false

Example: false

alert_on_last_attemptboolean

If true, do not send notifications to recipients specified in on_start for the retried runs and do not send notifications to recipients specified in on_failure until the last retry of the run.

Default: false

Example: false

webhook_notificationsobject

A collection of system notification IDs to notify when runs of this task begin or complete. The default behavior is to not send any system notifications.

Show child attributesHide child attributes
on_startarray of object

An optional list of system notification IDs to call when the run starts. A maximum of 3 destinations can be specified for the on_start property.

Show child attributesHide child attributes
idstring
on_successarray of object

An optional list of system notification IDs to call when the run completes successfully. A maximum of 3 destinations can be specified for the on_success property.

Show child attributesHide child attributes
idstring
on_failurearray of object

An optional list of system notification IDs to call when the run fails. A maximum of 3 destinations can be specified for the on_failure property.

Show child attributesHide child attributes
idstring
on_duration_warning_threshold_exceededarray of object

An optional list of system notification IDs to call when the duration of a run exceeds the threshold specified for the RUN_DURATION_SECONDS metric in the health field. A maximum of 3 destinations can be specified for the on_duration_warning_threshold_exceeded property.

Show child attributesHide child attributes
idstring
on_streaming_backlog_exceededarray of objectPublic Preview

An optional list of system notification IDs to call when any streaming backlog thresholds are exceeded for any stream. Streaming backlog thresholds can be set in the health field using the following metrics: STREAMING_BACKLOG_BYTES, STREAMING_BACKLOG_RECORDS, STREAMING_BACKLOG_SECONDS, or STREAMING_BACKLOG_FILES. Alerting is based on the 10-minute average of these metrics. If the issue persists, notifications are resent every 30 minutes. A maximum of 3 destinations can be specified for the on_streaming_backlog_exceeded property.

Show child attributesHide child attributes
idstring
on_maintenance_startarray of objectGA

An optional list of system notification IDs to call when platform-initiated maintenance starts for a continuous job. A maximum of 3 destinations can be specified for the on_maintenance_start property.

Show child attributesHide child attributes
idstring
on_maintenance_completearray of objectGA

An optional list of system notification IDs to call when platform-initiated maintenance completes for a continuous job. A maximum of 3 destinations can be specified for the on_maintenance_complete property.

Show child attributesHide child attributes
idstring
descriptionstring<= 1000 characters

An optional description for this task.

Example: This is the description for this task.

environment_keystring[ 1 .. 100 ] characters^[\\w\\-\\_]+$

The key that references an environment spec in a job. This field is required for Python script, Python wheel and dbt tasks when using serverless compute or a compute resource that uses Environments mode.

disabledbooleanGA

An optional flag to disable the task. If set to true, the task will not run even if it is part of a job.

Default: false

computeobjectPublic Preview

Task level compute configuration.

Show child attributesHide child attributes
hardware_acceleratorstringPublic Preview

Hardware accelerator configuration for Serverless GPU workloads.

Values:

  • GPU_1xA10
  • GPU_8xH100
  • GPU_1xH100
  • GPU_8xB300
notebook_taskobject

The task runs a notebook when the notebook_task field is present.

Show child attributesHide child attributes
notebook_pathstring

The path of the notebook to be run in the <Databricks> workspace or remote repository. For notebooks stored in the <Databricks> workspace, the path must be absolute and begin with a slash. For notebooks stored in a remote repository, the path must be relative. This field is required.

Example: /Users/user.name@databricks.com/notebook_to_run

base_parametersobject

Base parameters to be used for each run of this job. If the run is initiated by a call to jobs/run Now with parameters specified, the two parameters maps are merged. If the same key is specified in base_parameters and in run-now, the value from run-now is used. Use Task parameter variables to set parameters containing information about job runs.

If the notebook takes a parameter that is not specified in the job’s base_parameters or the run-now override parameters, the default value from the notebook is used.

Retrieve these parameters in a notebook using dbutils.widgets.get.

The JSON representation of this field cannot exceed 1MB.

sourcestring

Optional location type of the notebook. When set to WORKSPACE, the notebook will be retrieved from the local <Databricks> workspace. When set to GIT, the notebook will be retrieved from a Git repository defined in git_source. If the value is empty, the task will use GIT if git_source is defined and WORKSPACE otherwise.

  • WORKSPACE: Notebook is located in <Databricks> workspace.
  • GIT: Notebook is located in cloud Git provider.

Values:

  • WORKSPACE
  • GIT

Example: WORKSPACE

warehouse_idstring

Optional warehouse_id to run the notebook on a SQL warehouse. Classic SQL warehouses are NOT supported, please use serverless or pro SQL warehouses.

Note that SQL warehouses only support SQL cells; if the notebook contains non-SQL cells, the run will fail.

Example: ab12cd34efgh567i

spark_jar_taskobject

The task runs a JAR when the spark_jar_task field is present.

Show child attributesHide child attributes
jar_uristringDeprecated

Deprecated since 04/2016. For classic compute, provide a jar through the libraries field instead. For serverless compute, provide a jar though the java_dependencies field inside the environments list.

See the examples of classic and serverless compute usage at the top of the page.

main_class_namestring

The full name of the class containing the main method to be executed. This class must be contained in a JAR provided as a library.

The code must use SparkContext.getOrCreate to obtain a Spark context; otherwise, runs of the job fail.

Example: com.databricks.ComputeModels

parametersarray of string

Parameters passed to the main method.

Use Task parameter variables to set parameters containing information about job runs.

run_as_replbooleanDeprecated

Deprecated. A value of false is no longer supported.

Default: true

spark_python_taskobject

The task runs a Python file when the spark_python_task field is present.

Show child attributesHide child attributes
python_filestring

The Python file to be executed. Cloud file URIs (such as dbfs:/, s3:/, adls:/, gcs:/) and workspace paths are supported. For python files stored in the <Databricks> workspace, the path must be absolute and begin with /. For files stored in a remote repository, the path must be relative. This field is required.

Example: dbfs:/path/to/file.py

parametersarray of string

Command line parameters passed to the Python file.

Use Task parameter variables to set parameters containing information about job runs.

sourcestring

Optional location type of the Python file. When set to WORKSPACE or not specified, the file will be retrieved from the local <Databricks> workspace or cloud location (if the python_file has a URI format). When set to GIT, the Python file will be retrieved from a Git repository defined in git_source.

  • WORKSPACE: The Python file is located in a <Databricks> workspace or at a cloud filesystem URI.
  • GIT: The Python file is located in a remote Git repository.

Values:

  • WORKSPACE
  • GIT

Example: WORKSPACE

spark_submit_taskobjectDeprecated

(Legacy) The task runs the spark-submit script when the spark_submit_task field is present. Databricks recommends using the spark_jar_task instead; see Spark Submit task for jobs.

Show child attributesHide child attributes
parametersarray of string

Command-line parameters passed to spark submit.

Use Task parameter variables to set parameters containing information about job runs.

pipeline_taskobject

The task triggers a pipeline update when the pipeline_task field is present. Only pipelines configured to use triggered more are supported.

Show child attributesHide child attributes
pipeline_idstring

The full name of the pipeline task to execute.

Example: a12cd3e4-0ab1-1abc-1a2b-1a2bcd3e4fg5

parametersobjectBeta

Key/value-map of parameters passed to the pipeline execution. Limited to 10k characters in total.

full_refreshboolean

If true, triggers a full refresh on the spark declarative pipeline.

Default: false

refresh_selectionarray of stringBeta

A list of tables to update without fullRefresh.

full_refresh_selectionarray of stringBeta

A list of tables to update with fullRefresh.

reset_checkpoint_selectionarray of stringBeta

A list of streaming flows to reset checkpoints without clearing data.

refresh_flow_selectionarray of stringBeta

Flow names to selectively refresh. These are unioned with other selective refresh options (refresh_selection, full_refresh_selection) to determine the final set of flows to refresh.

python_wheel_taskobject

The task runs a Python wheel when the python_wheel_task field is present.

Show child attributesHide child attributes
package_namestring

Name of the package to execute

entry_pointstring

Named entry point to use, if it does not exist in the metadata of the package it executes the function from the package directly using $packageName.$entryPoint()

parametersarray of string

Command-line parameters passed to Python wheel task. Leave it empty if named_parameters is not null.

named_parametersobject

Command-line parameters passed to Python wheel task in the form of ["--name=task", "--data=dbfs:/path/to/data.json"]. Leave it empty if parameters is not null.

dbt_taskobject

The task runs one or more dbt commands when the dbt_task field is present. The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse.

Show child attributesHide child attributes
project_directorystring

Path to the project directory. Optional for Git sourced tasks, in which case if no value is provided, the root of the Git repository is used.

commandsarray of string

A list of dbt commands to execute. All commands must start with dbt. This parameter must not be empty. A maximum of up to 10 commands can be provided.

schemastring

Optional schema to write to. This parameter is only used when a warehouse_id is also provided. If not provided, the