Skip to main content

Pipeline

View as Markdown

PipelineSpec object​

idstring

Unique identifier for this pipeline.

namestring

Friendly identifier for this pipeline.

storagestring

DBFS root directory for storing checkpoints and tables.

configurationobject

String-String configuration for this pipeline execution.

clustersarray of object

Cluster settings for this pipeline deployment.

Show child attributesHide child attributes
labelstring

A label for the cluster specification, either default to configure the default cluster settings applied to both the update and maintenance clusters, updates to configure the update cluster, or maintenance to configure the maintenance cluster. This field is optional. The default value is default.

apply_policy_default_valuesboolean

Note: This field won't be persisted. Only API users will check this field.

spark_confobject

An object containing a set of optional, user-specified Spark configuration key-value pairs. See clusters/create for more details.

aws_attributesobject

Attributes related to clusters running on Amazon Web Services. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes
first_on_demandint32

The first first_on_demand nodes of the cluster will be placed on on-demand instances. If this value is greater than 0, the cluster driver node in particular will be placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring

Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero, this availability type will be used for the entire cluster.

Values:

  • SPOT
  • ON_DEMAND
  • SPOT_WITH_FALLBACK
zone_idstring

Identifier for the availability zone/datacenter in which the cluster resides. This string will be of a form like "us-west-2a". The provided availability zone must be in the same region as the <Databricks> deployment. For example, "us-west-2a" is not a valid zone id if the <Databricks> deployment resides in the "us-east-1" region. This is an optional field at cluster creation, and if not specified, a default zone will be used. If the zone specified is "auto", will try to place cluster in a zone with high availability, and will retry placement in a different AZ if there is not enough capacity.

The list of available zones as well as the default value can be found by using the List Zones method.

instance_profile_arnstring

Nodes for this cluster will only be placed on AWS instances with this instance profile. If omitted, nodes will be placed on instances without an IAM instance profile. The instance profile must have previously been added to the <Databricks> environment by an account administrator.

This feature may only be available to certain customer plans.

spot_bid_price_percentint32

The bid price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. For example, if this field is set to 50, and the cluster needs a new r3.xlarge spot instance, then the bid price is half of the price of on-demand r3.xlarge instances. Similarly, if this field is set to 200, the bid price is twice the price of on-demand r3.xlarge instances. If not specified, the default value is 100. When spot instances are requested for this cluster, only spot instances whose bid price percentage matches this field will be considered. Note that, for safety, we enforce this field to be no more than 10000.

ebs_volume_typestring

The type of EBS volumes that will be launched with this cluster.

Values:

  • GENERAL_PURPOSE_SSD
  • THROUGHPUT_OPTIMIZED_HDD
ebs_volume_countint32

The number of volumes launched for each instance. Users can choose up to 10 volumes. This feature is only enabled for supported node types. Legacy node types cannot specify custom EBS volumes. For node types with no instance store, at least one EBS volume needs to be specified; otherwise, cluster creation will fail.

These EBS volumes will be mounted at /ebs0, /ebs1, and etc. Instance store volumes will be mounted at /local_disk0, /local_disk1, and etc.

If EBS volumes are attached, <Databricks> will configure Spark to use only the EBS volumes for scratch storage because heterogeneously sized scratch devices can lead to inefficient disk utilization. If no EBS volumes are attached, <Databricks> will configure Spark to use instance store volumes.

Please note that if EBS volumes are specified, then the Spark configuration spark.local.dir will be overridden.

ebs_volume_sizeint32

The size of each EBS volume (in GiB) launched for each instance. For general purpose SSD, this value must be within the range 100 - 4096. For throughput optimized HDD, this value must be within the range 500 - 4096.

ebs_volume_iopsint32
ebs_volume_throughputint32
azure_attributesobject

Attributes related to clusters running on Microsoft Azure. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes
first_on_demandint32

The first first_on_demand nodes of the cluster will be placed on on-demand instances. This value should be greater than 0, to make sure the cluster driver node is placed on an on-demand instance. If this value is greater than or equal to the current cluster size, all nodes will be placed on on-demand instances. If this value is less than the current cluster size, first_on_demand nodes will be placed on on-demand instances and the remainder will be placed on availability instances. Note that this value does not affect cluster size and cannot currently be mutated over the lifetime of a cluster.

availabilitystring

Availability type used for all subsequent nodes past the first_on_demand ones. Note: If first_on_demand is zero (which only happens on pool clusters), this availability type will be used for the entire cluster.

Values:

  • SPOT_AZURE
  • ON_DEMAND_AZURE
  • SPOT_WITH_FALLBACK_AZURE
spot_bid_max_pricedouble

The max bid price to be used for Azure spot instances. The Max price for the bid cannot be higher than the on-demand price of the instance. If not specified, the default value is -1, which specifies that the instance cannot be evicted on the basis of price, and only on the basis of availability. Further, the value should > 0 or -1.

gcp_attributesobject

Attributes related to clusters running on Google Cloud Platform. If not specified at cluster creation, a set of default values will be used.

Show child attributesHide child attributes
google_service_accountstring

If provided, the cluster will impersonate the google service account when accessing gcloud services (like GCS). The google service account must have previously been added to the <Databricks> environment by an account administrator.

boot_disk_sizeint32

boot disk size in GB

availabilitystring

This field determines whether the spark executors will be scheduled to run on preemptible VMs, on-demand VMs, or preemptible VMs with a fallback to on-demand VMs if the former is unavailable.

Values:

  • PREEMPTIBLE_GCP
  • ON_DEMAND_GCP
  • PREEMPTIBLE_WITH_FALLBACK_GCP
zone_idstring

Identifier for the availability zone in which the cluster resides. This can be one of the following:

  • "HA" => High availability, spread nodes across availability zones for a <Databricks> deployment region [default].
  • "AUTO" => <Databricks> picks an availability zone to schedule the cluster on.
  • A GCP availability zone => Pick One of the available zones for (machine type + region) from https://cloud.google.com/compute/docs/regions-zones.
local_ssd_countint32

The number of local SSDs to attach to each worker and driver for this cluster. If left unspecified, the default number of local SSDs for the node type will be used.

NOTE: Each instance type can only support a certain number of attached local SSDs. The value specified in local_ssd_count must be valid for BOTH the driver and worker instance type. See GCP docs here: https://cloud.google.com/compute/docs/disks#local_ssd_machine_type_restrictions

Validation is performed at the RPC layer and the RPC will be rejected if the specified local_ssd_count is invalid.

node_type_idstring

This field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the clusters/listNodeTypes API call.

driver_node_type_idstring

The node type of the Spark driver. Note that this field is optional; if unset, the driver node type will be set as the same value as node_type_id defined above.

ssh_public_keysarray of string

SSH public key contents that will be added to each Spark node in this cluster. The corresponding private keys can be used to login with the user name ubuntu on port 2200. Up to 10 keys can be specified.

custom_tagsobject

Additional tags for cluster resources. <Databricks> will tag all cluster resources (e.g., AWS instances and EBS volumes) with these tags in addition to default_tags. Notes:

  • Currently, <Databricks> allows at most 45 custom tags

  • Clusters can only reuse cloud resources if the resources' tags are a subset of the cluster tags

cluster_log_confobject

The configuration for delivering spark logs to a long-term storage destination. Only dbfs destinations are supported. Only one destination can be specified for one cluster. If the conf is given, the logs will be delivered to the destination every 5 mins. The destination of driver logs is $destination/$clusterId/driver, while the destination of executor logs is $destination/$clusterId/executor.

Show child attributesHide child attributes
dbfsobject

destination needs to be provided. e.g. &#123; "dbfs" : &#123; "destination" : "dbfs:/home/cluster_log" &#125; &#125;

Show child attributesHide child attributes
destinationstring

dbfs destination, e.g. dbfs:/my/path

spark_env_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs. Please note that key-value pair of the form (X,Y) will be exported as is (i.e., export X='Y') while launching the driver and workers.

In order to specify an additional set of SPARK_DAEMON_JAVA_OPTS, we recommend appending them to $SPARK_DAEMON_JAVA_OPTS as shown in the example below. This ensures that all default databricks managed environmental variables are included as well.

Example Spark environment variables: {"SPARK_WORKER_MEMORY": "28000m", "SPARK_LOCAL_DIRS": "/local_disk0"} or {"SPARK_DAEMON_JAVA_OPTS": "$SPARK_DAEMON_JAVA_OPTS -Dspark.shuffle.service.enabled=true"}

init_scriptsarray of object

The configuration for storing init scripts. Any number of destinations can be specified. The scripts are executed sequentially in the order provided. If cluster_log_conf is specified, init script logs are sent to <destination>/<cluster-ID>/init_scripts.

Show child attributesHide child attributes
dbfsobject

destination needs to be provided. e.g. &#123; "dbfs" : &#123; "destination" : "dbfs:/init-scripts/my_script.sh" &#125; &#125;

Show child attributesHide child attributes
destinationstring

dbfs destination, e.g. dbfs:/my/path

s3object

destination and either region or endpoint should also be provided. e.g. &#123; "s3": &#123; "destination" : "s3://init-scripts/my_script.sh", "region" : "us-west-2" &#125; &#125; Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.

Show child attributesHide child attributes
destinationstring

S3 destination, e.g. s3://my-bucket/some-prefix Note that logs will be delivered using cluster iam role, please make sure you set cluster iam role and the role has write access to the destination. Please also note that you cannot use AWS keys to deliver logs.

regionstring

S3 region, e.g. us-west-2. Either region or endpoint needs to be set. If both are set, endpoint will be used.

endpointstring

S3 endpoint, e.g. https://s3-us-west-2.amazonaws.com. Either region or endpoint needs to be set. If both are set, endpoint will be used.

enable_encryptionboolean

Flag to enable server side encryption, false by default.

encryption_typestring

The encryption type, it could be sse-s3 or sse-kms. It will be used only when encryption is enabled and the default type is sse-s3.

kms_keystring

Kms key which will be used if encryption is enabled and encryption type is set to sse-kms.

canned_aclstring

Set canned access control list for the logs, e.g. bucket-owner-full-control. If canned_cal is set, please make sure the cluster iam role has s3:PutObjectAcl permission on the destination bucket and prefix. The full list of possible canned acl can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl. Please also note that by default only the object owner gets full controls. If you are using cross account role for writing data, you may want to set bucket-owner-full-control to make bucket owner able to read the logs.

instance_pool_idstring

The optional ID of the instance pool to which the cluster belongs.

policy_idstring

The ID of the cluster policy used to create the cluster if applicable.

enable_local_disk_encryptionboolean

Whether to enable local disk encryption for the cluster.

driver_instance_pool_idstring

The optional ID of the instance pool for the driver of the cluster belongs. The pool cluster uses the instance pool with id (instance_pool_id) if the driver pool is not assigned.

num_workersint32

Number of worker nodes that this cluster should have. A cluster has one Spark Driver and num_workers Executors for a total of num_workers + 1 Spark nodes.

Note: When reading the properties of a cluster, this field reflects the desired number of workers rather than the actual current number of workers. For instance, if a cluster is resized from 5 to 10 workers, this field will immediately be updated to reflect the target size of 10 workers, whereas the workers listed in spark_info will gradually increase from 5 to 10 as the new nodes are provisioned.

autoscaleobject

Parameters needed in order to automatically scale clusters up and down based on load. Note: autoscaling works best with DB runtime versions 3.0 or later.

Show child attributesHide child attributes
min_workersint32

The minimum number of workers the cluster can scale down to when underutilized. It is also the initial number of workers the cluster will have after creation.

max_workersint32

The maximum number of workers to which the cluster can scale up when overloaded. max_workers must be strictly greater than min_workers.

modestring

Databricks Enhanced Autoscaling optimizes cluster utilization by automatically allocating cluster resources based on workload volume, with minimal impact to the data processing latency of your pipelines. Enhanced Autoscaling is available for updates clusters only. The legacy autoscaling feature is used for maintenance clusters.

librariesarray of object

Libraries or code needed by this deployment.

Show child attributesHide child attributes
whlstringDeprecated

URI of the whl to be installed.

notebookobject

The path to a notebook that defines a pipeline and is stored in the <Databricks> workspace.

Show child attributesHide child attributes
pathstring

The absolute path of the source code.

fileobject

The path to a file that defines a pipeline and is stored in the Databricks Repos.

Show child attributesHide child attributes
pathstring

The absolute path of the source code.

globobjectPublic Preview

The unified field to include source codes. Each entry can be a notebook path, a file path, or a folder path that ends /**. This field cannot be used together with notebook or file.

Show child attributesHide child attributes
includestringPublic Preview

The source code to include for pipelines

Examples:

  • /Workspace/Users/user.name/pipeline-folder/my-file.py : use glob to include a file
  • /Workspace/Users/user.name/pipeline-folder/** : use glob to include a folder
  • /Workspace/Users/user.name/pipeline-folder/my-notebook : use glob to include a notebook
ingestion_definitionobjectPublic Preview

The configuration for a managed ingestion pipeline. These settings cannot be used with the 'libraries', 'schema', 'target', or 'catalog' settings.

Show child attributesHide child attributes
connection_namestringPublic Preview

The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

ingestion_gateway_idstringPublic Preview

Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

ingest_from_uc_foreign_catalogbooleanPublic Preview

Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.

Default: false

objectsarray of objectPublic Preview

Required. Settings specifying tables to replicate and the destination for the replicated tables.

Show child attributesHide child attributes
schemaobjectPublic Preview

Select all tables from a specific source schema.

Show child attributesHide child attributes
source_catalogstringPublic Preview

The source catalog name. Might be optional depending on the type of source.

source_schemastringPublic Preview

Schema name in the source database. Optional: some source types (for example streaming or message-bus connectors) do not use it, so it may be absent from a pipeline's definition. Clients that assume it is always present should handle its absence.

destination_catalogstringPublic Preview

Required. Destination catalog to store tables.

destination_schemastringPublic Preview

Required. Destination schema to store tables in. Tables with the same name as the source tables are created in this destination schema. The pipeline fails If a table with the same name already exists.

table_configurationobjectPublic Preview

Configuration settings to control the ingestion of tables. These settings are applied to all tables in this schema and override the table_configuration defined in the IngestionPipelineDefinition object.

Show child attributesHide child attributes
scd_typestringPublic Preview

The SCD type to use to ingest the table.

Default: SCD_TYPE_1

Values:

  • SCD_TYPE_UNSPECIFIED
  • SCD_TYPE_1
  • SCD_TYPE_2
  • APPEND_ONLY
primary_keysarray of stringPublic Preview

The primary key of the table used to apply changes.

sequence_byarray of stringPublic Preview

The column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.

include_columnsarray of stringPublic Preview

A list of column names to be included for the ingestion. When not specified, all columns except ones in exclude_columns will be included. Future columns will be automatically included. When specified, all other future columns will be automatically excluded from ingestion. This field in mutually exclusive with exclude_columns.

exclude_columnsarray of stringPublic Preview

A list of column names to be excluded for the ingestion. When not specified, include_columns fully controls what columns to be ingested. When specified, all other columns including future ones will be automatically included for ingestion. This field in mutually exclusive with include_columns.

row_filterstringPublic Preview

(Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.

query_based_connector_configobjectPublic Preview
Show child attributesHide child attributes
cursor_columnsarray of stringPublic Preview

The names of the monotonically increasing columns in the source table that are used to enable the table to be read and ingested incrementally through structured streaming. The columns are allowed to have repeated values but have to be non-decreasing. If the source data is merged into the destination (e.g., using SCD Type 1 or Type 2), these columns will implicitly define the sequence_by behavior. You can still explicitly set sequence_by to override this default.

deletion_conditionstringPublic Preview

Specifies a SQL WHERE condition that specifies that the source row has been deleted. This is sometimes referred to as "soft-deletes". For example: "Operation = 'DELETE'" or "is_deleted = true". This field is orthogonal to hard_deletion_sync_interval_in_seconds, one for soft-deletes and the other for hard-deletes. See also the hard_deletion_sync_min_interval_in_seconds field for handling of "hard deletes" where the source rows are physically removed from the table.

hard_deletion_sync_min_interval_in_secondsint64Beta

Specifies the minimum interval (in seconds) between snapshots on primary keys for detecting and synchronizing hard deletions—i.e., rows that have been physically removed from the source table. This interval acts as a lower bound. If ingestion runs less frequently than this value, hard deletion synchronization will align with the actual ingestion frequency instead of happening more often. If not set, hard deletion synchronization via snapshots is disabled. This field is mutable and can be updated without triggering a full snapshot.

auto_full_refresh_policyobjectPublic Preview

(Optional, Mutable) Policy for auto full refresh, if enabled pipeline will automatically try to fix issues by doing a full refresh on the table in the retry run. auto_full_refresh_policy in table configuration will override the above level auto_full_refresh_policy. For example, { "auto_full_refresh_policy": { "enabled": true, "min_interval_hours": 23, } } If unspecified, auto full refresh is disabled.

Show child attributesHide child attributes
enabledbooleanPublic Preview

(Required, Mutable) Whether to enable auto full refresh or not.

min_interval_hoursint32Public Preview

(Optional, Mutable) Specify the minimum interval in hours between the timestamp at which a table was last full refreshed and the current timestamp for triggering auto full If unspecified and autoFullRefresh is enabled then by default min_interval_hours is 24 hours.

table_propertiesobjectBeta

Table properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.

enable_auto_clusteringbooleanBeta

Whether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

Default: false

clustering_columnsarray of stringBeta

List of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.

source_metadata_columnstringBeta

(Optional) Name of the struct column added to each ingested record to hold per row source metadata.

connector_optionsobjectPublic Preview

(Optional) Source Specific Connector Options

Show child attributesHide child attributes
tiktok_ads_optionsobjectBeta
Show child attributesHide child attributes
lookback_window_daysint32Beta

(Optional) Number of days to look back for report tables during incremental sync to capture late-arriving conversions and attribution data.

sync_start_datestringBeta

(Optional) Start date for the initial sync of report tables in YYYY-MM-DD format. This determines the earliest date from which to sync historical data.

dimensionsarray of stringDeprecatedBeta

Deprecated. Use custom_report_options.dimensions instead.

metricsarray of stringDeprecatedBeta

Deprecated. Use custom_report_options.metrics instead.

report_typestringDeprecatedBeta

Deprecated. Use custom_report_options.report_type instead.

Default: BASIC

Values:

  • TIK_TOK_REPORT_TYPE_UNSPECIFIED
  • BASIC
  • AUDIENCE
  • PLAYABLE_AD
  • DSA
  • BUSINESS_CENTER
  • GMV_MAX
data_levelstringDeprecatedBeta

Deprecated. Use custom_report_options.data_level instead.

Default: AUCTION_CAMPAIGN

Values:

  • TIK_TOK_DATA_LEVEL_UNSPECIFIED
  • AUCTION_ADVERTISER
  • AUCTION_CAMPAIGN
  • AUCTION_ADGROUP
  • AUCTION_AD
query_lifetimebooleanDeprecatedBeta

Deprecated. Use custom_report_options.query_lifetime instead.

jira_optionsobjectBeta
Show child attributesHide child attributes
include_jira_spacesarray of stringBeta

(Optional) Projects to filter Jira data on

confluence_optionsobjectPublic Preview
Show child attributesHide child attributes
include_confluence_spacesarray of stringPublic Preview

(Optional) Spaces to filter Confluence data on

meta_ads_optionsobjectBeta
Show child attributesHide child attributes
levelstringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.level) Granularity of data to pull (account, ad, adset, campaign)

breakdownsarray of stringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.breakdowns) Breakdowns to configure

action_breakdownsarray of stringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.action_breakdowns) Action breakdowns

action_report_timestringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.action_report_time) Timing used to report action statistics (impression, conversion, mixed, or lifetime)

start_datestringBeta

(Optional) Start date in yyyy-MM-dd format (e.g. 2025-01-15). Data added after this date will be ingested, shared by prebuilt and custom reports.

custom_insights_lookback_windowint32Beta

(Optional) Window in days to revisit data during sync to capture updated conversion data from the API, shared by prebuilt and custom reports.

time_incrementstringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.time_increment) Value in string by which to aggregate statistics (can take all_days, monthly or number of days)

action_attribution_windowsarray of stringDeprecatedBeta

(Optional, DEPRECATED — use custom_report_options.action_attribution_windows) Action attribution windows for insights reporting (e.g. "28d_click", "1d_view")

zendesk_support_optionsobjectPublic Preview
Show child attributesHide child attributes
start_datestringPublic Preview

(Optional) Start date in YYYY-MM-DD format for the initial sync. This determines the earliest date from which to sync historical data.

kafka_optionsobjectBeta
Show child attributesHide child attributes
topicsarray of stringBeta

Topics to subscribe to. Only one of topics or topic_pattern must be specified.

topic_patternstringBeta

Java regex pattern to subscribe to matching topics. Only one of topics or topic_pattern must be specified.

key_transformerobjectBeta

(Optional) Transformer for the message key. If not specified, the key is left as raw bytes.

Show child attributesHide child attributes
formatstringBeta

Required: the wire format of the data.

Values:

  • FORMAT_UNSPECIFIED
  • STRING
  • JSON
  • AVRO
  • PROTOBUF
json_optionsobjectBeta
Show child attributesHide child attributes
as_variantbooleanBeta

Parse the entire value as a single Variant column.

schemastringBeta

Inline schema string for JSON parsing (Spark DDL format).

schema_file_pathstringBeta

Path to a schema file (.ddl).

schema_evolution_modestringBeta

(Optional) Schema evolution mode for schema inference.

Values:

  • SCHEMA_EVOLUTION_MODE_UNSPECIFIED
  • ADD_NEW_COLUMNS_WITH_TYPE_WIDENING
  • ADD_NEW_COLUMNS
  • RESCUE
  • FAIL_ON_NEW_COLUMNS
  • NONE
schema_hintsstringBeta

(Optional) Schema hints as a comma-separated string of "column_name type" pairs.

avro_optionsobjectBeta
Show child attributesHide child attributes
schemastringBeta

Inline Avro JSON schema string.

schema_file_pathstringBeta

Path to a schema file (.avsc).

parse_modestringBeta

(Optional) Parse mode for Avro data. Valid values: FAILFAST, PERMISSIVE. Defaults to PERMISSIVE.

Values:

  • PARSE_MODE_UNSPECIFIED
  • FAILFAST
  • PERMISSIVE
schema_registryobjectBeta

(Optional) Schema registry to resolve the Avro schema at runtime instead of providing it inline or via a file path.

Show child attributesHide child attributes
confluent_optionsobjectBeta

Required: Confluent-compatible schema registry options.

Show child attributesHide child attributes
subjectstringBeta

Required: subject name to resolve in the registry.

protobuf_message_namestringBeta

(Optional, Protobuf only) Selects a specific message from a schema that defines multiple Protobuf messages. Simple ("Location") or fully-qualified ("com.example.protos.Location"). Defaults to the first message.

connection_namestringBeta

(Optional) UC connection for registry authentication. Specify if different from the top-level source connection.

protobuf_optionsobjectBeta
Show child attributesHide child attributes
desc_file_pathstringBeta

Required: path to the .desc file (dbfs:/... or /Volumes/...).

message_namestringBeta

Required: fully-qualified message type name.

recursive_fields_max_depthint32Beta

(Optional) Maximum expansion depth for recursive protobuf fields. Spark SQL does not natively support recursive types, so recursive fields are expanded up to this depth and truncated beyond it. Valid values: -1 (disallow recursive fields), 0 (drop), 1-10.

parse_modestringBeta

(Optional) Parse mode for Protobuf data. Valid values: FAILFAST, PERMISSIVE. Defaults to PERMISSIVE.

Values:

  • PARSE_MODE_UNSPECIFIED
  • FAILFAST
  • PERMISSIVE
schema_registryobjectBeta

(Optional) Schema registry to resolve the Protobuf schema at runtime instead of providing it via desc_file_path.

Show child attributesHide child attributes
confluent_optionsobjectBeta

Required: Confluent-compatible schema registry options.

Show child attributesHide child attributes
subjectstringBeta

Required: subject name to resolve in the registry.

protobuf_message_namestringBeta

(Optional, Protobuf only) Selects a specific message from a schema that defines multiple Protobuf messages. Simple ("Location") or fully-qualified ("com.example.protos.Location"). Defaults to the first message.

connection_namestringBeta

(Optional) UC connection for registry authentication. Specify if different from the top-level source connection.

value_transformerobjectBeta

(Optional) Transformer for the message value. If not specified, the value is left as raw bytes.

Show child attributesHide child attributes
formatstringBeta

Required: the wire format of the data.

Values:

  • FORMAT_UNSPECIFIED
  • STRING
  • JSON
  • AVRO
  • PROTOBUF
json_optionsobjectBeta
Show child attributesHide child attributes
as_variantbooleanBeta

Parse the entire value as a single Variant column.

schemastringBeta

Inline schema string for JSON parsing (Spark DDL format).

schema_file_pathstringBeta

Path to a schema file (.ddl).

schema_evolution_modestringBeta

(Optional) Schema evolution mode for schema inference.

Values:

  • SCHEMA_EVOLUTION_MODE_UNSPECIFIED
  • ADD_NEW_COLUMNS_WITH_TYPE_WIDENING
  • ADD_NEW_COLUMNS
  • RESCUE
  • FAIL_ON_NEW_COLUMNS
  • NONE
schema_hintsstringBeta

(Optional) Schema hints as a comma-separated string of "column_name type" pairs.

avro_optionsobjectBeta
Show child attributesHide child attributes
schemastringBeta

Inline Avro JSON schema string.

schema_file_pathstringBeta

Path to a schema file (.avsc).

parse_modestringBeta

(Optional) Parse mode for Avro data. Valid values: FAILFAST, PERMISSIVE. Defaults to PERMISSIVE.

Values:

  • PARSE_MODE_UNSPECIFIED
  • FAILFAST
  • PERMISSIVE
schema_registryobjectBeta

(Optional) Schema registry to resolve the Avro schema at runtime instead of providing it inline or via a file path.

Show child attributesHide child attributes
confluent_optionsobjectBeta

Required: Confluent-compatible schema registry options.

Show child attributesHide child attributes
subjectstringBeta

Required: subject name to resolve in the registry.

protobuf_message_namestringBeta

(Optional, Protobuf only) Selects a specific message from a schema that defines multiple Protobuf messages. Simple ("Location") or fully-qualified ("com.example.protos.Location"). Defaults to the first message.

connection_namestringBeta

(Optional) UC connection for registry authentication. Specify if different from the top-level source connection.

protobuf_optionsobjectBeta
Show child attributesHide child attributes
desc_file_path