Aller au contenu principal
RSS Feed
Dernière mise à jour le

Databricks Runtime 18 LTS

This page contains unified release notes for Databricks Runtime 18 LTS, powered by Apache Spark 4.1.0. It lists all features, behavior changes, and fixes shipped throughout the Databricks Runtime 18 LTS release cycle.

remarque

Releases are staged. Your Databricks account might not update for a few days after the initial release date.

About unified release notes​

Databricks releases features incrementally throughout the release cycle without changing the version number. Rather than separate pages for each minor version, all updates appear on this single page as dated entries. Clusters receive updates when they restart.

Databricks Runtime 18 is the first release to use this unified format. Previously, each feature version (18.0, 18.1, 18.2) had its own release notes page. Those pages remain available for reference. Features that would previously have shipped as 18.3 or later now ship as dated updates to Databricks Runtime 18.

In the future, Databricks Runtime versions will follow a lifecycle from Beta (early evaluation) through GA (feature development, approximately six months) to LTS (long-term support, three years of stability and security fixes). Because 18.0 through 18.2 shipped under the previous model, Databricks Runtime 18 is a transitional release. It is currently in the GA phase and will transition to LTS when Databricks Runtime 19 is generally available. For full lifecycle details, see Databricks Runtime support lifecycles.

Databricks Runtime 18: June 10, 2026​

Databricks Runtime 18 is now generally available (GA). For lifecycle details, see Databricks Runtime support lifecycles.

New features and improvements​

The following features and improvements are available in this release.

  • Spark Declarative Pipelines on Lakeflow streaming query ID: Spark Declarative Pipelines on Lakeflow pipelines can now set the streaming query ID on demand.
  • New IP address functions (Public Preview): New SQL functions are available for working with IPv4 and IPv6 addresses and CIDR blocks, including ip_host, ip_cidr, ip_version, ip_prefix_length, ip_network, ip_network_last, ip_cidr_contains, ip_as_binary, ip_as_string, and try_* variants for null-safe behavior. See ip_host and related functions.
  • On-demand state repartitioning (Public Preview): Structured Streaming now supports changing the number of shuffle partitions for stateful queries without losing checkpoint state. See On-demand state repartitioning for stateful streaming queries.

Apache Spark fixes and improvements​

This release includes the following bug fixes and improvements made to Spark:

  • Fixed incorrect null handling in LEFT OUTER JOIN LATERAL expressions that contain EXPLODE. Previously, certain inputs produced incorrect null values in results.
  • Photon now defers duplicate field_id errors in Parquet schema resolution until the affected field is accessed. Queries that don't reference the duplicate field now succeed rather than failing at schema resolution time.

Databricks Runtime 18: June 8, 2026​

Apache Spark fixes and improvements​

This release includes the following bug fixes and improvements made to Spark:

  • NEAREST BY is now a supported SQL join type for top-K nearest-neighbor queries.
  • DESCRIBE TABLE ... PARTITION is now supported for v2 catalog tables. Previously, this syntax was only available for v1 tables.
  • SQL PATH support has been extended: SET PATH is now a supported SQL statement, CURRENT_PATH() returns the current path, and the PATH is now persisted in view and SQL function definitions and included in DESCRIBE output.
  • UPDATE queries on DSv2 tables now include operation metrics in query output.
  • DSv2 partition predicates now support nested partition columns.
  • Fixed a bug where aggregate queries using PartialMerge or Final aggregate execution could fail due to missing required input attributes.
  • Large external sorts now use bounded k-way merging to reduce out-of-memory errors when spilling data to disk.

Databricks Runtime 18: June 4, 2026​

Behavioral changes​

Review the following changes, which take effect when clusters restart on this runtime.

  • NATURAL JOIN case-insensitive column matching: NATURAL JOIN now respects the spark.sql.caseSensitive configuration (default: false) when matching common columns between the left and right sides of the join. Previously, column matching was always case-sensitive, causing columns that differ only in case (for example, ID vs id) to not be recognized as common columns. This resulted in a silent cross join instead of the expected equi-join. The equivalent USING join already handled this correctly. To restore the previous behavior, set spark.sql.legacy.naturalJoinCaseSensitiveColumnMatching to true.

Databricks Runtime 18: May 29, 2026​

  • Structured Streaming deduplication with NaN keys: Structured Streaming deduplication now treats NaN (Not-a-Number) values that have different bit patterns as duplicates when a double or float column is used as a deduplication key. Previously, NaN values with different internal representations were treated as distinct and were not deduplicated.
  • Fixed an issue where table-level permissions could be removed from Unity Catalog foreign catalog tables (for example, Snowflake connection tables) during metadata refresh, which caused queries to fail with INSUFFICIENT_PERMISSIONS errors. Permissions are now preserved when foreign table metadata is refreshed.

Databricks Runtime 18: May 18, 2026​

Behavioral changes​

Review the following changes, which take effect when clusters restart on this runtime.

  • CREATE OR REPLACE TABLE: CREATE OR REPLACE TABLE now preserves existing column and table comments by default. Previously, comments were dropped when recreating a table. Managed tables and views now match the existing behavior of materialized views and streaming tables.
  • DataFrame by-name writes: writeTo().append(), writeTo().overwrite(), writeTo().overwritePartitions(), and write.mode("append").saveAsTable() now automatically cast type-compatible columns (for example, int to long) to match the target Delta table schema. Previously, these operations failed with a DELTA_FAILED_TO_MERGE_FIELDS error when column types were compatible but not identical. Behavior now matches SQL INSERT INTO ... BY NAME. save() and saveAsTable() in overwrite mode are not affected.
  • ALTER TABLE SET TBLPROPERTIES for pipelines.pipelineId: ALTER TABLE <table> SET TBLPROPERTIES('pipelines.pipelineId' = '<pipeline-id>') now attempts to make the specified table eligible for writes by the pipeline. Previously, setting this property on a regular table had no effect. If the table isn't eligible for pipeline writes, the command throws SETTING_PIPELINES_PIPELINE_ID_NOT_SUPPORTED.
  • cloud_files_state: The cloud_files_state table-valued function (TVF) now includes a discovery_mode column showing how Auto Loader discovered each file. Files discovered before upgrading have a null value in this column.
  • DESCRIBE EXTENDED AS JSON: Now includes predictive optimization evaluation results in its output. Previously, this information wasn't returned in the JSON output.
  • Metric view window measures: Now return correct results when queries use GROUP BY, IN/BETWEEN filters, or mixed predicates on the window's order column. Previously, these filter patterns could produce incorrect results.

New features and improvements​

The following features and improvements are available in this release.

  • Cluster libraries: When you install PyPI libraries, wheels, or requirements.txt files on a cluster, the Libraries tab and Libraries REST API now display all resolved and installed packages, including transitive dependencies. For installations that resolve more than 500 packages, the list is truncated. The full installation report is available in the cluster's driver logs.
  • array_sort with custom comparators: Photon now accelerates array_sort expressions that use custom comparators to sort arrays of structs by one or more fields. Previously, these expressions fell back to non-Photon execution. To enable this optimization, set spark.databricks.photon.rewriteArraySortComparator.enabled to true.

Apache Spark fixes and improvements​

This release includes the following bug fixes and improvements made to Spark:

  • SQL queries now support a QUALIFY clause, which lets you filter on window function results directly in the query. Previously this required wrapping the query in a subquery.
  • spark.read.json, spark.read.csv, and spark.read.xml now accept a DataFrame as input, so you can parse in-memory data without saving it to a file first.
  • DataFrame.changes() is now available for reading change data capture (CDC) output through the DataFrame API.
  • When a DataFrame column is referenced from the wrong DataFrame, the error now names the specific column. You can also set spark.sql.columnResolution.strict to make Spark enforce stricter column matching and catch these mistakes earlier.
  • MERGE INTO schema evolution now works correctly when the statement includes WHEN MATCHED THEN DELETE or references nested columns by alias. Previously, these combinations could fail or silently produce wrong results.