Skip to main content
Last updated on

Databricks Runtime 16.4 LTS for Machine Learning

Databricks Runtime 16.4 LTS for Machine Learning provides a ready-to-go environment for machine learning and data science based on Databricks Runtime 16.4 LTS. Databricks Runtime ML contains many popular machine learning libraries, including TensorFlow, PyTorch, and XGBoost. Databricks Runtime ML includes AutoML, a tool to automatically train machine learning pipelines. Databricks Runtime ML also supports distributed deep learning training using TorchDistributor, DeepSpeed, and Ray.

tip

To see release notes for Databricks Runtime versions that have reached end-of-support (EoS), see End-of-support Databricks Runtime release notes. The EoS Databricks Runtime versions have been retired and might not be updated.

New features and improvements​

Databricks Runtime 16.4 LTS ML is built on top of Databricks Runtime 16.4 LTS. For information on what's new in Databricks Runtime 16.4 LTS, including Apache Spark MLlib and SparkR, see the Databricks Runtime 16.4 LTS release notes.

Other changes​

XGBoost4J-Spark is now deprecated​

XGBoost4J-Spark is now deprecated and will not be pre-installed in the next major Databricks Runtime ML version. Databricks recommends using the Python API for xgboost.spark. See Distributed training of XGBoost models using xgboost.spark.

TensorFlow 2.17.x cannot load cuDNN library​

It is a known issue in Databricks Runtime 16.4 LTS ML that TensorFlow 2.17.x (default version) cannot access GPU devices due to a cuDNN library loading error. To fix this issue, upgrade TensorFlow to version 2.18.x or above.

System environment​

The system environment in Databricks Runtime 16.4 LTS ML differs from Databricks Runtime 16.4 LTS as follows.

  • For GPU clusters, Databricks Runtime ML includes the following NVIDIA GPU libraries:
    • CUDA 12.6
    • cublas 12.6.0.22-1
    • cusolver 11.6.4.38-1
    • cupti 12.6.37-1
    • cusparse 12.5.2.23-1
    • cuDNN 9.3.0.75-1
    • NCCL 2.22.3
    • TensorRT 10.2.0.19-1

Libraries​

The following sections list the libraries included in Databricks Runtime 16.4 LTS ML that differ from those included in Databricks Runtime 16.4 LTS.

In this section:

Top-tier libraries​

Databricks Runtime 16.4 LTS ML includes the following top-tier libraries:

Python libraries​

Databricks Runtime 16.4 LTS ML uses virtualenv for Python package management and includes many popular ML packages.

In addition to the packages specified in the following sections, Databricks Runtime 16.4 LTS ML also includes the following packages:

  • hyperopt 0.2.8+db1
  • automl 1.30.0

To reproduce the Databricks Runtime ML Python environment in your local Python virtual environment, download the requirements-16.4.txt file and run pip install -r requirements-16.4.txt. This command installs all of the open source libraries that Databricks Runtime ML uses, but does not install libraries developed by Databricks, such as databricks-automl or the Databricks fork of hyperopt.

Python libraries on CPU clusters​

Library

Version

Library

Version

Library

Version

absl-py

1.0.0

accelerate

1.5.2

aiohttp

3.9.5

aiohttp-cors

0.8.1

aiosignal

1.2.0

alembic

1.14.1

annotated-types

0.7.0

anyio

4.2.0

argcomplete

3.6.2

argon2-cffi

21.3.0

argon2-cffi-bindings

21.2.0

arrow

1.2.3

astor

0.8.1

asttokens

2.0.5

astunparse

1.6.3

async-lru

2.0.4

attrs

23.1.0

audioread

3.0.1

autocommand

2.2.2

azure-core

1.33.0

azure-cosmos

4.3.1

azure-identity

1.21.0

azure-storage-blob

12.23.0

azure-storage-file-datalake

12.17.0

Babel

2.11.0

backoff

2.2.1

backports.tarfile

1.2.0

bcrypt

3.2.0

beautifulsoup4

4.12.3

black

24.4.2

bleach

4.1.0

blinker

1.7.0

blis

0.7.11

boto3

1.34.69

botocore

1.34.69

Brotli

1.0.9

cachetools

5.3.3

catalogue

2.0.10

category-encoders

2.6.3

certifi

2024.6.2

cffi

1.16.0

chardet

4.0.0

charset-normalizer

2.0.4

circuitbreaker

2.1.3

click

8.1.7

cloudpathlib

0.21.0

cloudpickle

2.2.1

cmdstanpy

1.2.5

colorful

0.5.6

colorlog

6.9.0

comm

0.2.1

composer

0.29.0

confection

0.1.5

configparser

5.2.0

contourpy

1.2.0

coolname

2.2.0

cryptography

42.0.5

cycler

0.11.0

cymem

2.0.11

Cython

3.0.11

dacite

1.9.2

databricks-automl-runtime

0.2.21

databricks-feature-engineering

0.10.2

databricks-sdk

0.30.0

datasets

3.5.0

dbl-tempo

0.1.26

dbus-python

1.3.2

debugpy

1.6.7

decorator

5.1.1

deepspeed

0.16.5

defusedxml

0.7.1

Deprecated

1.2.18

dill

0.3.8

distlib

0.3.8

dm-tree

0.1.9

docstring-to-markdown

0.11

einops

0.8.1

entrypoints

0.4

evaluate