Python virtual environments help to make sure that you are using the correct versions of Python and Databricks Connect together. Databricks Runtime 13.0 ML and Databricks Runtime 13.0 both use Python 3.10.ĭatabricks strongly recommends that you have a Python virtual environment activated for each Python version that you use with Databricks Connect. The minor version of your client Python installation must be the same as the minor Python version of your Databricks cluster. Only Databricks Runtime 13.0 ML and Databricks Runtime 13.0 are supported. These include clusters with Single User or Shared cluster access modes. Only clusters that are compatible with Unity Catalog are supported. See Get started using Unity Catalog and Enable a workspace for Unity Catalog. This section lists the requirements for Databricks Connect.Ī Databricks workspace and its corresponding account that are enabled for Unity Catalog. For more information about Spark Connect, see Introducing Spark Connect. Spark Connect can be embedded everywhere to connect to Databricks: in IDEs, notebooks, and applications, allowing individual users and partners alike to build new (interactive) user experiences based on the Databricks Lakehouse.
With this “V2” architecture based on Spark Connect, Databricks Connect becomes a thin client that is simple and easy to use. Spark Connect introduces a decoupled client-server architecture for Apache Spark that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. Because the client application is decoupled from the cluster, it is unaffected by cluster restarts or upgrades, which would normally cause you to lose all the variables, RDDs, and DataFrame objects defined in a notebook.įor Databricks Runtime 13.0 and higher, Databricks Connect is now built on open-source Spark Connect. Shut down idle clusters without losing work. You do not need to restart the cluster after changing Python library dependencies in Databricks Connect, because each client session is isolated from each other in the cluster.
Iterate quickly when developing libraries. Step through and debug code in your IDE even when working with a remote cluster. Databricks Connect for Databricks Runtime 13.0 and higher currently supports running only Python applications.
0 kommentar(er)
