Databricks

Creating a connection

To connect a Databricks table to Telmai, you'll need to gather specific information from your Databricks instance. Follow the steps below:

Capture Connection Details.

You can find the JDBC connectivity details in the Databricks workspace console:

Navigate to Compute -> Cluster -> Cluster Name -> Configuration -> Advanced Options -> JDBC/ODBC.

Capture the following details:

• Server Hostname

• Port

• HTTP Path

For more information, refer to the : Databricks ODBC and JDBC Drivers

Generate a Security Token

A security token is required to connect to the cluster remotely. Create this token from the Databricks workspace console:

• Go to the top right corner and click on your User Name -> User Settings -> Access Token -> Generate New Token.

• Capture the token created.

For detailed instructions, see theAuthentication for Databricks tools and APIs

Create a connection in Telmai

Once you have the necessary information, navigate to the "Connections" page, click +New Connection, select databricks, and enter the following properties:

• Host: Cluster hostname

• Port: Cluster port

• HTTP path: HTTP path of the cluster

• Token: API token

• Read from UnityCatalog: A flag that enables spark job to read from UnityCatalog directly. This option will only work in case of using Databricks for Telmai-Spark jobs.

Only use "Read from UnityCatalog" if running Telmai jobs in Databricks and provide access to Telmai service account to the data being accessed. Otherwise, data scans will fail.

Connecting an asset

Once a connection is defined, you can start using it to create assets. To create assets, you will need:

• Schema: Schema name (i.e., database name)

• Catalog (Optional): Unity catalog name

Alternatively, you can run a SQL query