# Getting Started
Telmai is a data quality analysis and monitoring platform that helps data teams proactively understand, detect, and resolve data quality issues without writing code, SQL, or rules.
It uses powerful statistical analysis and machine learning for this. Our business user-friendly UI helps users define correct vs. incorrect data and then use these definitions to proactively monitor data and alert when deviations from the normal are detected.
This user guide will enable you to easily integrate with Telmai to investigate data quality issues, translate them into your definitions of correctness, and then set up monitoring and alerting.
Telmai can run as a public SaaS as well as be deployed in the customer’s VPC (supported for AWS, GCP, and Azure)
# Connect to Data
This page and subpages describes how to connect your data to Telmai
## Create a Connection
Telmai provides a wide range of connectors, all of which can be found [here](https://www.telm.ai/integrations/). Some of the connectors are readily available in each tenant (see the list below), while the others can be enabled by contacting Telmai support.
To connect to your data source:
1. Navigate to Connections page using the left-side menu
2. Select desired scope:
1. Global scope: Connection can be reused for any project
2. Project scope: Connection can only be used for associated project
3. Click **+Add Connection** button
4. You will be prompted to specify the connector type and fill the details
Create a connection
{% hint style="info" %}
Once you have selected the connection type, you will be asked to fill information specified to your connection
Connection can be reused to connect multiple datasets, tables, etc.
{% endhint %}
## Connect to a data asset
Once a connection is available, you can navigate to "Asset" page, and add your connection. Data assets must belong to a project.
1. Navigate to **Asset** page
2. Select target project, and click **+Add** button
3. Select source type
4. Select a created connection from the dropdown menu
* You can also create a connection by clicking **+New Connection** button
5. Fill the data asset details, and follow onscreen instructions
### List of available connectors
* Google Big Query
* Google Cloud Storage
* AWS S3
* AWS Redshift
* AWS Athena
* Azure Blob
* Databricks
* Snowflake
* Salesforce
* SAP Hana
* Apache Iceberg
Telmai allows you to read a variety of file formats for the data stored in cloud buckets:
* Comma Separated Values (CSV)
* Tab Separated Values (TSV)
* Parquet
* JSON
* Databricks Delta
# Projects
A **Project** in Telmai is a way to organize your data sources or groups of sources, helping you manage your workspace or tenant efficiently.
To create a new project, you need to:
1. Click on the **Manage Projects** button. This will open a menu where you can create, modify, or delete projects, as shown in the image below.
2. Click on the **+Project** button.
3. Enter a **Project Name** and an optional **Description** when prompted.
4. Click **Save** to finalize the creation of your project.
You can modify the project description anytime by clicking the **Edit** button. Projects can only be deleted if they contain no sources, using the **Delete** button.
Access Permissions can be set at any time for the project. Please refer to the [RBAC support section](/telmai/authentication-and-access-control#role-based-access).
### Move Source Across Projects
Once a source is created, it can be moved to a different project with few easy clicks:
1. Select the source you like to move, and click the 3-dot menu
2. Click "Move to Project" option
* You will see list of projects you have access to
3. Select the desired project, and click "Move to Project"
{% hint style="info" %}
All rules & policies will move with the project except for rules using templates
{% endhint %}
# Data Connections
### List of connectors available
* [Google Big Query](/telmai/connect-to-data/data-connections/google-bigquery)
* [Google Cloud Storage](/telmai/connect-to-data/data-connections/google-cloud-storage)
* [Hive Metastore](/telmai/connect-to-data/data-connections/iceberg) / [Iceberg REST](/telmai/connect-to-data/data-connections/iceberg-rest)
* [AWS S3](/telmai/connect-to-data/data-connections/aws-s3)
* [AWS Redshift](/telmai/connect-to-data/data-connections/aws-redshift)
* [AWS Athena](/telmai/connect-to-data/data-connections/aws-athena)
* [Azure Blob](/telmai/connect-to-data/data-connections/azure-blob)
* [Databricks Delta](/telmai/connect-to-data/data-connections/databricks)
* [MySQL](/telmai/connect-to-data/data-connections/mysql)
* [Snowflake](/telmai/connect-to-data/data-connections/snowflake)
* [Salesforce](/telmai/connect-to-data/data-connections/salesforce)
* [SAP Hana](/telmai/connect-to-data/data-connections/sap-hana)
* [SQL Server / Azure SQL Database](/telmai/connect-to-data/data-connections/sql-server)
* [Trino](/telmai/connect-to-data/data-connections/trino)
Please click [here](https://www.telm.ai/integrations/) for full list of integrations. If the system you need isn't listed above, don't hesitate to contact Telmai support to enable it.
### **Supported File Formats**
Telmai supports reading various file formats stored in cloud buckets, including:
* Comma Separated Values (CSV)
* Parquet
* JSON
* Databricks Delta
[Learn more about supported file formats](/telmai/connect-to-data/supported-file-types).
### **Unsupported Data Types**
Telmai currently does **not support profiling, monitoring, or rule evaluation** for certain database data types that are non-relational, binary, object-based, or cursor-referenced in nature.
Columns defined with the following data types are excluded from Telmai’s processing pipeline.
1. BLOB
2. BINARY
3. VARBINARY
4. JAVA\_OBJECT
5. DATALINK
6. REF
7. REF\_CURSOR
# AWS Athena
## Creating a connection
Before connecting to AWS Athena, ensure that the IAM user, whose credentials will be used, has the following permissions granted:
**Athena Permissions**
```
athena:StartQueryExecution
athena:GetQueryExecution
athena:GetQueryResults
athena:StartQueryExecution
athena:GetWorkGroup
athena:StopQueryExecution
athena:CreatePreparedStatement
athena:UpdatePreparedStatement
athena:GetPreparedStatement
athena:DeletePreparedStatement
s3:ListBucket
s3:GetObject
s3:PutObject
s3:GetBucketLocation
s3:GetObject
s3:ListBucket
s3:PutObject
s3:ListMultipartUploadParts
s3:AbortMultipartUpload
glue:GetDatabase,
glue:GetDatabases,
glue:CreateDatabase
glue:CreateTable
glue:GetTable
glue:GetTables
```
**S3 Permissions:**
```
s3:ListBucket
s3:GetObject
s3:PutObject
s3:GetBucketLocation
s3:GetObject
s3:ListBucket
s3:PutObject
s3:ListMultipartUploadParts
s3:AbortMultipartUpload
```
**Glue Permissions:**
```
glue:GetDatabase,
glue:GetDatabases,
glue:CreateDatabase
glue:CreateTable
glue:GetTable
glue:GetTables
```
Once permissions are defined, you can setup the connection by defining:
* **AWS** **Key**
* **AWS** **Secret**
* **Database**
* **Region**
* **S3 Staging Directory**
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need to select existing table or run a custom SQL query.
# AWS Redshift
{% hint style="info" %}
Telmai only supports connecting to provisioned Redshift clusters. Serverless clusters are not supported
{% endhint %}
## Creating a connection
To connect to Redshift provisioned clusters, the following details are required:
* **Endpoint Prefix**:
* Navigate to **Amazon Redshift Clusters** -> **\** -> **General Information** -> **Endpoint**.
* Retrieve the value, which will be in the format `[endpoint_prefix].redshift.amazonaws.com:[port]/[database]`.
* Use the value of **\[endpoint\_prefix]** here.
* **User**: The username to use for the connection (must be created in Redshift).
* **Password**: The password to use for the connection (must be created in Redshift).
{% hint style="info" %}
**Note**: the username and password need to be created inside the Redshift and it's not the IAM user.
{% endhint %}
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Database**:
* Navigate to **Amazon Redshift Clusters** -> **\** -> **General Information** -> **Endpoint**.
* Use the value of **\[database]** from the endpoint.
* **Schema**: The schema where the tables are located.
Alternatively, you can run a SQL query
# AWS S3
## Creating a connection
Telmai can be setup to read files directly from your S3 bucket. This requires read access to your S3 bucket. Please add the following permissions to the S3 bucket for the IAM user whose credentials (key and secret) will be used:
1. s3:GetBucketLocation
2. s3:ListBucket
3. s3:GetObject
Once the permissions are granted, proceed to the Telmai UI and enter the required information:
* **AWS** **Key:** Your AWS access key.
* **AWS** **Secret Key:** Your AWS secret access key.
* **Region:** The AWS region where your S3 bucket is located.
* **Bucket**: The name of your S3 bucket.
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Path:** The full path to the file in the bucket or the full path of the folder containing all files to be scanned. If specifying a folder, ensure all files have the same extension (e.g., .csv, .json, .parquet).
* **File Type:** Please refer to [Supported File Types](/telmai/connect-to-data/supported-file-types).
* **Delimiter (Optional for CSV type):** Choose the delimiter used in your files (e.g., comma, tab, space).
# Azure Blob
Telmai connects to Azure Blob Storage (Azure File) using SAS (Shared Access Signature) token authentication. Follow the steps below to set up the connection.
## Creating a connection
To create a SAS token, follow the instructions provided here:
Once the SAS token is generated, navigate to the Telmai UI and enter the following details to create a new connections:
* **Storage Account:** Your Azure Storage account name.
* **SAS Key:** The SAS token generated in the previous step.
* **Bucket:** The name of the Azure Blob container.
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Path:** The path to a specific file or folder within the container.
* **File Type:** Please refer to [Supported File Types](/telmai/connect-to-data/supported-file-types).
* **Delimiter (Optional):** Specify the delimiter if the files are in CSV format.
# Azure OneLake
Telmai connects to Azure [OneLake](https://learn.microsoft.com/en-us/fabric/onelake/onelake-overview) using Client ID and Secret. Follow the steps below to set up the connection.
## Prerequisites
Before creating the connection in Telmai, register an application in Azure and generate a Client ID and Secret. Follow Microsoft's [Register an application](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) guide to complete this step.
You will need to collect the following from Azure:
* **Client ID** — from the registered application's Overview page
* **Client Secret** — generated under **Certificates & secrets**
* **Tenant ID** — your Azure Active Directory tenant ID
## Creating a connection
Once the client ID and secret are generated, navigate to the Telmai UI and enter the following details to create a new connection:
* **Workspace:** Fabric workspace id
* **Tenant**: Fabric tenant
* **Client ID**
* **Client Secret**
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Path:** The path to a specific file or folder within the container.
* **File Type (Optional)**
* **Delimiter (Optional):** Specify the delimiter if the files are in CSV format.
# Databricks
## Creating a connection
To connect a Databricks table to Telmai, you'll need to gather specific information from your Databricks instance. Follow the steps below:
### **Capture Connection Details.**
You can find the JDBC connectivity details in the Databricks workspace console:
Navigate to **Compute** -> **Cluster** -> **Cluster Name** -> **Configuration** -> **Advanced Options** -> **JDBC/ODBC**.
Capture the following details:
* **Server Hostname**
* **Port**
* **HTTP Path**
For more information, refer to the : [Databricks ODBC and JDBC Drivers](https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html)
### **Generate a Security Token**
A security token is required to connect to the cluster remotely. Create this token from the Databricks workspace console:
* Go to the top right corner and click on your **User Name** -> **User Settings** -> **Access Token** -> **Generate New Token**.
* Capture the token created.
For detailed instructions, see the[Authentication for Databricks tools and APIs](https://docs.databricks.com/dev-tools/api/latest/authentication.html#token-management)
### Create a connection in Telmai
Once you have the necessary information, navigate to the "Connections" page, click +New Connection, select databricks, and enter the following properties:
* **Host:** Cluster hostname
* **Port:** Cluster port
* **HTTP path:** HTTP path of the cluster
* **Token:** API token
* **Read from UnityCatalog:** A flag that enables spark job to read from UnityCatalog directly. This option will only work in case of using Databricks for Telmai-Spark jobs.
{% hint style="warning" %}
Only use "Read from UnityCatalog" if running Telmai jobs in Databricks and provide access to Telmai service account to the data being accessed. Otherwise, data scans will fail.
{% endhint %}
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Schema:** Schema name (i.e., database name)
* **Catalog (Optional):** Unity catalog name
Alternatively, you can run a SQL query
# Google Cloud Storage
To set up a Google Cloud Storage connection, you need to specify the corresponding GCS bucket name. The Telmai service account must be granted access to the buckets requiring connections.
## Prerequisites
Before setting up the connection, identify which service account you will use:
* **Tenant service account**: Telmai's managed (or impersonated) service account. This is only available for GCP deployments. To find your specific tenant service account, open the **Create Connection** dialog and select **Google Cloud Storage**. The service account will be displayed on that page.
* **Your own service account**: if you prefer to provide your own credentials.
This service account must have the right permissions to access and read the data.
### Setting up permissions in Google Cloud Storage
1. Go to **Google Cloud Storage** and locate your bucket: `Google Cloud Storage / / Permissions / Members`
2. Click **Add**.
3. In the **New Member** field, enter the service account identified in the prerequisites above.
4. Assign the following roles:
1. `Storage Legacy Bucket Reader`
2. `Storage Legacy Object Reader`
5. Click **Save** to apply the changes.
## Creating the connection in Telmai
GCS connections can be used to connect multiple data assets in Telmai using the same connection parameters. To add a GCS connection:
1. Navigate to the Telmai connection page and click the **+ Add Connection** button.
2. In the **Create Connection** dialog, select **Google Cloud Storage** as the Connection Type.
3. Enter a **Name** for the connection (required) and an optional **Description**.
4. Under **Properties**, enter the **Bucket** name (name only, not the full path).
5. Under **Credentials**, select one of the following:
* **Use service account credentials** - provide your own service account key.
* **Use tenant service account** - uses the Telmai-managed service account.
6. Click **Create**.
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* **Path**: Provide the full path of the file inside the bucket or the full path of the folder containing all files to be sent to Telmai. If specifying a folder, ensure all files have the same extension. Wildcards are accepted in this field.
* **File Type:** Please refer to [Supported File Types](/telmai/connect-to-data/supported-file-types).
* **Delimiter \[Optional]** : For CSV files, specify the delimiter (comma, tab, semicolon, space).
# Google BigQuery
To set up a BigQuery connection, you need to specify the corresponding Google Project name. The Telmai service account must be granted access to the datasets requiring connections.
## Prerequisites
Before setting up the connection, identify which service account you will use:
* **Tenant service account** : Telmai's managed (or impersonated) service account. This is only available for GCP deployments. To find your specific tenant service account, open the **Create Connection** dialog and select BigQuery. The service account will be displayed on that page.
* **Your own service account**: if you prefer to provide your own credentials.
This service account must have the right permissions to access and query the data.
### Setting up permissions in Google BigQuery
1. Go to Google BigQuery and locate your project: `Google BigQuery / / `
2. Click **Share** or **Add Member**.
3. In the **New Member** field, enter the service account identified in the prerequisites above.
4. Assign the roles:
1. `BigQuery Data Viewer`
2. `BigQuery Metadata Viewer`
5. Save the settings.
### Setting up permissions in GCP Project
The used service account must also be granted `BigQuery Job User` in the GCP project where the query is expected to run
## Creating the connection in Telmai
BigQuery connections can be used to connect multiple data assets in Telmai using the same connection parameters. To add a BigQuery connection:
1. Navigate to Telmai connection page and click the **+ Add Connection** button.
2. In the **Create Connection** dialog, select **Google BigQuery** as the Connection Type.
3. Enter a **Name** for the connection (required) and an optional **Description**.
4. Under **Properties**, enter your **Google Project** name.
* Optionally, check **Use dataset project for job** if you want the job to run under the dataset's project.
5. Under **Credentials**, select one of the following:
* **Use service account credentials** - provide your own service account key.
* **Use tenant service account** - uses the Telmai-managed service account.
6. Click **Create**.
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* Dataset name
* \[Optional] Custom SQL
# Hive Metastore
Connect using Hive Metastore with HIVE or ICEBERG catalog type (GCS & S3)
Feature only supported for GCP & AWS deployments (GCS & S3 metastore)
## Creating a connection
To set up a connection to Hive Metastore, provide the following:
* **Thrift URI**: The Thrift URI used to connect to the Hive Metastore using the Thrift protocol.
* **Location Type**: The type of storage where physical data files are present — **GCS** or **S3**.
* **Catalog Type**: Choose **HIVE** for a standard Hive catalog, or **ICEBERG** for an Iceberg catalog hosted in Hive Metastore.
### GCS credentials
GCS location type
| Field | Description |
| -------------- | ------------------------------ |
| Client Email | Service account email address |
| Client ID | Service account client ID |
| Project ID | GCP project ID |
| Private Key ID | Service account private key ID |
| Private Key | Service account private key |
*The specified GCS service account credentials should have read-access to the Hive Metastore warehouse. In the case of a private cloud deployment, write-access to the Telmai internal storage bucket must also be granted to this service account.*
### S3 credentials
S3 location type
| Field | Description |
| ---------- | ----------------------------------------- |
| Region | AWS region where the S3 bucket is located |
| Access Key | AWS access key ID |
| Secret Key | AWS secret access key |
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* Database name
* The next step will show you available tables
{% hint style="warning" %}
Group scans are not supported for Hive Metastore connections.
{% endhint %}
# Iceberg REST
BigLake Metastore via Iceberg REST catalog (GCP only)
Feature only supported for GCP-BigLake and GCP deployments
## Creating a connection
To setup connection to Iceberg REST metastore, you will need to provide
* **Catalog URI**: Enter the REST catalog URI for your BigLake Metastore.
* **Project ID**: GCP Project ID where the metastore exists
* **Region:** GCP region for metastore
* **Warehouse:** Warehouse name
* **Default Branch:** Default table branch name
## Required Permissions
The following permissions are required to enable reading data from BigLake:
On BigLake side, you will need to grant:
* `Big Lake Viewer`
* `Service Usage Consumer`
On the Storage Bucket used by BigLake, you will need to grant:
* `Storage Object Viewer`
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* Target Namespace
* Next step will show you available tables
{% hint style="warning" %}
Custom SQL Query is not currently supported for Iceberg connections
{% endhint %}
{% hint style="warning" %}
Delta scans is not currently supported for Iceberg connections
{% endhint %}
Once defined, you will be able to see your data asset in Telmai
# MySQL
Connect Telmai to a MySQL database hosted on-prem or in the cloud.
## Creating a connection
To connect to MySQL, provide the following details:
1. **Name**: A display name for this connection
2. **Description**: Optional description
3. **Database**: The database name to connect to
4. **Server**: The hostname or IP address of the MySQL server
5. **Port**: The port MySQL is listening on (default: `3306`)
6. **Use SSL**: Whether to require an SSL connection (`true` / `false`)
7. **User**: The username
8. **Password**: The password
MySQL connection form
{% hint style="info" %}
In order to connect to a privately hosted instance, the customer will need to whitelist the IPs from Telmai. Please reach out to Telmai support for the updated list of IPs for whitelisting.
{% endhint %}
## Connecting an asset
Once a connection is defined, you can start using it to create assets. Select an existing connection (or create a new one), then choose whether to create assets from an **Existing Table** or a **Custom Query**.
MySQL asset configuration step
{% hint style="info" %}
Please ensure that your environment has whitelisted [Telmai IP list](/telmai/api-reference/telmai-ip-list).
{% endhint %}
# SAP Hana
## Creating a connection
To connect to SAP Hana, please fill in the required fields on the form below:
* Database
* Server
* Port
* Schema
* Authentication: User and password
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need to select existing table, or run a custom SQL query.
# SQL Server
SQL Server can be hosted on prem or run in Azure, GCP or AWS.
## Creating a connection
To connect to SQL Server, you'll need to provide the below details:
{% hint style="info" %}
Fill in following details in the form. We use Azure SQL service as an example here:
{% endhint %}
1. **Server**: The server name. It can be found in Azure Portal: *SQL Server -> Server Name -> Overview -> Server Name*
2. **Database**: The database name. This can be found in the Azure Portal at the following location: *SQL Server -> Server Name -> Overview -> Available Resources -> (x) databases*.
3. **Database or Schema**: The schema to connect to
4. **User**: The username
5. **Password** : The password
{% hint style="info" %}
Note:
* The User and Password above are for the SQL Authentication.
* In order to connect to a privately hosted instance, the customer will need to whitelist the IPs from Telmai. Please reach out to Telmai support for the updated list of IP’s for whitelisting.
{% endhint %}
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need to select existing table, or run a custom SQL query.
{% hint style="warning" %}
JSON data type is not supported for SQL Server connections out of the box. Please reach out to Telmai support if it needs to be enabled
{% endhint %}
{% hint style="info" %}
Please ensure that your environment has whitelisted [Telmai IP list](/telmai/api-reference/telmai-ip-list).
{% endhint %}
# Salesforce
## Creating a connection
To create a connection to Salesforce, please enter the required fields in the form below:
* Database
* Login Url
* User name
* Password
* Security Token
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need to select existing table, or run a custom SQL query.
# Snowflake
Telmai integrates with Snowflake to monitor data, identifying anomalies such as outliers and drifts while processing data outside of your data warehouse (DW) architecture to reduce the monitoring load on your Snowflake DW.
This guide outlines the steps for integrating Telmai with Snowflake.
## Creating a connection
### **Authentication Options**
Telmai supports only **key pair-based connectivity** to Snowflake, offering enhanced security compared to basic authentication. Below are the detailed steps to integrate Snowflake with Telmai using this method.
**Setting up Authentication**
1. **Role and User Creation**: If you have an existing user with the necessary permissions to access the database, schema, table, or view, skip to Step 2. However, the best practice is to create a separate role and user specifically for Telmai.
* **Role**: Telmai requires permissions for database connections, schema and table listings, metadata retrieval, and data selection in tables and views. These are managed through `SELECT`, `USAGE`, and `MONITOR` privileges in Snowflake.\
For references on creating roles and assigning appropriate privileges, please refer to [1](https://docs.snowflake.com/en/sql-reference/sql/create-role/), [2](https://docs.snowflake.com/en/sql-reference/sql/grant-privilege/), and [3](https://docs.snowflake.com/en/user-guide/security-access-control-privileges/)
* **User**: Create a user account for Telmai to access Snowflake. Assign the user to the role created above. Detailed steps for creating a user can be found [here](https://docs.snowflake.com/en/user-guide/key-pair-auth/).
2. Generate an unencrypted key pair and assign it to the user following the instructions [here](https://docs.snowflake.com/en/user-guide/key-pair-auth/)
3. Assign the default role and warehouse for the user as described [here](https://docs.snowflake.com/en/sql-reference/sql/alter-user/)
4. Once the role, user, and key pair are configured, enter the following parameters in the Telmai Connection Wizard:
* Snowflake Account
* Snowflake Warehouse
* Snowflake Database
* Snowflake Schema
* Snowflake Username
* Snowflake Private Key
**Snowflake Configuration Template Script** To simplify configuration, you can use a Telmai Template Script, enter your parameters, and execute in the Snowflake console.
1. Generate key pair as described [here](https://docs.snowflake.com/en/user-guide/key-pair-auth/)
2. Once the key is generated, format it to create a single-line Base64 string for usage. If your public key file is rsa\_key`.pem`, use the commands below to produce a single-line string.
```bash
// For mac OS
grep -v "PUBLIC KEY" rsa_key.pub | tr -d '\n' > rsa_public_key_base64.txt
grep -v "PRIVATE KEY" rsa_key.p8 | tr -d '\n' > rsa_private_key_base64.txt
```
```powershell
// For Windows
(Get-Content rsa_key.pub | Where-Object {$_ -notmatch 'PUBLIC KEY'}) -join '' |
Set-Content -NoNewline rsa_public_key_base64.txt
(Get-Content rsa_key.p8 | Where-Object {$_ -notmatch 'PRIVATE KEY'}) -join '' |
Set-Content -NoNewline rsa_private_key_base64.txt
```
3. Create role for Telmai to read the data.\
The script below lists the commands to be run. Fill in the right values for the fields which are marked.
```sql
-- Create variables for user / password / role / warehouse / database (needs to be uppercase for objects)
-- The following part also needs to be found and changed in script to include your generated public key: "set rsa_public_key = 'MI...'"
set role_name = 'TELMAI_ROLE';
set user_name = 'TELMAI_USER';
set user_password = 'P@SSw0rd'; -- Change this password
set warehouse_name = 'COMPUTE_WH'; -- Change to your warehouse name
set database_name = 'TESTDB'; -- Change this database name to your database
set db_schema_name = 'TESTSCHEMA'; -- Change this schema name to your schema
set db_table_name = 'SUPPLIER'; -- Change this table name to your table
-- Change role for user / role steps
use role accountadmin;
-- Create role for telmai
create role if not exists identifier($role_name);
-- Create a user for telmai
create user if not exists identifier($user_name)
password = $user_password
default_role = $role_name
default_warehouse = $warehouse_name;
-- Set public key for user
alter user identifier($user_name) set rsa_public_key = 'MI...';
grant role identifier($role_name) to user identifier($user_name);
-- Grant telmai role access to warehouse
grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name);
grant USAGE on database identifier($database_name) to role identifier($role_name);
grant USAGE on schema identifier($db_schema_name) to role identifier($role_name);
grant SELECT on table identifier($db_table_name) to role identifier($role_name);
-- The following statement if all schema tables need to be accessible from telmai
grant SELECT on ALL TABLES in schema identifier($db_schema_name) to role identifier($role_name);
```
4. Run the updated script in your Snowflake console.
### **Whitelisting Telmai IP’s**
In some cases, Snowflake's security requires whitelisting the IPs from which connection to the database is allowed. For this purpose, Telmai provides a list of static IPs from which connections can be made. Please refer to [Telmai IP list](/telmai/api-reference/telmai-ip-list).
Please make sure you are whitelisting those IP addresses. These IPs are for the SaaS version of the product, which is running in the US West. They will be provided separately upon request for the private cloud or deployment in other regions.
Snowflake instances are open to every IP address by default, so no action is required. However, if you have set up network policies to restrict the IP addresses communicating with the Snowflake instance, you'll need to modify these policies to allow the Telmai IP address. There are two types of network policies:
* [Account level](https://docs.snowflake.com/en/user-guide/network-policies#modify-an-account-level-network-policy/): Apply to all users unless overridden by a user-level policy.
* [User level](https://docs.snowflake.com/en/user-guide/network-policies#modify-a-user-level-network-policy/): Apply only to specific users and override account-level policies.
Please refer to the Snowflake documentation for more information on modifying network policies.
### Create a connection
Once connection prerequisites are completed, you can add the connection details to Telmai. You will need to provide:
* Account identifier (`-
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need:
* Database name
* Next step will show you available tables
* Custom SQL Query
Once defined, you will be able to see your data asset in Telmai.
# Trino
## Creating a connection
To create a connection to Trino server, please enter the required fields in the form below:
* Database
* Server
* Port
* Catalog
* UseSSL (true/false)
* AuthScheme (true/false)
* Authentication: User and password
## Connecting an asset
Once a connection is defined, you can start using it to create assets. To create assets, you will need to select existing table, or run a custom SQL query.
# Supported File Types
When your connection is storage-based (e.g., cloud storage services), you can upload the following file formats:
* **Data Formats:**
* CSV
* JSON
* Parquet
* Avro
* Delta Lake
* **Document Formats:**
* PDF (GCP Support Only)
* XML
* **Automatic File Type Detection**\
There is also an Automatic file type option. When selected, Telmai attempts to determine the file type by inspecting the file extension.
{% hint style="info" %}
Reading PDF files requires enabling Gemini in your GCP workspace
{% endhint %}
# Connection Modes
In the "Connect to Data" dialog, Telmai offers several connection modes for SQL-based sources such as [BigQuery](/telmai/connect-to-data/data-connections/google-bigquery), [Redshift](/telmai/connect-to-data/data-connections/aws-redshift), [Snowflake](/telmai/connect-to-data/data-connections/snowflake), and others.
It can either connect to an existing table/view or allow you to define an SQL query and analyze the output.
### Connect to existing Table/View
Select the **“Existing Table“** radio button, and pick an option in the “Create“ dropdown:
* **Single Source:** Configure scans individually with custom schedules and settings, including metadata vs. deep scans.
* **New Group:** Scan multiple tables together for efficiency, using the same Spark cluster and optimized algorithms. However, group sources share the same schedule and scan settings, including metadata vs. deep scan configurations.
#### Single Source Connection
Select **Single Source** and click **Next**.
Choose the table you want to connect to. You can keep the default name or rename the data source as needed.
#### New Group Source Connection
Grouping datasets allows you to connect to multiple tables at once, simplifying configuration:
1. Reuse same credential to simplify configuration
2. Enable a shared scan schedule and settings.
3. Optimize scanning for increased throughput, although metadata vs. deep scan settings will apply uniformly across the group.
Select **New Group**, name it, and click **Next**.
Choose tables/views from the same schema/dataset/collection. You can rename the Telmai data source names if desired. Click **Create** to complete the setup.
Once the Group Source is created, you can now edit properties of the group or individual properties of each data source:
### Query Based Connection
Instead of connecting to an existing table or view, you can define a data source using an SQL query. This is useful for applying transformations or joining multiple tables to monitor the output.
1. **Select “Custom Query”** as the connection mode.
2. Click **Next**, input your query, and click **Validate Query** to ensure it’s correct. Once validated, click **Connect**.
A query-based data source functions like any other integration, allowing you to schedule scans, monitor drifts, and apply rules. Note: This option is currently available only for SQL databases like BigQuery, Databricks Delta, etc.
**Best Practices:**
1. **SQL Syntax**: The SQL syntax depends on the underlying system. The queries are pushed to the database. Always refer to the specific database syntax for functions, operators, etc, to ensure they are supported
2. **Full Table/View Names**:Use the full name (e.g., `schema.table`, `project.dataset.table`, `database.schema.table`, or `catalog.database.table`) for all tables in the query
3. **Permissions**: Ensure the service account has permissions for all tables/views used in the query
4. **Named Projections**: Use named projections (e.g., `SELECT column1 + column2 AS total FROM table1`) to capture the schema of the results correctly.
5. **Time Zone**: All queries are executed in the UTC time zone, so adjust any time/date manipulations accordingly.
# File Path Options
For GCS and Amazon S3, users can specify either a single file name as input or provide a folder containing multiple files as input.
#### Single file
If sending a single file, then specify the full path of a json/csv/parquet file inside the bucket.
Copy
```
path=’/file.csv’
path=’/file.parquet
path=’file.csv’
```
* Files cannot be compressed
* Files have an extension of csv, json or parquet.
* Csv files should have the header line.
Note:
* The JSON file should be in *Newline Delimited JSON* format - with .json extension.
* Column Headers in Parquet file should not contain any spaces.
#### Folder
If a folder contains multiple files that are to be used as input, then specify the path of the folder inside the bucket, and ensure
* Path does not have a trailing slash
* All files in the folder have the same extension, either csv/json/parquet.
* All csv files should have the same header line
If folder2 contains all input files, then
Copy
```
path=”/”
```
#### Wildcard Support
You can use a `*` in your file path to match your file path. To enable wildcard you need to select checkbox for "**Resolve wildcards (\*) to folder names**"
For example: `//log*.csv` will match paths like `/home/user/log_1.csv` or `/home/user/logs.csv`.
👉 **Important:** Currently only **one** `*` is supported in the path.
# Triggering Scans
#### Trigger job using Telmai UI
1. Navigate to **Assets** page
2. Click the 3 dot options tab for the corresponding data source
3. A window will be prompted, navigate to Scan->Data
#### Trigger job using Telmai API
Please refer to the [Upload Data API](/telmai/api-reference/upload-data-api) documentation.
#### Scheduling Scans
To schedule a scan, click on the **“Add Schedule”** option in the context menu and complete the form. Once created, the scan will automatically run at the specified time.
Scans can be scheduled or run on demand. You can trigger a scan at any time using the UI or API.
# Configuring a Data Source
After creating a Data Source, you can optionally configure advanced settings to tailor its behavior.
1. Configure advanced parameter attributes such as ID, Filters, Segmentation, and Scan Type.
2. Set the scan schedule according to your requirements.
To access these options, click the 3-dot icon next to the Data Source name:
### Advanced Parameters
1. Click Edit Connection in the context menu
2. Navigate to the **Advanced** tab to begin configuration.
#### Filter Attribute
Use this option to monitor only a subset of data that meets specific criteria. Provide the **Filter Attribute Name** and a list of values. Telmai will analyze only the records whose attribute values match those in the list.
#### Segmentation Attribute
Monitor data from different segments independently by specifying a **Segmentation Attribute**. You can list values to define segments. For example, setting segmentation based on `Address.State` might define segments for `CA`, `TX`, and others. In case of null values you can define it as `__null__`. Telmai will profile data, calculate metrics, and detect anomalies for each segment as well as for the entire dataset.
#### Id Attribute
The **ID Attribute** is optional but provides additional insights, including:
1. Tracking the uniqueness of records based on IDs.
2. Displaying record IDs for every anomalous value.
3. Splitting good records from bad ones using the Data Binning feature.
ID may be derived from a single attribute or a multi-attribute set. When a multi-attribute configuration is utilised, the resulting ID is constructed through the concatenation of all selected attribute values.
#### Timestamp Attribute
The **Timestamp Attribute** is also optional but can be valuable for identifying newly added records, enabling Delta-only scans.
When the **Timestamp Attribute** is provided and the **Delta Only** checkbox is checked, Telmai will fetch and scan only new records. Note that for some sources, like files or Databricks Delta, Delta detection occurs natively and does not require this attribute.
# Managing Assets
The **Asset Manager** is the central place to view, filter, and manage all of your assets. From here you can search for individual assets, run or schedule scans, move assets between projects, configure monitors, and inspect each asset's scan configuration and lineage.
{% hint style="info" %}
The Asset Manager is the newer assets experience. Navigate to **Asset Manager** page using the left-side menu.
**Note:** This feature is disabled by default. To enable it, navigate to **Administrator page** and enable `Show Asset Manager page` toggle
{% endhint %}
Open the Asset Manager from the left-side menu.
## Finding Assets
Use the filters at the top of the page to narrow down the asset list:
* **Project** — filter assets by the project they belong to.
* **Connection type** — filter by connection type (for example, Redshift, Athena, Azure).
* **Connection** — filter by a specific connection.
* **Search** — use the **Search asset by name or ID** box to find a specific asset.
The Asset Manager page, with assets grouped by project.
Assets are grouped by project, and each project group shows a summary of how many assets and open incidents each asset contains. The asset table includes the following columns:
| Column | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Asset | The asset name and its source type icon. |
| Connection | The connection the asset belongs to. |
| Schedule | The asset's scan schedule, or `N/A` if no schedule is set. |
| Last scan | When the asset was last scanned. A green check indicates a successful scan; a red indicator marks a failed scan or one with incidents; a warning icon indicates the asset has never been scanned. |
| Incidents | The number of open incidents for the asset. |
| Actions | Per-asset actions: run a scan, edit the asset, or delete it. |
## Adding and Managing Assets
* **Add Asset** — add a new asset to the selected connection.
* **Manage Projects** — create and organize the projects used to group assets.
* **Configure Lineage** — set up lineage relationships between assets.
## Bulk Actions
Select assets using the checkboxes in the asset list to perform actions on multiple assets at once. You can select up to **10 assets** for bulk actions. Once one or more assets are selected, a bulk action toolbar appears above the list with the following options:
* **Start scan** — trigger a scan for all selected assets.
* **Schedule** — set or update a scan schedule for all selected assets.
* **Move project** — move all selected assets to a different project.
* **Delete** — delete all selected assets.
The bulk action toolbar appears above the list once assets are selected.
## Asset Details
Selecting an asset opens a details panel with the following tabs:
* **Overview** — shows incidents from the last 30 days, the status of the most recent scan job, and the asset's scan configuration (ID Attributes, Timestamp, Delta Only, and Scan Mode). You can also start a **Data Comparison** to compare this asset's inspectors with another asset.
* **Attributes** — view the asset's attributes.
* **Settings** — edit the asset's scan configuration and other settings.
* **Lineage** — view the asset's lineage relationships.
From the bottom of the panel you can **Delete Asset**, **Configure Monitors**, or **Run Scan**.
The asset details panel, showing the Overview tab.
# Profiling Data
Before starting data profiling or monitoring, Telmai requires you to specify which attributes need to be monitored. When a data source is created, the schema is analyzed (excluding nested JSON attributes). By default, all attributes are disabled, so the first step is to enable the required ones:
1. Navigate to the **Configuration Page**
2. Select the desired **Data Source**
3. Click the **lightning icon** next to the attributes you want to profile. Alternatively, click **“Select All”** to enable all attributes
Open the context menu for the same data source and select **“Scan”**
You have two scan options:
* **Single Data Scan**
* **Historical Data Scan**
#### Single Data Scan
This option launches a job to scan the current data, which can be tracked and stopped if needed:
#### Historical Data Scan
For SQL-based sources with a configured schedule, you can perform a Historical scan. This option automatically breaks down records added over the past 10 periods and launches 10 independent scans for each period, allowing you to analyze historical data dynamics and metric behaviour over time.
For historical scans to work, the Timestamp Attribute (set under **Edit Connection → Advanced → Timestamp Attribute**) must be configured at the source. The column's data type should match the appropriate type for the specific database.
| Database | Timezone Data Type | Date Data Type |
| ---------- | ------------------------ | -------------- |
| BigQuery | TIMESTAMP | DATE |
| Snowflake | TIMESTAMP\_TZ | DATE |
| Iceberg | TIMESTAMP WITH TIME ZONE | DATE |
| Redshift | TIMESTAMPTZ | DATE |
| Athena | timestamp with time zone | date |
| Databricks | TIMESTAMP | DATE |
| SQL Server | DATETIMEOFFSET | DATE |
| Trino | timestamp with time zone | date |
# Data Health Metrics
Telmai automatically calculates a comprehensive set of health metrics to monitor your data. These metrics include:
| Column | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *Total Record Count* | The total size of the monitored table. This is only available when CDC is enabled (i.e., the “delta only” flag is on). If the flag is off or the monitored source is not an SQL database, this field will be set to N/A. |
| *Record Count* | Reflects the size of the delta when CDC is enabled; otherwise, it represents the size of the entire dataset. |
| *Completeness* |
Percentage of non-null/missing/placeholder values, tracked at both the data source and attribute levels:
Attribute level: Percentage of records where the attribute is not null, empty, or a placeholder (e.g., N/A)
Data Source level: Compounded average of attribute-level completeness within a data source
|
| *Correctness* |
Correctness is tracked both at the data source and attribute levels. Correctness is calculated only for attributes where Expectations are set, otherwise, the default is 100%.
Attribute level: Percentage of records where the attribute meets all set expectations.
Source level: Compounded average of attribute-level correctness within a data source.
|
| *Freshness* |
Freshness is tracked both at data source and record level.
Record level: Determined by setting an expectation on a timestamp attribute (e.g., “Record Update Date” should be no more than one month old). Configure this in the Advanced section of the Edit Connection menu.
Table level:Based on the time since the table was last updated.
|
| *Uniqueness* |
Uniqueness of the records based on an ID attribute. The KPI measures the ratio of records with unique id versus the total number of records. For example, if out of 10 records, there are 2 records sharing the same value the uniqueness is 80%.
An attribute can be marked as an ID attribute in the Advanced section of the Edit Connection menu. If ID attribute is not configured this KPI will be N/A
|
| *Accuracy* |
The accuracy of the values is determined through analysis of historical data. It detects discrepancies when current values deviate from predictions.
For example, if the revenue for the company Acme Inc was slowly growing from $4M to $5M over the past year, but in today’s observation it’s 20M, then such a value is considered inaccurate. Accuracy is only calculated for attributes that have “Custom Metrics” Configured, otherwise, the default is 100%.
Attribute level: Ratio of dimensions with no drift detected to the total number of dimensions. If multiple custom metrics are defined, the minimum accuracy is used.
Data Source level: Compounded average of attribute-level accuracy within a data source.
|
These metrics can be found in different places across the application, depending on the scope you’re looking at. In case [lightweight scan](#lightweight-scans) is enabled, only a subset of these metrics are measured based on the source type.
### Lightweight Scans
Telmai allows you to scan only a table’s metadata instead of scanning the whole table. This helps you monitor high level information about your table’s health, and still allow you to alert for quality issues.
#### When to use?
This configuration tells Telmai to only check the table metadata rather than scanning the whole dataset. If the asset is SQL based and user have defined SQL-based metric, these queries will also be run and validated against corresponding policies.
To enable this feature, just check the “Metadata Only Scan” option under advanced options for the corresponding asset. In case of group, the option will be under “Group Settings”.
{% hint style="info" %}
Note: This feature is only supported for BigQuery, Snowflake and Databricks. Future Telmai versions will support more sources.
{% endhint %}
# Data Health Overview Page
The **Dataset Health Overview** page provides insights into a dataset's status and attributes during a specific scan.
On this page, you can see the Health KPIs defined in [Calculated Health Metrics](/telmai/profiling-data/data-health-metrics). You can also see attribute-level metrics:
| KPI | Description |
| ------------- | ------------------------------------------------------------------------------- |
| *Duplicates* | Number of records with duplicate values in the attribute |
| *Unique* | Number of records with unique values in the attribute |
| *Distinct* | Count of distinct values in the attribute |
| *Empty* | Count of empty values, including nulls or empty strings, in the attribute |
| *Cardinality* | The cardinality of values in the attribute, indicating whether it's low or high |
Note: For all attribute-level metrics, table-level metrics are calculated as the average across different columns.
**Selector Component**
The **Selector Component** allows you to customize the data view by selecting:
* **Dataset**: Choose the dataset you want to analyze
* **Attributes**: Select the attributes to calculate metrics on
* **Segments**: Filter the data based on specific segments
* **Scan Time**: Select the time of the data scan
**Health KPIs Summary**
The **Health KPIs Summary** provides a quick overview of key Data Quality KPIs at both the table and attribute levels.
**Attribute(s) KPIs**
For each attribute (or column), Telmai calculates and displays the corresponding profiling metrics.
# Interactive Profiling Tool: Investigator
The **Investigator** page allows you to drill down into your dataset for a deeper analysis, similar to the Overview page
It includes a Selector Component for easy navigation and filtering.
Other components can be seen on this page:
**Patterns**
This section shows the distribution of various value patterns. This can be used to understand the data and build more data validations, like enforcing specific patterns for product identifiers.
Telmai automatically calculates patterns for all attribute values. There are two kinds of patterns calculated, each of which may be useful based on the type of the data:
* Compressed Patterns - Replaces sequences of letters (L) or digits (D) with a single character
* Expanded Patterns - Replace each letter or digit with L or D, respectively
**Values**
Shows the distribution of occurrences for the top values in an attribute.
**Drill Down**
Provides a sample of the dataset’s top values with similar properties. Clicking on any value in this table reveals more details about its properties. This feature can also help identify data quality (DQ) validation violations.
This table can also be used to understand DQ validation violations.
In this table, you can find the following columns per group/cohort of values
| Column | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Value | Sample data with similar characteristics |
| Expectation Violation | Indicates if the value violates a correctness expectation |
| Failing Expectation | The associated correctness rule (only valid for violations) |
| Values Count | Number of times the value appears within its attribute |
| Expanded Pattern | Anomaly score, based on value’s pattern, i.e. representation of the value string but with each character that belongs to one of these types: alphabet/letter (L), digit (D), or space (S) being replaced by the character representing the type. A long sequence of the same type character is represented by the character and a number indicating the length of the sequence |
| Compressed Pattern | Anomaly score, based on value’s short pattern, i.e. representation of the value string but with each sequence of characters that belong to one of these types: alphabet/letter (L), digit (D), or space (S) being reduced to the single character representing the type |
| Frequency | Anomaly score based on how frequently the value occurs. A score of 1 is normal; 0 is abnormal. |
| Length | Anomaly score based on the number of characters in the value; 1 is normal, 0 is abnormal. |
| Special Characters | Anomaly score based on the presence of special characters in the value; 1 is normal, 0 is abnormal. |
| Spaces | Anomaly score based on the number of whitespace characters; 1 is normal, 0 is abnormal. |
| Is Date | True if value is ISO date, false otherwise |
| Is DateTime | True if the value is a datetime, false otherwise. |
| Is Number | True if the value is numeric, false otherwise. |
| Is Alpha | True if the value consists of alphabetical characters, false otherwise. |
# Data Diff
Telmai allows you to compare the differences between your datasets. This feature is useful when you want to ensure data consistency across different tables, and can be used in data reconciliation or migration cases.
As part of scanning a dataset, Telmai checks for the differences between source and target tables (if configured). In the app, Telmai will provide summary stats on the number of new, missing and changed records. Outside the app, Telmai will store the changed or different records for further usage or analysis.
Telmai’s Data Diff feature runs as part of tables’ regular scan. You will first need to connect the two datasets you want to compare to Telmai using the steps defined in [Connect to Data](/telmai/connect-to-data). Then, you will need to update the configs for the table you want to compare (target table) using these steps:
1. Define the ID Attribute
2. From the 3 dot menu, click “Data Comparison” option
3. You will be prompted to fill details:
1. Source table: Dataset you want to compare to
2. Result Destination: Output for parquet files (S3, Azure Blob, or GCP storage)
4. Once the details is selected, you will be prompted to fill more details on associated bucket
5. Next scan will analyze the deltas between both datasets, and alerts will be created if deltas exist
**Data Diff Alert Example**
If any differences are detected across the source and target datasets, a “Data Difference” alert is created similar to below picture:
Clicking on the alert, will show more details on changed schema and records similar to picture below:
Lastly, navigating to the output parquet files, you can see more details on changed records, such as individual changes per each record.
**Note:** The Data Diff feature is not supported for assets belonging to a group.
# Compound Attributes
Telmai offers a **Compound Attributes** feature, which allows you to define an SQL-like expression, validate it in real-time, and inspect outputs similar to single-attribute rules.
#### Adding a Compound Attribute
1. Navigate to the **Configuration** page and select the corresponding dataset.
2. Click the "**Add Compound Attribute"** button
3. **Define the Attribute**:
* **Attribute ID**: Name the compound attribute.
* **Attributes to Compound**: List the attributes to be used.
* **Expression**: Enter an SQL-like expression that defines the transformation.
4. Enter an expression. Please see below for additional information on supported functions and examples of expressions
You can validate the expression output by entering values for each corresponding attribute. When clicking on the edit field, Telmai will offer a few samples taken from the actual data to help
Once the attribute is created, you can add an expectation to it the same way you would for a standard attribute — by navigating to the **Monitoring** page for the dataset and configuring a threshold on the compound attribute.
**Important Usage Notes**:
* As the user inputs an expression, an evaluation is done to ensure a valid SQL expression
* There are some cases where the validation will pass but will fail during executions due to casting-related errors. The values should be cast appropriately to the expected type to mitigate this. The expression output must always be a string type. In case it is not, users need to cast to a string explicitly. Example:
* If the expression has contains function; the output is boolean `contains( last_name , 'Smith')`
The user needs to cast the value string `cast ( contains( last_name , 'Smith') as string)`
* All parameters are treated as strings. The user needs to cast the parameters to the corresponding function param type. Example:
* If the user wants to use the date\_add function, which takes in number and a date (which is a string) The user will need to cast `my_number` first like this: `date_add('2016-07-30', cast (my_number as int))`
**Examples**
**Example 1:**
```sql
LENGTH(CONCAT_WS(\",\", firstname, lastname)) > 100
```
The output expression is boolean, so it needs to be converted to string in order to become a value of a new compound attribute:
```
CAST(LENGTH(CONCAT_WS(\",\", firstname, lastname)) > 100 as STRING)
```
**Example 2:**
```
IF( DATEDIFF(current_timestamp(), orderdate) > 10 , true , false)
```
Convert boolean to string values:
```
IF( DATEDIFF(current_timestamp(), orderdate) > 10 , 'true' , 'false' )
```
**Example 3:**
```
IF( ISNOTNULL(country) and ISNOTNULL(city) and (price <= 100)), 'true', 'false')
```
price is an attribute, so it’s value comes as a string, so it needs to be first cast to double
```
IF( ISNOTNULL(country) and ISNOTNULL(city) and (CAST(price as DOUBLE) <= 100)),'true', 'false')
```
# List of Supported Functions
Below is a list of whitelisted functions:
```
// misc non-aggregate functions
abs,
coalesce,
explode,
explode_outer,
greatest,
if,
inline,
inline_outer,
isnan,
ifnull,
isnull,
isnotnull,
least,
nanvl,
nullif,
nvl,
nvl2,
posexplode,
posexplode_outer,
rand,
random,
randn,
stack,
when,
// math functions
acos,
acosh,
asin,
asinh,
atan,
atan2,
atanh,
bin,
bround,
cbrt,
ceil,
ceiling,
cos,
sec,
cosh,
conv,
degrees,
e,
exp,
expm1,
floor,
factorial,
hex,
hypot,
log,
log10,
log1p,
log2,
ln,
mod,
negative,
pi,
pmod,
positive,
pow,
power,
radians,
rint,
round,
shiftleft,
shiftright,
shiftrightunsigned,
sign,
signum,
sin,
csc,
sinh,
str_to_map,
sqrt,
tan,
cot,
tanh,
width_bucket,
try_add,
try_divide,
try_subtract,
try_multiply,
try_element_at,
try_avg,
try_sum,
try_to_binary,
// string functions
ascii,
char,
chr,
contains,
startswith,
endswith,
base64,
bit_length,
char_length,
character_length,
concat_ws,
decode,
elt,
encode,
find_in_set,
format_number,
format_string,
to_number,
try_to_number,
get_json_object,
initcap,
instr,
lcase,
length,
levenshtein,
like,
ilike,
lower,
octet_length,
locate,
lpad,
ltrim,
json_tuple,
parse_url,
position,
printf,
regexp_extract,
regexp_extract_all,
regexp_replace,
repeat,
replace,
overlay,
rlike,
regexp_like,
regexp,
rpad,
rtrim,
sentences,
soundex,
space,
split,
split_part,
substr,
substring,
left,
right,
substring_index,
translate,
trim,
btrim,
ucase,
unbase64,
unhex,
upper,
xpath,
xpath_boolean,
xpath_double,
xpath_number,
xpath_float,
xpath_int,
xpath_long,
xpath_short,
xpath_string,
// datetime functions
add_months,
current_date,
current_timestamp,
current_timezone,
datediff,
date_add,
date_format,
date_sub,
day,
dayofyear,
dayofmonth,
from_unixtime,
from_utc_timestamp,
hour,
last_day,
minute,
month,
months_between,
next_day,
now,
quarter,
second,
to_timestamp,
to_date,
to_binary,
to_unix_timestamp,
to_utc_timestamp,
trunc,
date_trunc,
unix_timestamp,
dayofweek,
weekday,
weekofyear,
year,
window,
session_window,
make_date,
make_timestamp,
make_interval,
make_dt_interval,
make_ym_interval,
extract,
date_part,
date_from_unix_date,
unix_date,
timestamp_seconds,
timestamp_millis,
timestamp_micros,
unix_seconds,
unix_millis,
unix_micros,
// collection functions
concat,
// misc functions
assert_true,
raise_error,
crc32,
md5,
uuid,
hash,
xxhash64,
sha,
sha1,
sha2,
aes_encrypt,
aes_decrypt,
spark_partition_id,
input_file_name,
input_file_block_start,
input_file_block_length,
monotonically_increasing_id,
current_database,
current_catalog,
current_user,
reflect,
java_method,
version,
typeof,
// bitwise
bit_count,
bit_and,
bit_or,
bit_xor,
bit_get,
getbit,
// json
to_json,
from_json,
schema_of_json,
json_array_length,
json_object_keys,
// cast
cast,
// Cast aliases
boolean,
tinyint,
smallint,
int,
bigint,
float,
double,
decimal,
date,
timestamp,
binary,
string,
```
# Monitoring Data
Trends and Alerts view — monitor scan results with triggered alerts.
Monitors are policies that define the expected state or behaviour of your data. When data deviates from these policies, alerts are automatically triggered to notify your team. Monitors help you proactively manage data quality by continuously evaluating metrics, rules, and custom queries against your data assets.
## Key Concepts
* A **Monitor** is a policy that defines the expected state or behaviour of your data. It continuously evaluates metrics, rules, or custom queries against your data assets. When data deviates from the defined policy (violates a threshold), the monitor generates an **alert.**
* An **Alert** is the initial notification that a system's metric has violated a predefined threshold. It's an automatic signal that something is amiss, such as a spike in CPU usage or a high number of transaction failures.
* An **Incident** is a broader event that is triggered by one or more alerts. It represents an ongoing issue that requires investigation and resolution. A single incident may contain multiple, related alerts that help tell the story of the problem.
Think of it this way: a single data point violating a threshold generates an Alert. When multiple alerts occur that are related to the same problem, they can be grouped into a single Incident to provide a consolidated view of the issue.
Once an asset is connected to Telmai, Telmai will start monitoring the asset's health with the [out-of-box monitors](/telmai/monitoring-data/out-of-box-monitors). At any time, users can build and [customize their own monitors](/telmai/monitoring-data/user-defined-monitors).
If alerts and incidents are generated, user can navigate to the [Incident Portal ](#incidents-portal)to learn more about the underlying issues.
## Related Documentation
* [Out-of-box Monitors](/telmai/monitoring-data/out-of-box-monitors)
* [User-defined Monitors](/telmai/monitoring-data/user-defined-monitors)
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Incident Portal](/telmai/monitoring-data/incident-portal)
* [Data Trends and Alerts](/telmai/monitoring-data/data-trends-and-alerts)
# Out-of-box Monitors
Alerting Monitors page — all prebuilt monitors available for a connected asset.
Out-of-the-box monitors are automatically enabled when you connect a data source to Telmai. These monitors provide immediate visibility into your data quality by tracking essential metrics across your tables and datasets. No additional setup is required. Telmai begins collecting and analyzing these metrics as soon as your asset is connected.
## How Out-of-the-Box Monitors Work
1. **Automatic Availability**: All prebuilt monitors are available immediately upon asset connection
2. **Baseline Learning**: Telmai establishes normal patterns in your data over initial monitoring cycles
3. **Drift Detection**: Monitors continuously compare current metrics against learned baselines
4. **Alert Generation**: When drifts or anomalies exceed configured thresholds, alerts are triggered
5. **Continuous Adaptation**: Baselines adapt over time to account for legitimate data evolution
## Available Out-of-the-Box Monitors
### **Record Count Drifts**
Detects unexpected changes in the number of records scanned during monitoring cycles. This monitor identifies unusual spikes or drops in data volume.
**Use Case**: Monitor data volume trends and catch pipeline issues, data loss, or unexpected duplicate records.
### **Record Count Correlation Drifts**
Tracks relationships between record counts across related tables or datasets. Alerts when correlations between volumes deviate from expected patterns. This monitor requires assets lineage to be defined in Telmai first.
**Use Case**: Detect cascading pipeline failures or referential integrity issues affecting multiple related tables.
### **Freshness Drifts**
Tracks changes in data freshness patterns by monitoring the time elapsed since the last table update. Alerts trigger when data becomes stale or update frequencies deviate from predicted patterns.
**Use Case**: Detect when upstream data pipelines experience delays, ensuring your data remains current for analytics and reporting.
### **Completeness Drifts**
Detects unexpected changes in the percentage of complete (non-null, non-empty) values across your data attributes. This monitor alerts you when data completeness patterns deviate from established baselines.
**Use Case**: Identify when upstream data sources begin sending incomplete records, ensuring data reliability for downstream consumers.
### **DQ Rules Violation**
Monitors compliance with defined data quality rules and business logic. This monitor tracks when records fail to meet validation requirements and correctness standards.
**Use Case**: Ensure data accuracy by detecting violations of business rules, data formats, and expected value ranges.
### **Record Id Uniqueness Drifts**
Monitors changes in the uniqueness of configured ID attributes. Detects when duplicate identifiers appear or uniqueness percentages drift from baseline patterns.
**Use Case**: Catch duplicate primary keys that could cause issues in downstream systems or violate database constraints.
### **Uniqueness Drifts**
Tracks changes in the percentage of unique values within specific attributes at the column level.
**Use Case**: Monitor attributes that should contain unique values (such as email addresses, account numbers, or transaction IDs) to detect data quality degradation.
### **Uniqueness Correlation Drifts**
Monitors relationships between uniqueness metrics across different attributes or tables, detecting when correlations deviate from normal patterns. This monitor requires assets lineage to be defined in Telmai first.
**Use Case**: Identify systemic data quality issues affecting multiple related attributes simultaneously
### **Schema Drifts**
Detects changes to table schemas including added columns or removed columns.
**Use Case**: Track schema evolution and catch breaking changes that could impact downstream applications or analytics queries.
### **Data Drifts**
Monitors overall statistical distributions of data values, detecting when data patterns shift from established baselines.
**Use Case**: Identify fundamental changes in data characteristics that may indicate upstream process changes or data quality issues.
### **Numeric Value Drift**
Tracks changes in numeric value distributions, including mean, median, standard deviation, and other statistical properties.
**Use Case**: Detect anomalies in numeric attributes like revenue figures, quantities, or performance metrics.
### **Top Values Drifts**
Monitors changes in the most frequently occurring values within attributes, detecting shifts in categorical data distributions.
**Use Case**: Identify when dominant categories change unexpectedly, such as sudden shifts in product types, user segments, or geographic distributions.
### **Value Distribution Drifts**
Tracks comprehensive changes in value distributions across all data types, including frequency patterns and outlier detection.
**Use Case**: Monitor complex distribution shifts that might not be captured by simple statistical measures.
### **Data Difference**
Compares data between two sources or time periods to detect discrepancies and changes. Compared-to table must be defined first to see alerts on this table.
**Use Case**: Validate data migrations, compare production vs. staging environments, or track changes between data refreshes.
### **PII Exposure**
Detects potential exposure of Personally Identifiable Information (PII) in datasets where sensitive data should be masked or encrypted.
**Use Case**: Ensure compliance with privacy regulations by monitoring for unexpected PII in non-production environments or public-facing datasets.
### **Data Process Failure**
This monitor is created to group Telmai-job failures to alert you when Telmai processes are unable to scan the data.
## Related Documentation
* [User-defined Monitors](/telmai/monitoring-data/user-defined-monitors)
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Incident Portal](/telmai/monitoring-data/incident-portal)
# User-defined Monitors
Alerting Monitors — the starting point for creating and managing user-defined monitors.
User-defined monitors are policies that define the expected state or behaviour of your data. When data deviates from these policies, alerts are automatically triggered to notify your team. Monitors help you proactively manage data quality by continuously evaluating metrics, rules, and custom queries against your data assets.
When creating a user-defined monitor, you must attach it to an asset, and define the following components:
1. **Metric:** A quantifiable measure of data health (e.g., row count, null percentage, data freshness).
2. **Threshold:** A predefined value or range that, when exceeded, triggers an alert. Thresholds can be:
1. **Automatic**: ML-based thresholds that analyze historical datapoints to determine dynamic boundaries
2. **Relative**: Percentage-based boundaries that calculate moving averages from historical data
3. **Absolute**: Constant, fixed boundaries
3. **Notification endpoints:** Which endpoints to notify when the monitor is alerting (Optional)
***
## Monitor Types
Telmai supports three primary monitor types, each designed for different data quality scenarios:
1. [Built-in Metric](#built-in-metric)
2. [User-Defined Metric](#user-defined-metric)
3. [Record Validation Rule](#record-validation-rule)
4. [SQL Based Rule](#sql-based-rule)
### Built-in Metric
Monitor [built-in metrics](/telmai/monitoring-data/user-defined-monitors/built-in-metrics) managed by the Telmai platform.
**Use cases:**
* Track predefined data quality metrics (row count, null percentage, data freshness, completeness, etc.)
* Monitor standard data health indicators across your datasets
* Leverage out-of-the-box metrics without custom configuration
**Example:**
* Monitor `record_count` to ensure tables are being populated
* Check `freshness` to detect data delays
### User-Defined Metric
Define and monitor custom metrics using [Telmai expressions ](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics#metric-expressions)or [push-down custom SQL](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics#raw-sql-queries).
**Use cases:**
* Create custom calculations and aggregations
* Monitor business-specific KPIs
* Define complex metric logic using SQL queries
**Example expression:**
```sql
SUM(salary)/COUNT(*)
```
**Example custom SQL:**
```sql
SELECT emp_salary, emp_Region FROM `employee_table` WHERE emp_Age > 60
```
### Record Validation Rule
Define record validation checks and monitor the number of records passing these checks.
**Use cases:**
* Validate individual records against business rules
* Ensure data integrity at the row level
* Check data completeness and correctness across records
* Monitor the percentage of valid records over time
**Example expression:**
```
validate part_date expect is_date
```
The monitor tracks the percentage of records that pass the validation rule, and alerts are triggered when this percentage falls outside acceptable thresholds.
Click [here](/telmai/monitoring-data/user-defined-monitors/record-validation-rules) to learn more about using and creating Data Quality rules.
### SQL Based Rule
Define record validation checks and monitor the number of records passing these checks using a pushdown SQL query.
**Use cases:**
* Validate individual records against business rules that joins multiple datasets
* Ensure data integrity at the row level
* Monitor the percentage of valid records over time
**Example query:**
```sql
SELECT
CASE WHEN transaction_amount > 0 THEN 1 ELSE 0 END AS is_valid,
transaction_id AS record_id
FROM transactions
```
The monitor tracks the percentage of records that pass the validation rule, and alerts are triggered when this percentage falls outside acceptable thresholds.
Click [here](/telmai/monitoring-data/user-defined-monitors/sql-based-rules) to learn more about using and creating SQL based rules.
***
## Monitor Properties
Each monitor has the following properties:
| Property | Description | Required |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------- |
| **Monitor Name** | Must be unique per asset (e.g., "Freshness Monitor" on `sales_data` is different from "Freshness Monitor" on `customer_data`) | Yes |
| **Monitor ID** | System-generated unique identifier | Auto-generated |
| **Monitor Type** | Built-in Metric, User-Defined Metric, Record Validation Rule, or SQL-based rule (cannot be changed after creation) | Yes |
| **Description** | Brief description for additional context | Optional |
| **Monitor Tags** | Tags for organizing and categorizing monitors | Optional |
| **Impact** | Severity level of the monitor (Critical, High, Medium, Low) | Optional |
| **Data Quality Metric** | For Built-in Metrics: the specific metric to monitor | Conditional |
| **Attributes** | For metrics that support scope: specific attributes to monitor | Conditional |
| **Threshold Configuration** | Automatic, Acceptable Drift %, or Acceptable Range | Yes |
| **Creator/Editor** | User who created or last modified the monitor | Auto-tracked |
| **Creation/Update Time** | Timestamp of creation and last update | Auto-tracked |
| **History** | Version history of monitor changes | Auto-tracked |
{% hint style="info" %}
**Note:** Monitor type can not be changed once set. To change the type, you must create a new monitor.
{% endhint %}
## Related Documentation
* [Built-in Metrics](/telmai/monitoring-data/user-defined-monitors/built-in-metrics)
* [User-defined Metrics](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics)
* [Record Validation Rules](/telmai/monitoring-data/user-defined-monitors/record-validation-rules)
* [SQL Based Rules](/telmai/monitoring-data/user-defined-monitors/sql-based-rules)
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Incident Portal](/telmai/monitoring-data/incident-portal)
* [Replay Scan](/telmai/monitoring-data/user-defined-monitors/replay-scan)
# Built-in Metrics
Below is the list of metrics calculated in Telmai out-of-box
Metric
Description
Total Record Count
The total size of the monitored table. This is only available when CDC is enabled (i.e., the “delta only” flag is on). If the flag is off or the monitored source is not an SQL database, this field will be set to N/A.
Record Count
If CDC is enabled it reflects the size of the delta, otherwise, it's the size of the entire set of data.
Completeness
Percentage of not null/missing/placeholder values. Completeness is tracked both at the data source level as well as the attribute level:
Attribute level: percent of records where the attribute value is not null, not empty, or not one of the user-defined placeholders like N/A.
Data Source level: the compounded average of all attribute level completeness within a data source.
Freshness
Freshness is tracked both at data source and record level.
Record level freshness is defined by setting an expectation on a timestamp attribute within the data source (e.g.,. “Record Update Date” attribute to be no more than 1 month from now). To mark an attribute as a timestamp use the Advanced section of the Edit Connection menu. If the timestamp attribute is not configured this KPI will be N/A.
Table level freshness is based on the time since the table was last updated
Uniqueness
Uniqueness of the records based on an ID attribute. The KPI measures the ratio of records with unique id versus the total number of records. For example, if out of 10 records, there are 2 records sharing the same value the uniqueness is 80%.
An attribute can be marked as an ID attribute in the Advanced section of the Edit Connection menu. If ID attribute is not configured this KPI will be N/A
Contains PII
Value contains PII pattern for either of
Credit cards
IP Addresses
Social security numbers
Phone numbers
Zip codes
Email
Credit card
Length
String length
# User-defined Metrics
Creating a user-defined metric monitor — select the metric type, write an expression or SQL query, then configure thresholds.
Users can define custom metrics to monitor specific data quality concerns and receive alerts when thresholds are violated. Each monitored metric is validated against its corresponding monitor, which defines thresholds, scope, notification settings, and alerting rules. When a metric violates the defined policy, an alert is automatically generated.
Telmai provides two methods for defining custom metrics:
1. **Metric Expressions** - Simple aggregations with grouping
2. **Raw SQL Queries** - Complex queries with full SQL capabilities
***
## Metric Expressions
Metric expressions enable you to specify simple aggregations and groupings using a straightforward syntax.
#### Syntax Rules
* Attribute names must be wrapped in backticks: `` `attribute_name` ``
* Maximum of 4 dimensions in `group by` clauses
* Supports standard aggregation functions
#### Available Aggregations
in
max
count
avg
sum
distinct
variance
median
stddev
#### Examples
**Simple aggregation:**
```
SUM(`salary`) / COUNT(*)
```
**Aggregation with grouping:**
```
SUM(`sales`) GROUP BY `region`, `country`
```
**Average by multiple dimensions:**
```
AVG(`order_value`) GROUP BY `region`, `customer_type`, `product_category`
```
***
## Raw SQL Queries
For more complex monitoring scenarios, you can write custom SQL queries that return a single metric value along with optional dimensions.
#### Requirements
* **First column**: Must return a numeric value (this is the tracked metric)
* **Subsequent columns**: Used as dimensions for grouping and filtering
* The query runs against the specified data connector
* Not limited to a single table - you can join multiple tables
#### Supported Data Connectors
Raw SQL queries are available for the following connectors:
* BigQuery
* Amazon Athena
* Databricks
* Trino
* Snowflake
* Amazon Redshift
#### Syntax Rules
* Table names must be wrapped in backticks: `` `table_name` ``
* Use valid SQL syntax for your specific connector
* Ensure the first selected column returns a numeric value
* Additional columns become dimensions for the metric
#### Examples
**Basic query with dimension:**
```sql
SELECT emp_salary, emp_region
FROM `employee_table`
WHERE emp_age > 60
```
* **Tracked metric**: `emp_salary`
* **Dimension**: `emp_region`
**Query with multiple dimensions:**
```sql
SELECT AVG(order_value), customer_region, product_category
FROM `orders`
WHERE order_date >= CURRENT_DATE - 30
GROUP BY customer_region, product_category
```
* **Tracked metric**: `AVG(order_value)`
* **Dimensions**: `customer_region`, `product_category`
**Complex query with joins:**
```sql
SELECT COUNT(*), o.region, c.customer_tier
FROM `orders` o
JOIN `customers` c ON o.customer_id = c.id
WHERE o.status = 'failed'
GROUP BY o.region, c.customer_tier
```
* **Tracked metric**: `COUNT(*)`
* **Dimensions**: `region`, `customer_tier`
***
### Best Practices
#### Choosing Between Expressions and SQL
**Use Metric Expressions when:**
* You need simple aggregations on a single table
* Your logic fits within 4 dimensions
* You want a quick, straightforward configuration
**Use Raw SQL when:**
* You need complex joins across multiple tables
* Your logic requires advanced SQL features (CTEs, window functions, etc.)
* You need fine-grained control over the query
* You're working with connector-specific SQL dialects
#### Performance Considerations
* Keep queries efficient to avoid impacting your data warehouse
* Use appropriate filters to limit data scanned
* Consider query execution time when setting monitoring frequency
* Test queries directly in your data warehouse before adding them as monitors
***
### Troubleshooting
#### Common Issues
**Metric expression fails:**
* Verify attribute names are wrapped in backticks
* Check that you're not exceeding 4 dimensions in `GROUP BY`
* Ensure aggregation function is supported
**SQL query returns no data:**
* Verify table names are correct and wrapped in backticks
* Check that your filter conditions return results
* Confirm you have permissions to query the tables
**First column is not numeric:**
* Ensure your first `SELECT` column returns a number
* Use `CAST()` or `CONVERT()` if needed to ensure numeric type
* Aggregations like `COUNT()`, `SUM()`, `AVG()` automatically return numeric values
**Query timeout:**
* Optimize your query to reduce execution time
* Add more restrictive `WHERE` clauses
* Consider pre-aggregating data in your warehouse
# Record Validation Rules
One of the key metrics tracked by Telmai is the “**Correctness**” metric. It is defined as the percentage of records which are valid based on a corresponding rule or combination of rules. Correctness is - For example, for a table with 1M rows; if 50,000 records are invalid, correctness is then 95%.
The row validity checks (data expectations) are defined by users via **Correctness Rules.**
## Correctness Rules (Record Validation)
Correctness Rules are evaluated at row level. A single rule may validate one or more attribute values (plain or nested object) against a condition.
Rules are created following SQL-like syntax that uses variables and functions to perform the validation. The main key features for rules:
* Eases the process of applying the same rule against multiple tables through templating
* Rules DSL (domain specific language) allows for version control and management in a simple human-readable format
* Optimized algorithms for unlimited number of rules for faster and more efficient processing
### Building a Correctness Rule
Creating a correctness rule requires performing the following steps:
1. Define variables that will be used in the rule
* Variables can be assigned to static lists of values, range, entered manually or uploaded, from files; etc.
2. Map variables to attributes used in the expression
3. Write rule expression
#### Define and Map Variables
Users can define variables to be used in their rule. Supported variable types:
* Array of strings
* Constant string value
* Range (date-time or integer)
* Reference to an attribute
Example:
We can define the following two variables (`accepted_country_list, var_1)`:
* `accepted_country_list = [USA, Canada]`
* `var_1 = country_name`
User can then write a rule like following
**`VALIDATE`**` `` ``var_1 `
**`EXPECT`**` ``value`` `**`IN`**` `` ``accepted_country_list `
This rule checks if a value from the attribute *country\_name* is in the list of accepted values. First it defines an expression *var\_1*, which is a trivial expression, which doesn’t add any transformation to the original value. And after that it sets an expectation that the value of the expression should be in the list to which the previously defined variable is assigned.
#### Writing Rule Expression
A correctness rule consists of three parts:
| **Clause** | **Description & Example** |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **`WITH `** |
Variable transformation:
Optional section. It allows the user to create a transformation of one or more attributes to be validated and reused within the expression multiple times
Example:
WITH concat(country, '
|
|
VALIDATE
\
|
Validated expression:
This could be simply validating an attribute as is, or adding transformation or case clause.
Example:
VALIDATE
case
when loc = 'US
|
|
EXPECT
\
|
Expected results
Expectation for the expression above. After the “EXPECT” keyword, the validation begins.
Example:
EXPECT is\_true
this means the value of validated expression is expected to be true, and anything else is a violation)
|
### How to get alerted for rule violation?
Once a rule is created, users can configure a threshold on the monitor to trigger alerts when the Correctness score falls below the defined level.
## Variables transformation
This step allows defining variables from combining existing attributes to be used at a later step of validation. This can simplify writing the evaluated expression especially in case when multiple checks are applied.
To define a transformation, you can follow the following syntax:
`WITH AS `
Telmai uses standard Spark SQL syntax for those transformations.
## Evaluated expression
This is a Spark SQL expression. It allows users to define what will be evaluated, and exposes a wide range of SQL functions to allow for conditional checks.
### Scenario 1: Validate attribute value as is, without transformations
```
VALIDATE my_column
```
This validates “`my_column`” without any transformation
### Scenario 2: Apply standard functions to attribute value
```sql
VALIDATE foo(my_column)
```
This validates “`my_column`” after applying the “`foo`” function to it.
### Scenario 3: Cases
```sql
VALIDATE case when my_colA <> 0 then my_colB else my_colC end
```
This validates the rule by using `my_colB` when `my_colA` is not 0; otherwise, it uses `my_colC`
## Expectation
Expectation is defined through the context and the expected value.
```
EXPECT []
```
### Expectation context
The expectation context (``) is how the value gets evaluated. It can be any of the following:
| Context | Description |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COMPRESSED_PATTERN` | Compressed pattern of the expression value |
| `DATE_TIME_VALUE` | Expression value as Datetime. Refer [here](https://learn.microsoft.com/en-us/sql/t-sql/functions/date-and-time-data-types-and-functions-transact-sql?view=sql-server-ver16#date-and-time-data-types) for datetime patterns |
| `EXPANDED_PATTERN` | Expanded pattern of the expression value |
| `FREQUENCY` | Frequency of the expression value |
| `LENGTH` | String length of the expression value |
| `NUMERIC_VALUE` | Expression value as number |
| `SPACE_COUNT` | Count of spaces in the expression value |
| `SPEC_CHAR_COUNT` | Count of special characters in the expression value |
| `VALUE` | String representation of the expression value |
### Expectation operator
The expectation operator (``) is an optional field. It’s the condition operator for the context and the expected value. It can be from the following list:
| Operator | Description |
| -------------- | ----------------------------------------------------------------------------------- |
| `IN` |
Value is in the list.
Expected value must be in the list.
|
| `IN_RANGE` |
Value in range including boundaries. Expected value must be in the range.
|
### Expected value
Expected value (\) can be set using a static value or the boolean-functions belo
| Static value | Example |
| ------------ | ----------------- |
| List | `["a", "b", "c"]` |
| Range | `(0,100)` |
| String | `"abcd"` |
| Variable | `var_1` |
| Boolean function | Description |
| --------------------------- | ---------------------------------------------------------------- |
| `CONTAINS_PII` | True if expression contains PII data |
| `CONTAINS_PII_CREDIT_CARD` | True if expression contains credit card number pattern |
| `CONTAINS_PII_IP_ADDRESS` | True if expression contains IP address |
| `CONTAINS_PII_PHONE_NUMBER` | True if expression contains phone number pattern |
| `CONTAINS_PII_SSN` | True if expression contains social security number pattern |
| `IS_ALPHA` | Value is only from alphabetic characters |
| `IS_DATE` | Value is UTC date pattern |
| `IS_DATE_TIME` | Value is UTC date-time pattern |
| `IS_EMAIL` | Value is email |
| `IS_FALSE` | Value is false (case insensitive string comparison, not boolean) |
| `IS_NUMBER` | Value is numeric |
| `IS_TRIMMED_STRING` | Value is a trimmed string |
| `IS_TRUE` | Value is true (case insensitive string comparison, not boolean) |
| `REGEX(pattern)` | Value satisfies regex pattern |
Examples:
```sql
VALIDATE Var_1 expect is_date
# Check if Var_1 is a valid date.
VALIDATE Var_1 expect contains_pii
# Check if Var_1 contains Personally Identifiable Information (PII)
VALIDATE Var_1 expect numeric_value in_range (1, 20)
# Ensure Var_1 is a number within range 1 to 20
VALIDATE Var_1 expect numeric_value out_of_range (50, 60)
# Ensure Var_1’s numeric value is outside a length range
VALIDATE Var_1 expect expanded_pattern IN ['D', 'DD']
# Validate Var_1 using an expanded pattern
VALIDATE Var_1 expect compressed_pattern in ['D', 'L']
# Validate Var_1 using a compressed pattern
VALIDATE Var_1 expect value IN ['USA', 'Germany', 'India']
# Check if Var_1 value is in the accepted country list
VALIDATE Var_1 expect value='Telm'
# Ensure Var_1 exactly matches the value Telm
VALIDATE concat(Var_1, '.', Var_2) expect value='Telm.ai'
# Validate full name format as Telm.ai, where Var_1=Telm and Var_2=ai
VALIDATE Var_1 expect length<>5
# Ensure Var_1 value is of length not equal to 5
VALIDATE Var_1 expect frequency=1
# Ensure Var_1 has exactly one occurrence
VALIDATE Var_1 expect regex('^[a-zA-Z]+$')
# Validate Var_1 contains only letters (no numbers/special characters)
```
### Combining Expectations
User is able to combine multiple expectations using AND/OR operator:
```
EXPECT AND OR
```
Examples:
```robotframework
VALIDATE case when Status='Active' and LastLogin < current_date - 30 then 'Inactive' else 'Valid' end expect value='Valid'
# Mark users as inactive if they haven't logged in for 30+ days, otherwise expect 'Valid'
with concat(Country1, ' | ', Country2) AS loc VALIDATE loc expect length<20
# Ensure concatenated location (Country1 | Country2) has a length less than 20
VALIDATE Password expect spec_char_count>=1 AND space_count=0 AND length>=8
# Ensure Password has at least 1 special character, no spaces, and is at least 8 characters long
VALIDATE Password expect (spec_char_count>=1 AND space_count=0) OR length>=20
# Ensure Password has at least 1 special character and no spaces, or is at least 20 characters long
```
## Rule Templates
Rule Templates is a predefined rule where the expectation is defined, then applied to 1 or more attributes. Users are recommended to use rule templates when the same check is applied on multiple tables. This centralizes the definition of the rule, and allows managing the rule from a single place.
To use a rule template, the user only maps attributes within their table to variables within the rules.
## Defining a Rule
The following flow diagram describes the steps to create a new rule or rule template:
1. Define rule info:
* Rule name
* Rule description
* \[Optional] Template being used (this is only if creating a rule)
2. Set static variables:
* These are variables that are validated against, and are not dynamic in nature; ex: list of allowed countries
3. Set input variables:
* These are variables that are mapped to attributes in the table
* In case this is a rule, user will need to map the attributes to variables
4. Write expression
Creating a Rule
Note; In case user is using an existing template, only input variables need to be mapped to variables in the table
## Creating an Expression-Based Rule
1. Navigate to **Alerting Monitors** and click **New Monitor**
2. Choose **Record Validation Rule** as the monitor type
3. Enter a monitor name and optional description
4. Write your rule expression following the [steps](#creating-a-rule) above
5. Validate and save the rule
{% hint style="info" %}
Threshold will be applied on each rule individually
If policy is used in DataBinning, all rules are checked when binning the data
{% endhint %}
# Rules Expression Examples
### 1. Expanded / Compressed Pattern Rule
**Example:** Column: **ZipCode** Requirement: ZipCode should have 4 or 5 digits only.
* **Input Variable**: `Var_1` (mapped to column ZipCode)
* **Static Variable**: `valid_zip` (List type: \["DDDD", "DDDDD"])
| Expression | List Type | Sample Value | Expected Result |
| ---------------------------------------------------------------------- | ------------------- | ------------ | --------------- |
| `validate Var_1 expect expanded_pattern IN valid_zip` | \["DDDD", "DDDDD"] | 1234 | True |
| `validate Var_1 expect expanded_pattern IN valid_age` | \["D", "DD", "DDD"] | 1008 | False |
| `validate Var_1 expect compressed_pattern IN valid_compressed_pattern` | \["DLD"] | 123ab9 | True |
### 2. Range Rule
**Example:** Column: **Age** Requirement: Age should be between 18 and 60.
* **Input Variable**: `Var_2` (mapped to column Age)
* **Expression**: `validate Var_2 expect numeric_value in_range (18, 60)`
| Sample Value | Expected Result |
| ------------ | --------------- |
| 25 | True |
| 17 | False |
### 3. Regex Pattern Rule
**Example:** Column: **Email** Requirement: Email should be in valid format.
* **Input Variable**: `Var_3` (mapped to column Email)
| Expressions | Sample Value | Expected Result |
| -------------------------------------------------------------------------------- | ------------------ | --------------- |
| `Validate var_3 expect regex('^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,}$')` | | True |
| `Validate var_3 expect regex('^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Za-z]{2,}$')` | invalid-email.com | False |
| `validate var_3 expect regex('\d+')` | 123456789 | True |
### 4. Length Rule
**Example:** Column: **Username** Requirement: Username should be between 5 to 15 characters long.
* **Input Variable**: `Var_4` (mapped to column Username)
| Expressions | Sample Value | Expected Result |
| ---------------------------------------------- | ------------------------ | --------------- |
| `validate Var_4 expect length in_range(5, 15)` | JohnD | True |
| `validate Var_4 expect length in_range(5, 15)` | JD | False |
| `validate Var_4 expect length=5` | India | True |
| `validate Var_4 expect length<>5` | India | False |
| `validate Var_4 expect spec_char_count=1` | NY 10011, USA | True |
| `validate Var_4 expect space_count<=1` | United States Of America | False |
| `validate Var_4 expect frequency=1` | India | True |
### 5. List Rule
**Example:** Column: **Country** Requirement: Country should be either USA, Canada, or UK.
* **Input Variable**: `Var_5` (mapped to column Country)
* **Static Variable**: `valid_countries` (List type: \["USA", "Canada", "UK"])
| Expressions | Sample Value | Expected Result |
| ---------------------------------------------------- | ------------ | --------------- |
| `validate Var_5 expect Value IN valid_countries` | USA | True |
| `validate Var_5 expect value IN valid_countries` | Germany | False |
| `validate Var_5 expect value not_in valid_countries` | India | True |
### 6. Null Rule
**Example:** Column: `Var_6` Requirement: Validate different null handling scenarios.
* **Input Variable:** `Var_6`
| Expression | Sample Value | Expected Result |
| -------------------------------------------------------- | ------------ | --------------- |
| `validate Var_6 expect is_null` | None | False |
| `validate Var_6 expect is_null` | \_\_null\_\_ | True |
| `validate Var_6 expect not is_null` | \_\_null\_\_ | False |
| `validate Var_6 expect not is_null` | 12345 | True |
| `validate Var_6 expect not is_null` | (Empty) | (Empty) |
| `validate if(Var_6 is null, true, false) expect is_true` | \_\_null\_\_ | True |
### 7. Custom Condition Rule
**Example:** Column: **Salary** Requirement: If the job title is "Manager", the salary should be greater than 50,000.
* **Input Variables**: `Var_7` (mapped to column JobTitle), `Var_8` (mapped to column Salary)
* **Expression**: `validate case when Var_7 == 'Manager' then Var_8> 5000 else false end expect is_true`
| Job Title | Salary | Expected Result |
| --------- | ------ | --------------------- |
| Manager | 60000 | True |
| Developer | 45000 | True (Not applicable) |
| Manager | 40000 | False |
### 8. Date Format Rule
**Example:** Column: **TransactionDate** Requirement: Date should be in YYYY-MM-DD format.
* **Input Variable**: `Var_8` (mapped to column TransactionDate)
| Expression | Sample Value | Expected Result |
| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `validate Var_8 expect is_date` | 2024-02-01 | True |
| `validate Var_8 expect is_date` | 2021-01-01T00:00:00Z | False |
| `validate Var_8 expect is_date_time` | 2011/12/03 10:15:30 | True |
| `validate Var_8 expect date_time_value in_range valid_datetime` |
Static Variable ⇒ Date Time Range From=-2d, To=@now Sample Value ⇒ 2025-07-16 05:24:36.441
| True |
***
This documentation provides a structured guide for different rule expressions, helping users understand rule syntax, sample values, and expected outcomes.
# SQL Based Rules
Creating an SQL-based rule monitor — write a pushdown SQL query to define record-level validation logic.
SQL-Based Validation Rules give you full SQL pushdown for record-level data quality checks. Instead of [Telmai's rule expression builder,](/telmai/monitoring-data/user-defined-monitors/record-validation-rules) you write a standard SQL query directly against your data source. Telmai executes it natively on the compute engine (Snowflake, BigQuery, Databricks, etc.) and ingests the results.
This is ideal when:
* Your validation logic is complex and already expressed in SQL
* You need to join across multiple tables or use warehouse-specific functions
* You want to reuse existing SQL-based data quality checks without rewriting them in DSL
{% hint style="info" %}
SQL-Based Validation Rules follow the same **Correctness** metric model as [Record Validation Rules](/telmai/monitoring-data/user-defined-monitors/record-validation-rules). The key difference is execution: Expression rules run inside Telmai's Spark engine, while SQL-Based rules are pushed down and executed natively on your connected data source
{% endhint %}
***
### How It Works
You provide a SQL query (`Q`) that Telmai executes against your data source. The query must return a specific output schema. Telmai reads the results and computes the **Correctness** metric as the percentage of records where `is_valid = 1`.
For example, for a table with 1M rows: if 50,000 records return `is_valid = 0`, the Correctness score is **95%**.
***
### Query Structure
Your SQL query must return columns in the following exact order:
| Column | Type | Required | Description |
| ---------------- | ----------------- | -------- | ---------------------------------------------------------------------------------- |
| `is_valid` | Integer | Yes | Validation result. `1` = valid, `0` = invalid. |
| `record_id` | String or Integer | No | The unique identifier for the record. Used to surface failing records in the UI. |
| `record_id_name` | String | No | A static label for the identifier field. Used as metadata in incident drill-downs. |
{% hint style="warning" %}
\*\*Column order matters.\*\* Telmai reads output positionally. \`is\_valid\` must always be the first column, followed by \`record\_id\`, then \`record\_id\_name\`.
{% endhint %}
***
### Example Queries
#### Minimal — validation result only
```sql
SELECT
CASE WHEN transaction_amount > 0 THEN 1 ELSE 0 END AS is_valid
FROM my_asset.transactions
```
#### With record identifier
```sql
SELECT
CASE WHEN transaction_amount > 0 THEN 1 ELSE 0 END AS is_valid,
transaction_id AS record_id
FROM my_asset.transactions
```
#### Full output — with metadata label
```sql
SELECT
CASE WHEN transaction_amount > 0 THEN 1 ELSE 0 END AS is_valid,
transaction_id AS record_id,
'transaction_id' AS record_id_name
FROM my_asset.transactions
```
#### Cross-table join validation
```sql
SELECT
CASE WHEN o.status = 'CLOSED' AND p.paid_at IS NOT NULL THEN 1 ELSE 0 END AS is_valid,
o.order_id AS record_id,
'order_id' AS record_id_name
FROM orders o
LEFT JOIN payments p ON o.order_id = p.order_id
```
#### Conditional logic with warehouse functions
```sql
SELECT
CASE
WHEN region = 'US' AND TRY_CAST(zip_code AS INT) IS NULL THEN 0
WHEN region != 'US' AND zip_code IS NULL THEN 0
ELSE 1
END AS is_valid,
customer_id AS record_id,
'customer_id' AS record_id_name
FROM customers
```
***
### Creating a SQL-Based Rule
1. Navigate to **Alerting Monitors** and click **New Monitor**
2. Choose **SQL-Based Rule** as the monitor type
3. Enter a monitor name and optional description
4. Write your SQL query following the [required output schema](#query-structure) above
5. Validate and save the rule
# Replay Scan
The monitor editor — use Save & Replay to retroactively run the monitor against historical scans.
The **Replay** feature allows you to retroactively run SQL-based monitors against historical data scans. This enables you to validate new monitor logic against past data and backfill results to identify historical anomalies without waiting for new data to arrive.
{% hint style="info" %}
**Why use Replay?** If you create a new Data Quality rule today, Replay lets you see how that rule would have performed over previous scans, providing immediate historical context.
{% endhint %}
## Prerequisites & Availability
To ensure data consistency during historical runs, the Replay feature has specific requirements:
* **Scope:** Available for **SQL-based monitors** only.
* **Required Parameter:** The monitor's SQL query **must** contain the `@now` parameter. This ensures the logic anchors correctly to the specific timestamp of the historical scan.
## Triggering a Replay
The Replay flow is initiated exclusively from the **Monitor Editor** page.
1. **Save & Replay:** When saving changes to a monitor, you are presented with two options:
* **Save:** Saves the monitor configuration without a backfill.
* **Save & Replay:** Saves the configuration and opens the replay range selection.
2. **Configuration:** Select the historical depth.
* **Maximum Depth:** Up to **20** previous scans.
* **Actual Limit:** Calculated as `min(available_scans, 20)`.
## Validation & Error States
| Condition | UI Behavior |
| ----------------------- | --------------------------------------------------------------- |
| **@now not in query** | Replay disabled; tooltip shows "Replay requires @now parameter" |
| **No historical scans** | Replay disabled; indicator shows "No data to replay" |
| **Replay in progress** | Progress indicator shown; re-triggering is disabled |
### API Reference
Please refer to [Replay Monitor Scan ](/telmai/api-reference/upload-data-api/replay-monitor-scan)page for API references
# PII Data Detection Monitors
Telmai's PII Data Detection feature enables identification of Personally Identifiable Information (PII) within monitored attributes. This functionality helps organizations proactively manage data privacy risks by detecting and alerting PII exposure, ensuring compliance, and enhancing data governance.
To enable PII data detections, in **Alerting Monitors** page:
1. Select the **PII Exposure Monitor**.
2. Enable the policy (disabled by default).
3. Optionally, limit the scope to specific attributes for focused detection.
[Learn more about Managing Monitors](/telmai/monitoring-data/monitors-management)
### What happens when PII data is detected
If Telmai detected the presence of PII data, you will see created alerts and incidents similar to below image:
In the picture above it detected PII Exposure in attribute “comment 2“. By clicking on the attribute name it will navigate to Investigator page to inspect values which were detected as violations
You can also learn more about what PII data Telmai detected by navigating to the Investigator page. The Values table will show you true and false values for detections as “Contains PII”, “Contains Phone”, etc.
In the example above Contains PII field displays true for 2 of the values, which means it contains PII data. Same for other fields, representing other classes of PII Data, ex. Credit Card Number, Phone etc.
## PII Data Validation Rules
Telmai's rules engine supports custom validation and remediation of PII data violations. This includes actions like segregating PII-exposing records using [Data Binning](/telmai/remediation/data-binning).
To learn more about creating these rules, please visit our [Record Validation Rules ](/telmai/monitoring-data/user-defined-monitors/record-validation-rules)documentation
## Related Documentation
* [User-defined Monitors](/telmai/monitoring-data/user-defined-monitors)
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Incident Portal](/telmai/monitoring-data/incident-portal)
# Monitors Management
## Accessing the Monitors Page
Navigate to **Alerting Monitors** from the main navigation menu to access all monitor management functionality.
The Monitors page provides:
* A comprehensive list of all monitors organized by asset
* Quick access to enable/disable monitors
* Edit and delete actions for each monitor
* Access to monitoring templates via the **Rule templates** tab
## Monitor Organization
#### Asset Grouping
Monitors are organized hierarchically by asset in the left sidebar:
* **Projects**: Collapsible groups containing related assets
* **Assets**: Individual data sources (tables, views, files) with their associated monitors
* **Monitor List**: All monitors for the selected asset displayed in the main panel
**Navigation tips:**
* Click the arrow (▸) to expand/collapse asset groups
* Use the search bar to quickly find specific assets
* Select an asset to view all its monitors
#### Monitor List View
The main panel displays all monitors for the selected asset with the following information:
| Column | Description |
| ----------- | -------------------------------------------------------- |
| **Name** | Monitor name with type indicator (NEW, built-in, custom) |
| **Type** | Monitor type |
| **Enabled** | Toggle switch to enable/disable the monitor |
| **Actions** | Edit (✏️) and Delete (🗑️) buttons |
## Monitors Actions
The Monitors page provides several top-level actions:
### Configure Data Binning
Set up data binning rules to control how Telmai bins your data. [Learn more about data binning](/telmai/remediation/data-binning).\
### Manage Alert Channels
Configure notification destinations including Slack, email, webhooks, and other integrations to ensure alerts reach the right teams. [Learn more about managing notifications](/telmai/monitoring-data/monitors-management/configuring-notifications).
### + New Monitor
Click **+ New Monitor** to build custom monitors tailored to your specific data quality requirements. [Learn more about creating monitors](/telmai/monitoring-data/monitors-management/creating-a-monitor).
## Related Documentation
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Monitor Recommendations](/telmai/monitoring-data/monitors-management/monitor-recommendations)
* [Managing Existing Monitors](/telmai/monitoring-data/monitors-management/managing-existing-monitors)
* [Configuring Data-Binning](/telmai/monitoring-data/monitors-management/configuring-data-binning)
* [Configuring Notification](/telmai/monitoring-data/monitors-management/configuring-notifications)
# Creating a Monitor
## Starting Monitor Creation
1. Navigate to the **Alerting Monitors** page
2. Select the asset you want to monitor from the left sidebar
3. Click the **"+ New Monitor"** button in the top-right corner
### Monitor Creation Panel
The monitor creation panel opens on the right side with two main sections:
**Section 1: Monitor Policy:** Choose your monitoring type
**Section 2: Configuration:** Set thresholds and conditions
**Section 3: Notification:** Set notification channels \[optional]
**Section 4: Ticketing:** Set auto-ticketing requirements \[optional]
***
## Choosing Monitor Type
Select from three monitor types based on your monitoring needs:
* [Built-in Metric](/telmai/monitoring-data/user-defined-monitors/built-in-metrics)
* [User-Defined Metric](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics)
* [Record Validation Rule](/telmai/monitoring-data/user-defined-monitors/record-validation-rules)
{% hint style="warning" %}
**Important:** Monitor type cannot be changed after creation. Choose carefully based on your monitoring requirements.
{% endhint %}
### Configuration Section
After selecting your monitor type, configure the monitor details:
#### Basic Information
Section
Details
Monitor Name (Required)
Must be unique per asset
Use descriptive names that clearly indicate what's being monitored
Set the severity level for alerts generated by this monitor:
High: Important issues requiring prompt resolution
Medium: Notable issues that should be addressed
Low: Minor issues for awareness
***
### Configuring Built-in Metrics
When you select **Built-in Metric**, you'll see the predefined metric configuration section.
#### Data Quality Metric (Required)
Select from available predefined metrics:
* **record\_count**: Total number of records
* **null\_percentage**: Percentage of null values
* **freshness**: Time since last data update
* **completeness**: Data completeness score
* **uniqueness**: Duplicate detection
* **schema\_drifts**: Schema change detection
* [And more...](/telmai/monitoring-data/user-defined-monitors/built-in-metrics)
#### Attributes (Optional)
For metrics that support attribute-level monitoring:
* Select specific columns to monitor
* Leave blank for table-level metrics
* Multiple attributes can be selected
**Examples:**
* Monitor `uniqueness` on `customer_id` field
***
### Configuring User-Defined Metrics
When you select **User-Defined Metric**, you can define custom metrics using expressions or SQL.
#### Method 1: Expression-Based Metrics
Write simple aggregations with grouping. Example:
```
AGGREGATION(`attribute`) GROUP BY `dimension1`, `dimension2`
```
See [full expression documentation](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics#metric-expressions) for more examples.
#### Method 2: Custom SQL Queries
Write SQL queries for complex monitoring logic. Example:
```sql
SELECT AVG(order_value), customer_region, product_category
FROM `orders`
WHERE order_date >= CURRENT_DATE - 30
GROUP BY customer_region, product_category
```
See [push down SQL documentation ](/telmai/monitoring-data/user-defined-monitors/user-defined-metrics#raw-sql-queries)for more examples.
***
### Configuring Record Validation Rules
When you select **Record Validation Rule**, you define checks for individual records.
#### Rule Definition
Write validation expressions for row-level checks:
**Syntax:**
```
validate CONDITION expect TELM_EXPECTATION
```
See [record violation rules ](/telmai/monitoring-data/user-defined-monitors/record-validation-rules)for more examples.
***
## Threshold Configuration
All monitor types require threshold configuration to determine when alerts are triggered.
### Threshold Types
Choose from three threshold approaches:
#### **Automatic (ML-Based)**
Let machine learning determine dynamic boundaries based on historical patterns.
**Best for:**
* Metrics with seasonal patterns
* Establishing initial baselines
* Adapting to data evolution
#### **Acceptable Drift %**
Set percentage-based boundaries using moving averages.
**Best for:**
* Relative comparisons to recent performance
* Allowing controlled variance
* Gradual data changes
**Configuration:**
* Define acceptable drift percentage: Set minimum & maximum drift percentage values
#### **Acceptable Range**
Define constant, fixed boundaries.
**Best for:**
* Hard business rules
* Strict compliance requirements
* Known acceptable limits
**Configuration:**
* Set minimum & maximum value
### Additional Configuration Options
#### Compare to Past # Scans
Specify how many historical datapoints to consider:
* More scans = more stable baseline
* Fewer scans = more responsive to recent changes
* Default: `Automatic` (system-determined)
#### Replace Missing Values
Choose how to handle gaps in your metric time series:
* **Ignore**: Skip missing datapoints
* **Use average**: Fill with calculated average
* **Use zero**: Treat missing as zero
***
## Notification & Ticketing Configuration
Each monitor can be configured to send alerts and automatically create tickets when data quality issues are detected. These settings help ensure the right teams are notified and issues are tracked through resolution.
Configure where alerts should be sent when this monitor detects data quality issues.
### **Setting Up Alert Channels**
1. Click **+ Add Channel** in the Notifications section
2. Add as many predefined notification destination as needed
3. Use the **Enabled** toggle to activate or deactivate notifications for this monitor
**Note**: Notifications are marked as **OPTIONAL**. Monitors will continue to function and track data quality even without configured alert channels.
[Learn more about managing alert channels](/telmai/monitoring-data/monitors-management/configuring-notifications)
***
### Ticketing
Enable automatic ticket creation to track and manage data quality issues through your existing workflow tools.
**Auto-Ticketing Setup**
When enabled, Telmai automatically creates a ticket in your ticketing system whenever this monitor's policy conditions are met. This ensures data quality issues are formally tracked and assigned for resolution.
**To enable auto-ticketing:**
1. Check the **Create ticket automatically** checkbox
2. Select a ticket **Template** from the dropdown
* Templates define ticket fields, priority, assignment rules, and other properties
* Templates must be configured in advance in your ticketing integration settings
3. Save your monitor configuration
**Note**: Ticketing is marked as **OPTIONAL**. This feature requires a configured ticketing integration and defining templates before this feature is enabled. [Learn more about managing ticketing integration with Jira.](/telmai/integrations/jira-integration)
***
## Saving Your Monitor
#### Save and Cancel Actions
After configuring your monitor:
1. **Review** all settings
2. Click **"Save"** to create the monitor (green button, bottom-right)
3. Or click **"Cancel"** to discard changes
### What Happens After Saving
* Monitor is immediately created and enabled (by default)
* Appears in the monitor list for the selected asset
* Begins evaluating data according to configured schedule
* Will generate alerts when thresholds are violated
***
## Related Documentation
* [Managing Existing Monitors](/telmai/monitoring-data/monitors-management/managing-existing-monitors)
* [Configuring Data-Binning](/telmai/monitoring-data/monitors-management/configuring-data-binning)
* [Configuring Notification](/telmai/monitoring-data/monitors-management/configuring-notifications)
***
# Monitor Recommendations
{% hint style="info" %}
Monitor Recommendations requires the Telmai AI Agent to be installed and enabled for your tenant. If the AI Agent is not installed, the **Recommended Rules** panel will not appear on the asset's monitor list.
{% endhint %}
Monitor Recommendations uses the Telmai AI Agent to automatically suggest monitors for a connected asset, removing the need to manually decide which rules and metrics to configure. After the AI Agent profiles an asset's initial scan and learning cycle, it analyzes the data's structure, content, and patterns to propose relevant rules and built-in metrics tailored to that asset.
The Recommended Rules panel appears below the asset's monitor list, showing AI Agent–suggested rules with type tags and a + button to add each one.
## How It Works
1. **Automatic Generation**: Once an asset completes its initial scan and the AI Agent finishes learning its data patterns, recommendations are generated automatically and appear in the **Recommended Rules** panel below the asset's monitor list.
2. **Refresh Recommendations**: Click the refresh icon in the panel header to re-run the AI Agent's analysis and regenerate suggestions, for example after the data or schema has changed.
3. **Review Suggestions**: Each recommended rule displays a name, a short description of what it detects, and a type tag (such as **Business Rule**, **Range**, or **Uniqueness**) indicating what kind of monitor is being proposed.
4. **Add a Recommendation**: Click the **+** button next to a recommended rule to open the monitor configuration dialog, pre-filled with the AI Agent's suggested settings. Review and adjust the configuration as needed, then save to create the monitor.
## What Can Be Recommended
The AI Agent can recommend both:
* **Record validation rules** — business rules that check individual records against expected patterns, ranges, or uniqueness constraints
* **Built-in metrics and monitors** — out-of-the-box monitor types relevant to the asset's data characteristics
## Related Documentation
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Out-of-box Monitors](/telmai/monitoring-data/out-of-box-monitors)
* [User-defined Monitors](/telmai/monitoring-data/user-defined-monitors)
# Managing Existing Monitors
## Enabling and Disabling Monitors
Toggle monitors on/off directly from the list view:
* **Green toggle (ON)**: Monitor is active and evaluating data
* **Gray toggle (OFF)**: Monitor is paused and not generating alerts
**Use cases for disabling:**
* Temporary maintenance periods
* Known data issues being resolved
* Testing alternative configurations
## Editing Monitors
To modify an existing monitor:
1. Locate the monitor in the list
2. Click the **edit icon (✏️)** in the Actions column
3. Make your changes in the configuration panel
4. Click **"Save"** to apply changes
**Editable properties:**
* Monitor name
* Description
* Tags
* Impact level
* Threshold configuration
* Notification settings
* Ticketing settings
**Non-editable properties:**
* Monitor type (cannot be changed after creation)
* Asset association
## Deleting Monitors
To remove a monitor:
1. Locate the monitor in the list
2. Click the **delete icon (🗑️)** in the Actions column
3. Confirm the deletion
{% hint style="danger" %}
**Warning:** Deleting a monitor:
* Stops all monitoring and alerting immediately
* Cannot be undone
* Removes historical alert data associated with this monitor
* Consider disabling instead if you may need it again
{% endhint %}
## Download and Upload Monitors
Download or upload monitors configuration directly from Alerting Monitors page for corresponding asset.
* Download button: Downloads monitors configuration in JSON format
* Upload button: Uploads monitors configuration in JSON format.
To learn more about the configuration file format, [please see our APIs reference](/telmai/api-reference/monitors-api)
***
## Related Documentation
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Configuring Data-Binning](/telmai/monitoring-data/monitors-management/configuring-data-binning)
* [Configuring Notification](/telmai/monitoring-data/monitors-management/configuring-notifications)
# Configuring Data-Binning
Data binning enables you to automatically separate valid data from problematic data at the source, ensuring that only high-quality records flow through your data pipelines. By routing bad or suspicious data to a separate location for review, you can prevent costly downstream processing of invalid records and maintain pipeline efficiency.
Telmai's data binning feature monitors data correctness and categorizes records into two bins:
* **Valid Data**: Records that meet all [defined data quality rules](/telmai/monitoring-data/user-defined-monitors/record-validation-rules) and continue through your pipeline
* **Invalid Data**: Records that violate quality rules and are flagged for review
This automated quality gate helps you maintain data integrity while reducing the cost of processing bad data through expensive transformation and analytics pipelines.
### Prerequisites
Before configuring data binning, ensure you have:
1. A connected data source in Telmai
2. Defined [record validation rules](/telmai/monitoring-data/user-defined-monitors/record-validation-rules) for your dataset
3. A cloud storage destination (AWS S3, GCP Cloud Storage, or Azure Blob Storage) with appropriate permissions
## Configuration Steps
### Step 1: Configure ID Attribute
Set the ID attribute for your data source to enable record-level tracking. This unique identifier allows Telmai to categorize individual records into valid or invalid bins.
**To configure:**
1. Navigate to your connected asset settings
2. Locate the **ID Attribute** configuration
3. Select the column that uniquely identifies each record (e.g., `record_id`, `transaction_id`, `user_id`)
### Step 2: Define Data Quality Monitors
Create one or more validation monitors that determine whether data is valid or invalid as defined in [Record Validation Monitors](/telmai/monitoring-data/user-defined-monitors/record-validation-rules).
### Step 3: Enable Data Binning
Data Binning Settings — configure storage destination and paths for valid and invalid data.
Now you're ready to configure and activate data binning.
**To enable:**
1. Click the **Configure Data Binning** button\
2. Select Correctness Monitor:\
Choose one or more correctness monitors you created in Step 2. These are the monitors/rules that defines which records are considered valid vs. invalid.
3. Choose Storage Destination:\
Select your cloud storage type:
* **AWS S3**
* **GCP Cloud Storage**
* **Azure Blob Storage**
4. Enter Storage Credentials: Provide the authentication credentials for your chosen storage destination. See [Storage Permissions](#storage-permissions) below for required access levels.
5. Define Storage Paths
1. **Valid Data Path**: The destination path for records that pass all quality rules (e.g., `s3://my-bucket/valid-data/`)
2. **Invalid Data Path**: The destination path for records that fail quality rules (e.g., `s3://my-bucket/invalid-data/`)
6. Click **Save** to activate data binning
### Step 4: Verify Binning Activation
Once enabled, data binning automatically takes effect during your next data scan job. Telmai will begin routing records to the appropriate storage paths based on your correctness policy.
**To verify:**
1. Wait for the next scheduled scan or trigger a manual scan
2. Check both the valid and invalid data paths in your cloud storage
3. Review the binning results in the Telmai monitoring dashboard
### Storage Permissions
#### AWS S3
Ensure your AWS credentials have the following permissions:
* `s3:PutObject` - Write objects to buckets
* `s3:DeleteObject` - Remove objects from buckets
* `s3:ListBucket` - List bucket contents
#### GCP Cloud Storage
Your service account must have the following IAM roles:
* **Storage Legacy Bucket Writer** - Write access to buckets
* **Storage Legacy Object Owner** - Full control over objects
#### Azure Blob Storage
Your SAS token or service principal must have:
* **Write** permissions - Create and write blobs
* **Delete** permissions - Remove blobs (required for data binning operations)
## Feature Limitation
* **Group Assets**: Data binning is not currently supported for assets belonging to a group. Configure binning at the individual asset level.
* **Data Diff Compatibility**: The Data Diff feature is not available for assets with data binning enabled.
* **Scan Frequency**: Binning operates during scheduled scans. Ensure your scan frequency aligns with your data freshness requirements.
## Related Documentation
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Managing Existing Monitors](/telmai/monitoring-data/monitors-management/managing-existing-monitors)
* [Configuring Notification](/telmai/monitoring-data/monitors-management/configuring-notifications)
# Configuring Notifications
Telmai notifications are used to notify users about new or ongoing alerts. Telmai supports different notification channel types.
## Supported Notification Types
* Microsoft Teams
* Slack
* Webhooks
* Emails
## Notification Modes
Notifications can be set at tenant level based on 3 modes:
* **Priority summary:** In this setting, detections are summarized by count and priority before being sent to the specified channel(s). The notification interval acts as a grace period, during which all alerts are grouped into a single notification.
* **Monitors overview:** In this setting, detections from configured monitors are consolidated into one notification. Only the most significant detections are included.
* **Monitors insights**: In this setting, each detection triggers an individual notification. If applicable, historical charts are included in the notification.
To select the notification mode, tenant admins can select it from the "Administration" page
## Add or Modify Notification Channels
Alert Channels — manage email and Slack notification destinations.
Notification channels are the destinations that can be used for alerting. To create a notification channel, navigate to the **Alerting Monitors** page and follow these steps:
1. Click **Manage Alert Channels**
2. Click **+ Alert Channel**
3. Select the channel type and enter a channel name
4. Fill in the required details for the selected type (see below)
5. Click **Validate** — Telmai will send a test message to confirm the connection
6. Click **Save**
### Slack Channel Setup
When adding a Slack channel, you will need:
* **Channel Name** — a label used to identify this channel within Telmai
* **Bot User OAuth Access Token** — the token generated from your Slack app (see [Slack Integration](/telmai/integrations/slack-integration))
* **Slack Channel(s)** — the Slack channel(s) where alerts will be posted (e.g. `#data-alerts`). Click **Add** to include multiple channels.
## Related Documentation
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Managing Existing Monitors](/telmai/monitoring-data/monitors-management/managing-existing-monitors)
# Monitors Troubleshooting
## Monitor Not Generating Alerts
**Possible causes:**
* Monitor is disabled (check toggle)
* Thresholds are too lenient
* Data isn't violating conditions
* Asset isn't receiving new data
**Solutions:**
1. Verify monitor is enabled
2. Review threshold configuration
3. Check asset data freshness
4. Test with known violation scenario
## Too Many False Positives
**Possible causes:**
* Thresholds too strict
* Not accounting for normal variance
* Using Absolute when Automatic would be better
**Solutions:**
1. Switch to Automatic thresholds
2. Widen Acceptable Range
3. Increase Acceptable Drift %
4. Review historical data patterns
## Monitor Configuration Errors
**Common issues:**
* Invalid expression syntax
* SQL query errors
* Unsupported aggregations
* Attribute not found
**Solutions:**
1. Verify syntax follows documentation
2. Test SQL queries in your data warehouse first
3. Confirm attributes exist in asset
4. Check for typos in attribute names
## Cannot Edit Monitor Type
**Issue:** Monitor type cannot be changed after creation
**Solution:**
1. Create a new monitor with desired type
2. Copy configuration settings
3. Disable or delete old monitor
4. Enable new monitor
# Incident Portal
This page provides a comprehensive view of all incidents, helping you quickly understand the status of your monitored systems. The portal is designed to give you a high-level overview of ongoing issues while also allowing you to drill down into specific details.
### Key Features
The main dashboard is organized into a few key sections:
* **Incidents Cards:** At the top of the portal, you'll see a series of cards that provide a quick snapshot of incident activity, including the number of open and closed incidents, the meantime to resolve (MTTR), and tags distribution.
* **Open Incidents by Impact:** This graph tracks the number of open incidents over time, categorized by their impact level. This allows you to quickly see the severity of current issues and identify any spikes in high-impact incidents.
* **Incidents Table:** The table provides a detailed, sortable list of all incidents, both active and closed. You can review key information at a glance, including the incident's status, creation time, and any associated external tickets.
* **Incident Details:** Clicking on any incident in the table will open a new view with more granular information. This view provides a full history of the incident, including a list of the alerts that triggered it. This helps you understand the specific events and data points that led to the incident's creation.
To learn more about how to setup your thresholds, please refer to the following pages:
* [Creating a Monitor](/telmai/monitoring-data/monitors-management/creating-a-monitor)
* [Data Trends and Alerts](/telmai/monitoring-data/data-trends-and-alerts)
* [Metrics Inspector](/telmai/monitoring-data/metrics-inspector)
# Alert Policies
{% hint style="danger" %}
This feature is deprecated as of 25.4.4 (2025-12-30)
{% endhint %}
Telmai provides a set of out-of-the-box policies that are automatically created when a new dataset is added. Users also have the flexibility to create their own custom policies
## Out-of-box Policies
Below is the list of pre-defined policies
| Policy | What is Monitored? |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *Completeness Drifts* | Changes in percentage of empty records |
| *DQ Rules Violation* | Detection for any rule violation |
| *Data Difference* |
Addition/Removal of different records across tables.
|
| *Data Drifts* | Changes in various value formats (length, special characters, number of tokens, etc.) |
| *Data Process Failures* | Detection of any scan failure |
| *Freshness Drifts* | Table freshness drifts |
| *Numeric Value Drifts* | Changes in averages of numeric values for an attribute |
| *PII Exposure* |
Presence of PII data in your dataset (email, IP, etc).
Note: This policy is disabled by default
|
| *Record Count Correlation Drifts* |
Changes in data volume across correlated tables with defined lineage.
Note: For this policy to take effect, you need to define the tables lineage
|
| *Record Count Drifts* | Changes in the number of scanned rows |
| *Record Id Uniqueness Drifts* |
Changes in data uniqueness of record Id
Note: For this policy to take effect, you need to enable table Id
|
| *Schema Drifts* | Changes in schema; example: column added or removed |
| Top Values Drifts | This feature monitors the changes for the top 10 values. An alert is generated if new values are added or removed |
| *Uniqueness Correlation Drifts* |
Changes in data uniqueness across tables with defined lineage
Note: For this policy to take effect, you need to define the tables lineage
|
| Uniqueness Drifts | Changes in data uniqueness across attributes |
| *Value Distribution Drifts* | Changes in the distribution of the top 20 most frequent categorical values |
| ~~*Correctness Drifts*~~ | ~~Changes in the percentage of correct records based on defined expectations~~ |
## Modifying/Reviewing Existing Policies
You can view, modify, or disable existing policies anytime by navigating to the **“Alerting Policies”** page and selecting the relevant dataset. The policies are displayed in a table with the following properties:
* **Name**: Policy name (User-defined policies are marked in yellow; Telmai out-of-the-box policies are marked in green)
* **Metric**: The monitored metric
* **Tags:** A list of text tags. These tags are user defined, and can be used for tracking purposes
* **Impact:** User defined property for impact reporting. Value can be `LOW`, `MED` or `HIGH`
* **Status**: Indicates whether the policy is enabled or disabled
* **Create ticket automatically:** Optional to auto create ticket when incidents are created
* If checked, user will need to specify a [ticket template](/telmai/api-reference/ticket-integration-api)
* Ticketing integration must be setup for this feature to be visible (ex: [Jira](/telmai/integrations/jira-integration))
* **Actions**:
* **Edit**: Modify policy properties or scope.
* **Notifications**: Toggle notifications on or off
* **Delete**: Remove the policy (Available only for custom policies)
## User Defined Policy
Users can create custom policies to monitor any of the calculated metrics. These metrics are tracked by specifying a threshold that triggers an alert when violated. The supported thresholds include:
### **Supported Threshold**
* **Telmai-ML**\
Telmai’s built-in anomaly detection using machine learning
* **Relative Drift**\
Percentage change in data compared to previous scans or the average of scans
* **Acceptable Range**\
Check if a metric falls within a specified range
### **Steps to Create a Custom Policy**
1. Navigate to “Alerting Policies” page & choose the “Policies” Tab
2. Click the "New Policy" button to start defining your policy.
3. A side menu will appear similar to this image
4. Select desired “Tracked Metric” from the list of
* User-defined metrics, or
* Health metric calculated by Telmai
5. Based on Metric selection, the user may need to select tracked attributes (ex: correctness, the user will need to specify associated attributes)
6. Select detection type\
Note: In case of Relative and Range detections, the user can specify how to handle the metric when not available (replace by zero or average)
7. Add [notification](/telmai/monitoring-data/monitors-management/configuring-notifications) channels, by clicking "Notifications" button
1. Click +Add Channel
2. Select a value from the list of defined channels.
3. Repeat above for all desired channels
8. Click Save
# Data Trends and Alerts
This page allows you to explore detections (alerts) within a given dataset:
This page has the following sections:
* **Data Scan & Alert Type Selector**
* This selector allows you to pick
* Dataset to be checked
* Filter on alert types (if exists)
* Filter on segments (if exists)
* **Data Scans Stats Graph**
* This graph allows you to:
* See record count per scan
* Number of alerts per scan
* Exclude scan from analysis to not be used in Telmai’s learning about your dataset
* Each bar is clickable to look at graphs within the corresponding scan
* **Alerts**
* List of alerts within the corresponding scan
* If applicable, each alert will have a clickable component that visualizes the detection
#### Feedback Loop
Telmai offers both ways of managing thresholds for alerts: automatic, using ML, or manual.\
In the case of automatic thresholds, it’s crucial to provide the system feedback when anomalous data was scanned and when it should not be used for model fine-tuning for future detections.
This can be done by either
1. Hovering over a particular scan and marking it to be ignored - this will exclude the whole scan
2. Excluding a particular metric from a specific row/alert (if applicable)
# DQ Report Summary
DQ Report Summary — cross-asset view of monitor scan results with pass rate indicators.
The DQ Report Summary page provides a cross-asset view of monitor scan results over a selected time period. It is accessible from the **Explore** section of the Telmai navigation.
Use this page to quickly identify which assets and monitors are producing failures, track overall data quality health, and drill into specific scan runs.
{% hint style="info" %}
DQ Report Summary only shows results from [Record Validation Rules](/telmai/monitoring-data/user-defined-monitors/record-validation-rules) and [SQL-based Rules](/telmai/monitoring-data/user-defined-monitors/sql-based-rules)
{% endhint %}
{% hint style="info" %}
This report can be published to external storage using the [DQ Reporting APIs](/telmai/api-reference/dq-reporting-apis)
{% endhint %}
{% hint style="warning" %}
If asset has metadata scans enabled & SQL-based rules, results will not show in DQ report — Future versions of Telmai will show results
{% endhint %}
### Filters
The following filters appear at the top of the page and apply to all summary statistics and table rows simultaneously.
| Filter | Description |
| ------------------- | ------------------------------------------------------------------------------ |
| **Project** | Filter results to one or more projects. Defaults to all projects. |
| **Connection Type** | Filter by the data source connection type (e.g., AWS S3, Snowflake, BigQuery). |
| **Assets** | Filter to specific assets within the selected project and connection. |
| **Period** | Time window for scan results. Options: **1 week**, **2 weeks**, **1 month**. |
***
### Summary Statistics
Four aggregate metrics are displayed at the top of the results view, reflecting all records matching the active filters.
| Metric | Description |
| -------------------------- | --------------------------------------------------------------------------------- |
| **Total Records Scanned** | Total number of records evaluated across all monitor scans in the period. |
| **Total Records Failed** | Total number of records that failed validation (`is_valid = 0`) across all scans. |
| **Total Assets Monitored** | Count of distinct assets with at least one scan result in the period. |
| **Unique Monitors** | Count of distinct monitors that produced scan results in the period. |
***
### Results Table
Each row represents a single monitor scan run against an asset. Rows are sorted by **Scan Time** descending by default.
| Column | Description |
| ------------------- | ------------------------------------------------------------------------ |
| **Scan Time** | Timestamp of when the monitor scan completed. |
| **Project** | The project the scanned asset belongs to. |
| **Asset** | The name of the scanned asset. |
| **Monitor** | The name of the monitor that ran. |
| **Records Scanned** | Number of records evaluated in that scan run. |
| **Records Failed** | Number of records that did not pass validation. |
| **% Passed** | Percentage of records that passed. Color-coded by threshold (see below). |
#### % Passed Color Coding
Pass rate is color-coded with fixed thresholds to give an at-a-glance health signal:
| Color | Threshold | Meaning |
| --------- | --------------- | --------------- |
| 🟢 Green | ≥ 90% | Healthy |
| 🟠 Orange | ≥ 70% and < 90% | Needs attention |
| 🔴 Red | < 70% | Critical |
You can filter the table to a specific health band using the **Filter by % Success** toggle in the top-right of the results section (**All** / **≥ 90%** / **≥ 70%** / **< 70%**).
***
### Prerequisites
This page requires [Record Validation Rules ](/telmai/monitoring-data/user-defined-monitors/record-validation-rules)or [SQL Based Rules](/telmai/monitoring-data/user-defined-monitors/sql-based-rules) to be setup, and scans to run for results to show on this page.
# Metrics Exclusions
The Metric Point Exclusion feature allows users to remove specific data points from analysis while maintaining historical visibility and keeping monitors active. This is particularly useful when a single scan produces anomalous results that should not influence future predictions or trigger alerts.
## Common Use Case
Consider a scenario where you're monitoring multiple metrics (record count, freshness, completeness). During a specific scan, the freshness metric reports an incorrect value due to a temporary system issue, but all other metrics are normal. Rather than disabling the entire freshness monitor, you can exclude just that single metric point from future analysis.
## Key Capabilities
### What Happens When You Exclude a Metric Point
* **Alert Status**: The associated alert/metric-point from the scan is marked as "excluded"
* **Incident Display**: The metric-point shows as disabled in the corresponding incident
* **Historical Data**: All historical data remains intact and viewable for audit purposes
* **Monitor Status**: The underlying monitor continues to run normally
* **Other Alerts**: Alerts from the same scan for different metrics remain active
* **Future Analysis**: The excluded point is removed from future predictions and threshold calculations
* **Visibility**: Excluded metrics are clearly marked and can be reviewed at any time
### What Happens When You Re-include a Metric Point
* **Alert Reactivation**: The associated alert is marked as "active" again
* **Future Analysis**: The re-included metric-point is used in subsequent scan analysis
## Limitations
### Supported Monitors
The exclusion feature works with all monitor types with exception to (Schema drifts, Top value alerts, and Distribution alerts).
### Other Limitations
1. **Alert Requirement**: Currently, you can only exclude a metric point if an alert has been generated for it. Future versions will allow exclusion from plotted metrics without alerts
2. **Segment Support**: This feature is not available for Segments.
## Usage
To exclude a metric point,
1. navigate **Trends Page**
2. Select your target asset
3. Select the associated monitor and alert
4. Click the checkmark to exclude the metric. You will be prompted to confirm exclusion\
Only supported metrics will allow you to exclude the metric
# Metrics Inspector
This Document provides guidance on how to create and manage dashboards with multiple charts in Metrics Inspector, ensuring better data visualization and analysis.
{% hint style="info" %}
Feature must be enabled at deployment to allow using it:
* Set `telmai-ui-app-2-chart.metricsInspectorEnabled` to `true`
{% endhint %}
### 1. Creating a Dashboard
#### Steps:
1. Navigate to the **Metrics Inspector** page.
2. Click on the **+ (Add Dashboard)** button.\
3. Enter a name for the new dashboard.
4. Optionally, add a description.
5. Click **Create** to create the dashboard.
**Note:** You can create multiple dashboards within a project, allowing you to organize data efficiently.
### 2. Adding Charts to a Dashboard
#### Steps:
1. Select the dashboard where you want to add charts.
2. Click on the **+ Chart** button.
3. The "New Chart Inspector" form will appear.
4. Configure the data source and parameters:
* **Name**: Provide a unique name for the chart.
* **Source**: Select the data source for the chart.
* **Monitor**: Select the monitor for the chart. Based on the monitor, choose Dimensions/Segment/Group-By and attributes as needed.
* Dimensions - Represent the value of \ column name, while creating business metrics
* **Description** (optional): Add a brief description of the chart.
* Click **Save** to add the chart to the dashboard.
#### Example 1:
#### Example 2:
#### Example 3:
**Note:** You can add multiple charts within a single dashboard to provide a comprehensive view of your data.
### 3. Managing Dashboards and Charts
#### Editing or Deleting a Dashboard
1. Select the dashboard from the list.
2. Click the **Edit (Pencil Icon)** button.
1. Modify the name or description, and Save.
3. To delete a dashboard, Click the **Delete (Trash Icon)** button, and confirm the deletion.
#### Editing or Deleting Charts
1. Select the dashboard containing the chart.
2. Locate the chart and click on the **Edit (Pencil Icon)** button to modify.
3. To delete a chart, click the **Delete (Trash Icon)** button next to it.
# Data Quality Score
## Overview
The Data Quality (DQ) Score is a normalized, business-relevant measure of data health that provides a single indicator of a dataset's fitness for use. Expressed as a percentage from 0% to 100%, the DQ Score helps organizations quickly assess and monitor the overall quality of their data assets.
## Purpose
The DQ Score methodology ensures:
* **Consistency** - Standardized measurement across all data assets
* **Normalization** - Comparable scores regardless of data volume or complexity
* **Business Relevance** - Weighted dimensions that reflect organizational priorities
* **Actionability** - Clear identification of data quality issues requiring attention
## Core Dimensions
The DQ Score is calculated as a weighted average of four fundamental data quality dimensions:
### Completeness
Measures the extent to which required data fields are populated.
**Metric**: Percentage of required fields containing values\
**Score Range**: 0-100%
### Validity
Measures compliance with defined business rules and data constraints.
**Metric**: Percentage of records passing validation rules\
**Score Range**: 0-100%
### Freshness (Timeliness)
Measures whether data is up-to-date and meets timeliness requirements.
**Metric**: Binary indicator of freshness incidents\
**Score Range**: 0 or 100
### Integrity (Incident Health)
Measures operational stability through the volume of open data quality incidents.
**Metric**: Count of open incidents relative to threshold\
**Score Range**: 0-100
## Score Calculation
### Normalization Process
All source metrics are normalized to scores between 0 and 100 before being integrated into the final DQ Score calculation.
**Completeness (S\_C)**
```
S_C = Completeness percentage
```
Directly uses the percentage of populated required fields.
**Validity (S\_V)**
```
S_V = Validity percentage
```
Directly uses the percentage of records passing business rules.
**Freshness (S\_F)**
```
If No Freshness Incidents: S_F = 100
If Freshness Incidents Exist: S_F = 0 (or configured penalty score, e.g., 80)
```
**Integrity/Incidents (S\_Inc)**
```
S_Inc = MAX(0, 100 × (1 - (I_current / I_max)))
Where:
- I_current = Number of open incidents
- I_max = Maximum tolerable incident threshold (default: 20)
```
**Constraints**:
* If I\_current ≥ I\_max, then S\_Inc = 0
* If I\_current = 0, then S\_Inc = 100
### Final DQ Score Formula
```
DQ Score = ((S_C × W_C) + (S_V × W_V) + (S_F × W_F) + (S_Inc × W_Inc)) / W_Total
Where:
- S = Normalized dimension score (0-100)
- W = Dimension weight
- W_Total = W_C + W_V + W_F + W_Inc
```
**Output**: Value between 0 and 100
## Default Weights
Telmai provides industry-standard default weights that prioritize data accuracy and fundamental usability:
| Dimension | Default Weight | Rationale |
| ------------------------- | -------------- | ----------------------------------------------------------------------- |
| **Validity** | 40% | Highest priority - measures compliance with critical business rules |
| **Completeness** | 30% | Second priority - measures availability of required information |
| **Integrity (Incidents)** | 20% | High priority penalty - reflects operational stability and issue volume |
| **Freshness** | 10% | Contextual priority - importance varies by use case |
| **TOTAL** | **100%** | Simplifies calculation denominator |
#### Weight Customization
Weights can be adjusted per dataset to reflect specific business requirements:
* **Real-time systems**: Increase Freshness weight (e.g., 25-30%)
* **Analytical systems**: Prioritize Completeness and Validity
* **Mission-critical systems**: Increase Integrity/Incidents weight
## Configuration
{% hint style="info" %}
UI-based configuration for DQ Score is coming soon. Configuration is currently available via API only.
{% endhint %}
Configuration is managed via the [DQ Score APIs](/telmai/api-reference/dq-score-api). The following parameters can be set per dataset:
### Dimension Weights
Weights can be configured dynamically per dataset. All four weights must sum to 100.
### Incident Threshold (I\_max)
Configure the maximum tolerable incident threshold per dataset:
* **Default**: 20 open incidents
* **Low-tolerance assets**: 5-10 incidents
* **High-volume assets**: 30-50 incidents
The threshold should reflect:
* Dataset criticality
* Typical incident volumes
* Business impact tolerance
## Best Practices
#### Interpreting DQ Scores
| Score Range | Quality Level | Recommended Action |
| ----------- | ------------- | ----------------------------------------------------- |
| 90-100 | Excellent | Maintain current practices |
| 75-89 | Good | Monitor trends, address minor issues |
| 60-74 | Fair | Investigate dimension contributors, plan improvements |
| 0-59 | Poor | Immediate attention required, escalate issues |
To start using this feature, please refer to [DQ Score APIs](/telmai/api-reference/dq-score-api)
# Remediation
* [Data Binning](/telmai/remediation/data-binning)
* [Circuit Breaker](/telmai/remediation/circuit-breaker)
# Data Binning
#### Data Binning Policy
Telmai offers a data binning feature to help manage data quality issues at the source. This feature allows you to define a policy where Telmai monitors data correctness and categorizes your data into "good" and "bad" bins. Good data continues through your pipeline, while bad or suspicious data is flagged for review
This feature can help you ensure that only good (or expected) data is flowing into your ecosystem. By doing so, you can ensure that your costly pipelines are only running on healthy datasets
To learn more about using this feature, please check [Configuring Data-Binning](/telmai/monitoring-data/monitors-management/configuring-data-binning) documentation.
# Circuit Breaker
Using Telmai’s APIs, users can query information about their data health. That is querying any of the health KPIs or checking for ongoing policy violations.
One use case of this is circuit breaker.
Circuit breaker is a case where you want to switch off the ETL operation because bad data has been introduced into your system. This could be because processing costs are very high or because of other factors.
Here’re the steps you can follow to create the circuit breaker:
1. Trigger Telmai data scan ([API reference](/telmai/api-reference/upload-data-api))
2. Check for Telmai job completion
1. Polling mechanism ([API reference](/telmai/api-reference/upload-data-api/track-job-status))
2. Using webhooks for job notification
3. Check for alerts ([API reference](/telmai/api-reference/upload-data-api/check-for-alerts))
4. Decide best action
These APIs can be integrated in ETL orchestration tools like Airflow or other.
# Integrations
Telmai integrates with external tools to help you route alerts, catalog data quality results, and manage reporting workflows.
| Integration | Description |
| --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [Jira](/telmai/integrations/jira-integration) | Automatically create Jira issues from Telmai incidents and alerts |
| [Slack](/telmai/integrations/slack-integration) | Send alert notifications to Slack channels |
| [Alation](/telmai/integrations/alation-integration) | Push Telmai data quality results to your Alation data catalog |
| [Data Quality External Reporting](/telmai/integrations/data-quality-external-reporting) | Centralize DQ scan results into an external destination table for cross-asset reporting |
| [Jobs Status Notification](/telmai/integrations/jobs-status-notification) | Receive webhook callbacks when dataset processing jobs change state |
| [Telmai MCP Server](/telmai/integrations/telmai-mcp-server) | Connect AI assistants (Claude, VSCode Cline) to Telmai via the Model Context Protocol |
# Catalog Integration
Telmai integrates with your data catalog for dataset discovery, metadata enrichment, CDE coverage tracking, and data governance. Catalog metadata enriches Telmai's monitoring with business context so your team can discover, prioritize, and monitor critical datasets faster.
## Supported Catalogs
* [Zeenea](/telmai/integrations/catalog-integration/zeenea)
# Zeenea
Telmai integrates with [Zeenea](https://zeenea.com/) to connect data quality monitoring with your Zeenea data catalog. Zeenea metadata enriches Telmai's monitoring configuration and AI-powered recommendations, helping your team discover, prioritize, and monitor critical datasets faster.
## Overview
Telmai pulls the following metadata from Zeenea:
| Metadata type | How Telmai uses it |
| --------------------------- | --------------------------------------------------------------- |
| Business glossary terms | Informs AI-generated monitoring rules with business definitions |
| Lineage data | Powers asset prioritization and lineage visualization |
| Sensitivity classifications | Surfaces sensitivity context alongside monitors |
| Stewardship information | Identifies the data steward responsible for each asset |
Column-level metadata from Zeenea: CDE tiers, sensitivity classifications, and glossary terms.
{% hint style="info" %}
The Zeenea catalog integration is supported for the following Telmai data connection types: **BigQuery**, **Databricks**, **Snowflake**, and **OneLake**.
{% endhint %}
***
## Step 1: Create an API Key in Zeenea
Telmai authenticates to Zeenea with an API key. To create one:
1. In your Zeenea (Actian Data Intelligence) instance, go to **Administration > API Keys**.
2. Create a new API key and give it a descriptive name (e.g., `telmai-integration`).
3. Select the **scanner** permission for the key.
4. Save the key. Zeenea displays the secret once, inside a configuration snippet:
```hocon
api-key {
secret = "eyJ0eXAiOiJKV1QiLCJhbGciOi..."
}
```
{% hint style="warning" %}
When copying the API key into Telmai, copy **only the secret value** — the token between the quotation marks (it starts with `eyJ`). Do not include the surrounding `api-key { secret = "..." }` wrapper or the quotation marks themselves.
{% endhint %}
***
## Step 2: Connect to Zeenea
In Telmai, go to **Administration** and select **Catalog** from the left menu.
The Catalog connection form in Administration, showing Catalog Type, API Endpoint, API Key/Token, and Sync Interval.
Enter the following details:
| Field | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Catalog Type | ADI Catalog (preset for Zeenea) |
| API Endpoint | Your Actian Data Intelligence instance URL (e.g., `https://your-org.zeenea.app/`) |
| API Key / Token | The API key secret created in Step 1 (only the token value, not the full `api-key { ... }` snippet). Leave blank when updating other fields to keep existing credentials. |
| Sync Interval (hours) | How often Telmai syncs metadata from Zeenea. Accepts values between 1 and 168 hours. |
Save the connection. Once connected, Telmai displays **Connected to ADI Catalog** and two additional actions become available:
* **Sync Now** — triggers an immediate metadata sync
* **Remove Connection** — disconnects Telmai from Zeenea
{% hint style="info" %}
After connecting, go to **Catalog Browser** to explore discovered datasets and link them to your Telmai data connections.
{% endhint %}
***
## Step 3: Catalog Browser
**Catalog Browser** is available in the sidebar under **Configure**. It is the central place for browsing catalog-discovered datasets and managing which tables are connected for monitoring in Telmai.
The Catalog Browser, with catalog connections on the left and discovered sources on the right.
### Catalog Connections
The left panel lists your Telmai data connections and their catalog link status:
* **Connected** — the Telmai connection is linked to a catalog source. Telmai maps catalog metadata to assets from this connection.
* **Not Connected** — the Telmai connection has not yet been linked to a catalog source.
Use the link icon next to each connection to associate it with the corresponding catalog source.
### All Sources
The right panel shows all tables discovered from the catalog. Tables are grouped into two sections:
* **Connected tables** — already linked to Telmai assets
* **Non connected tables** — discovered from the catalog but not yet monitored in Telmai
Each table shows the following columns:
| Column | Description |
| -------------------- | ----------------------------------------------------------------------- |
| Score | Current data quality score for the table |
| Details | Number of columns |
| Last changed | When the table was last updated in the catalog |
| Why it's recommended | AI-generated explanation of why Telmai recommends monitoring this table |
***
## Intelligent Recommendations
Telmai uses catalog metadata to prioritize which tables most need monitoring. The **Why it's recommended** column in the Catalog Browser surfaces this reasoning for each table, drawing on:
* Tier classification (e.g., Tier 1 critical data elements)
* Presence of sensitive or PII data
* Financial or business-critical context from the catalog
* Stewardship completeness
Select any table in the Catalog Browser to open its **Catalog Details**, which shows the dataset's catalog metadata, the full AI recommendation with its reasoning and priority, connection status, and suggested timestamp columns.
The Catalog Details view, with the AI recommendation and its reasoning for a Tier 1 dataset.
***
Metadata syncs on the interval configured in **Administration > Catalog**. Use **Sync Now** to trigger an immediate sync at any time.
***
## Business-Aware Rule Generation
When suggesting monitoring rules, Telmai uses the business definitions from the Zeenea glossary rather than relying on technical column names alone, making suggestions easier to evaluate and accept.
***
## Lineage Visualization
Telmai displays upstream and downstream data dependencies sourced from Zeenea lineage data, giving you context for how quality issues propagate through your data pipelines.
# Atlan Integration
## Overview
This guide walks you through setting up the Telmai connector in Atlan so that data quality metadata — including DQ dimensions, monitor results, and policy health — surfaces directly on your Atlan assets. Once configured, your data consumers can assess quality without leaving the catalog. The Telmai–Atlan integration embeds quality intelligence directly within the Atlan catalog, so stakeholders can assess data trustworthiness at the point of discovery — without switching tools or second-guessing data reliability.
## Prepare Atlan
In Atlan, create a new workflow connector for your data source. This registers your data in the Atlan catalog and is required for the workflow mapping step that follows in Telmai.
* Use a recognizable name — e.g., ***Snowflake\_Enterprise*** — for easy reference in the mapping step.
## Prepare Telmai
In your Telmai environment, choose the specific connection which points to the assets under consideration — Let's call it ***Snowflake\_DQ***. Ensure it points to the same underlying data source as ***Snowflake\_Enterprise***; this is the pair the workflow will map together.
#### Create and Scan an Asset in Telmai
Using ***Snowflake\_DQ***, create a new asset — Let's call it ***product\_catalog*** — and run a scan. The sync workflow pulls quality metadata from this scan result. If the asset has never been scanned, the Atlan panel will not display any DQ data after the sync.
## Configure the Telmai Workflow
In Atlan, navigate to Workflows → Marketplace. Search for "Telmai" to locate the Telmai connector package. Select it and click Setup Workflow.
Fig. 1 — Atlan Workflow Marketplace: search for "Telmai" to find the Telmai connector.
### Configure Credentials
The workflow setup opens on the Credentials tab. Enter your Telmai environment details:
* Telmai API URL - Your Telmai environment base URL, e.g.
* Telmai Tenant ID - Your tenant identifier — available from your Telmai admin dashboard.
* Telmai API Key - Generate or retrieve an API key from your Telmai account settings.
* Click Test Authentication to validate before proceeding
Fig. 2 — Step 1: Enter Telmai API URL, Tenant ID, and API Key. Use Test Authentication to verify before moving on.
### Map Connections
Click Next to advance to the Connection Mapping step. Here you pair your Atlan connection with the corresponding Telmai connection.
* Under Atlan Connection, select ***Snowflake\_Enterprise***
* Under Telmai Connection, select ***Snowflake\_DQ***
* Click Add to Connection Mapping to confirm the pair
* Run Preflight Checks - to verify permissions are in place before running.
* Click Run (immediate) or Schedule & Run (recurring sync)
Fig. 3 — Step 2: Map Snowflake_Enterprise to Snowflake_DQ, run preflight checks, then Run or Schedule & Run.
## Verify the Integration
### Confirm Asset Visibility and DQ Data in Atlan
Once the workflow run completes successfully, navigate to the Assets tab in Atlan and search for the asset ***product\_catalog***. Open the asset detail page, then select the Telmai tab in the right-hand panel to view the synced quality metadata.
Fig. 4 — The Telmai panel on an Atlan asset showing DQ dimension scores, policy results, and the Open in Telmai deep-link.
## Data Quality Metadata in Atlan
The Telmai panel on each mapped asset exposes three sections:
### DQ Dimensions
A scored breakdown of the asset's quality health across five dimensions (visible in Fig. 4 above):
Completeness
Measures the percentage of complete records
Uniqueness
Measures the percentage of unique records
Correctness
Measures the percentage of correct values
Freshness
Measures how recent the data is
Consistency
Measures data consistency across sources
### Policies
A summary of monitor results — how many quality monitors passed vs. failed. In the example above (Fig. 4), the asset has 8 policies that failed and created alerts. This headline indicator lets data stewards spot quality regressions without logging into Telmai.
### Open in Telmai
A direct deep-link to the asset's full detail page in Telmai — useful for investigating specific dimension failures, reviewing historical trends, or setting up new monitors.
# Alation Integration
Telmai integrates with [Alation](https://www.alation.com/) to push data quality results directly into your Alation data catalog. The integration is one-directional: Telmai sends results to Alation, not the other way around.
Once an asset is mapped, Telmai automatically pushes updated data quality results to Alation after every completed scan of that asset — no manual sync is required.
## Overview
Setup involves steps in both platforms:
| Platform | What you do |
| ----------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Alation** | Create a custom Rich Text field named **Telmai Observability**, generate a Refresh API Token, and note your numeric user ID |
| **Telmai** | Connect to Alation using the token, then map Telmai assets to Alation tables |
***
## Step 1: Prepare Alation
### Create the Custom Field
Telmai writes results into a [Rich Text custom field](https://docs.alation.com/en/latest/steward/TemplatesAndCustomFields/AboutTemplatesAndFields.html#rich-text) in Alation. The field must be named exactly:
```
Telmai Observability
```
Create this field in Alation under **Settings > Customize Catalog > Custom Fields** and attach it to the Table object type.
### Generate a Refresh API Token
Telmai authenticates to Alation using a Refresh API Token tied to a specific user account.
To generate one in Alation:
1. Sign in as the Alation user account Telmai will use
2. Click your user avatar in the upper right and select **Account Settings** (called **Profile Settings** in some versions)
3. Open the **Authentication** tab
4. Click **Create Refresh Token**, give the token a name, and click **Create Refresh Token** again
5. Copy the **Token Secret Key** shown — this is the value you will paste into Telmai
For full details, see Alation's guide on [generating API tokens](https://developer.alation.com/dev/docs/authentication-into-alation-apis).
{% hint style="warning" %}
Store this token securely. Telmai uses it to write data to your catalog on an ongoing basis.
{% endhint %}
### Find Your User ID
Telmai also needs the **numeric Alation user ID** of the account that owns the token. This is a number (for example, `81`) — not the login username or email address.
The quickest way to find it: open your own profile page in Alation and look at the browser address bar — the number at the end of the URL is your user ID:
```
https://your-org.alationcloud.com/user/81/
^^ your user ID
```
***
## Step 2: Connect Telmai to Alation
In Telmai, go to the **Administration** page and open **Settings > Alation**.
The Alation connection form under Administration — enter your Alation URL, User ID, and Refresh API Token, then click Connect.
Fill in the three fields using the values from Step 1:
* **Alation Cloud URL** — the base URL of your Alation instance (e.g., `https://your-org.alationcloud.com`)
* **User ID** — the numeric Alation user ID (not the login username or email)
* **Refresh API Token** — the Token Secret Key you copied when creating the token
Click **Connect**. Once the connection succeeds, the panel shows an **Alation is successfully integrated** confirmation, and an **Asset Mappings** section appears below it (used in Step 3).
The connection can also be managed via API — see [Add or Update Integration](/telmai/api-reference/alation-integration-apis#add-or-update-integration).
#### Modifying or Removing the Integration
To update the integration, enter new credentials in the same panel and reconnect. To remove it entirely, click **Remove Connection** in the same panel or call the [Remove Integration](/telmai/api-reference/alation-integration-apis#remove-integration) endpoint.
***
## Step 3: Map Assets
Data mappings link Telmai data assets to their corresponding tables in Alation. Without a mapping, Telmai cannot write quality results to the correct catalog entry.
Mappings are managed from the **Asset Mappings** section of the **Settings > Alation** panel, which appears once the connection from Step 2 is established. All mapping operations are also available via the [Alation Integration APIs](/telmai/api-reference/alation-integration-apis#mappings).
The Asset Mappings section — use Auto-map or + Add Mapping to create mappings; each row links a Telmai asset to an Alation table.
### Automatic Mapping (Recommended)
Telmai scans both platforms and creates mappings automatically by matching Telmai assets to Alation tables.
Click **Auto-map** to start the mapping job. By default, existing mappings are kept; use the dropdown next to the button to overwrite them instead. The job runs in the background — via API, the [Start Automatic Mappings](/telmai/api-reference/alation-integration-apis#start-automatic-mappings) endpoint returns a `job_id` for tracking progress.
To check on a running or completed job via API, pass the `job_id` to [Get Mapping Job Result](/telmai/api-reference/alation-integration-apis#get-mapping-job-result). On completion, the result lists each new mapping with its Telmai asset ID and the matched Alation data source, schema, and table (IDs and names).
### Manual Mapping
To control mappings individually, click **+ Add Mapping** and pair a Telmai asset with the corresponding Alation table. Via API, use [Create a New Mapping Manually](/telmai/api-reference/alation-integration-apis#create-a-new-mapping-manually) with the Telmai asset ID and the Alation table ID.
### Managing Mappings
Existing mappings are listed in the **Asset Mappings** table, showing each Telmai asset alongside its mapped Alation data source, schema, and table. To remove a mapping, click the delete icon in its **Actions** column, or call [Delete a Mapping](/telmai/api-reference/alation-integration-apis#delete-a-mapping) with the Telmai asset ID.
***
## Step 4: Verify the Integration
To confirm everything is wired up:
1. In Telmai, run a scan on an asset that has a mapping
2. After the scan completes, open the mapped table's catalog page in Alation
3. Check the **Telmai Observability** field on the table — it should now show the data quality results from the scan
From this point on, the field is refreshed automatically after every scan of the mapped asset. If the field is empty, verify that the asset is mapped (Step 3) and that at least one scan has completed since the mapping was created.
***
## API Reference
All Alation integration operations are also available via API. See [Alation Integration APIs](/telmai/api-reference/alation-integration-apis) for the full reference.
# Jira Integration
You can now integrate your Jira instance with Telmai to simplify issue creation and tracking. This integration allows you to create and manage Jira tickets directly from generated alerts in Telmai.
Tickets can be created manually or dynamically with incidents.
## Configuring Jira Integration
Only tenant administrators are able to configure Jira integration.
### Connect to Jira
To integrate your Jira instance:
1. Navigate to the "**Administration**" page.
2. Choose connection Mode (OAuth or API Token).
3. Click **Connect.** You’ll be redirected to Jira for authentication.
4. You have now connected to your Jira instance.
### Create Ticket Templates
Templates can be used for defining what fields to be used when creating a new ticket. This can be helpful when your Jira instance requires specific parameters or you want to automate the ticket creation process.
To define the templates:
1. Navigate to the "**Administration**" page.
2. Click "+Add Template" button
3. The below dropdown menu will appear
4. Fill the required details (Template name & Description)
5. Specify the target **Jira Project** and **Issue Type**
6. New fields will show that are required based on the Issue Type requirement
7. You can add more fields using the "**+Add Field**" button
8. Once done, click "**Save**" button, and the new template will show up under list of templates
## Create Jira Tickets
Users have the option of creating tickets dynamically when incidents are opened or manually for existing incidents or alerts.
{% hint style="info" %}
Telmai only creates Jira tickets and links them to existing incidents or alerts within Telmai application. Telmai does not automatically resolve the tickets.
{% endhint %}
### Dynamically Create Tickets
To allow for auto-ticket creation, user needs to specify which [template](#create-ticket-templates) to be used at a monitor level. This means a Jira ticket will be created once an incident is observed.
To setup dynamic ticket creation:
1. Navigate to "**Alerting Monitors**" page and select target asset
2. Click on the target monitor to see the monitor configuration
3. Scroll to the bottom of the configuration to see the "**Ticketing**" section like below
4. Select "Create ticket automatically" and choose the desired template
5. Save the new configuration
### Manually Create Tickets
Users can also create tickets manually on existing alerts, or incidents.
#### Incident scoped tickets
To create a ticket associated to a given incident:
1. Navigate to the **“Overview”** page to see the [incidents table](/telmai/monitoring-data/incident-portal)
2. The Ticket column will have "**Create**" button if the incident is applicable for ticket creation
{% hint style="info" %}
You can only create a ticket at incident level if no ticket already exists.
Tickets can only be created for open incidents.
{% endhint %}
#### Alert scoped tickets
To create a ticket associated to a given alert:
1. Navigate to the **“Trends & Alerts”** page.
2. Choose a scan with alerts.
3. Click the **Create Issue** button.
4. A modal will appear to request additional details:
5. Specify the **project** and **issue type**.
6. Telmai will auto-fill the **summary** and **description**, but you can edit them if needed.
7. Click **Create** to generate the Jira ticket.
After creating alert-scoped tickets, you can view and manage them directly by clicking "**See Issues**" button on the same page. This option will allow you to:
1. Click the ticket to navigate to your Jira instance
2. Unlink the ticket from Telmai
3. View ticket status
## Related Documentation
* [User-defined Monitors](/telmai/monitoring-data/user-defined-monitors)
* [Monitors Management](/telmai/monitoring-data/monitors-management)
* [Ticket Integration API](/telmai/api-reference/ticket-integration-api)
# Slack Integration
This article explains how to integrate Slack for receiving alert notifications from Telmai.
## Overview
Setting up Slack notifications requires two steps:
1. Create a Slack app and obtain a Bot OAuth token
2. Add a Slack notification channel in Telmai and assign it to an alert policy
***
## Step 1: Create a Slack App
1. Navigate to [Slack API apps](https://api.slack.com/apps)
2. Sign in and click **Create an App**
3. Set **App Name** to `Telmai Notifier`
4. Select the Slack workspace where the bot should post messages, then click **Create App**
5. In the left navigation, select **OAuth & Permissions**
6. Scroll to the **Scopes** section. Under **Bot Token Scopes**, click **Add an OAuth Scope** and add:
* `chat:write`
* `chat:write.public`
7. Scroll back to the top of the **OAuth & Permissions** page and click **Install App to Workspace**
8. In the confirmation dialog, click **Allow**
9. Navigate to Interactivity & Shortcuts in the left sidebar
10. Toggle Interactivity to On
11. Set a Request URL: [https://env1.telm.ai/api/backend/slack/interactions](https://dev.telm.ai/api/backend/slack/interactions) (replace host with proper one in case of private cloud deployment)
12. Copy the **Bot User OAuth Access Token** — you will need it in the next step
***
## Step 2: Add a Slack Notification Channel in Telmai
Use the token from Step 1 to create a Slack notification channel in Telmai, then assign it to an alert policy.
See [Configuring Notifications](/telmai/monitoring-data/monitors-management/configuring-notifications) for the full walkthrough.
# Jobs Status Notification
This page describes a way to listen for updates happening with your connected datasets through setting up webhooks.
Telmai allows you to setup webhooks that get triggered when processing job status changes. If defined, the callback gets triggered when:
* Job is running
* Intermediate job state changes
* Job is completed
### Definition
The following params are sent as part of the body
Param
Description
tenant
Associated tenant ID where the callback happened
source
Associated dataset source ID
jobId
Associated job ID
status
Job status (IN_PROGRESS, FINISHED, FAILED, PREPARING, BATCH_WAIT)
details
Description of job status
### Example Payload
When Telmai triggers a webhook, it sends a `POST` request to your URL with a JSON body similar to:
```json
{
"tenant": "your-tenant-id",
"source": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"jobId": "job-20260417-001",
"status": "FINISHED",
"details": "Job completed successfully"
}
```
### Usage
To use this feature, you will need to:
1. Navigate to the configuration page
2. Open the settings for the associated dataset
3. Click on **Job Notifications Settings**
4. In the **Webhook URL** field, enter the endpoint URL where Telmai should send callbacks
5. Click **Save** to apply the configuration
{% hint style="info" %}
Your webhook endpoint must be publicly accessible and respond with an HTTP `2xx` status code. Telmai retries delivery on failure.
{% endhint %}
# Data Quality External Reporting
This page explains the Centralized DQ Monitor Scan Reporting feature, which generates a comprehensive, standardized report for every Data Quality (DQ) Monitor scan and appends the results to a designated external table. This mechanism centralizes DQ metrics from various sources and monitors, providing a unified, historical view of data quality performance.
***
## Report Structure (External Destination Table Schema)
The external table is the central repository for all DQ scan results. It provides a standardized format that allows users to query and analyze DQ performance across all projects, data assets, and monitors.
| Column Name | Data Type | Description |
| --------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------- |
| project\_id | `Long` | Identifier for the project where the data asset resides. |
| data\_asset\_id | `String` | Identifier for the specific data asset (e.g., table/stream) being monitored. |
| monitor\_Id | Long | Unique identifier for the Monitor that performed the check. This is the primary key for tracking the rule execution. |
| scan\_timestamp | `Timestamp` | The exact time the DQ job completed execution. |
| total\_records\_failed | `Integer` | Count of records that failed the specific Monitor's check. |
| total\_records\_scanned | `Integer` | Total count of records processed by the job for this check. |
| record\_id\_attribute\_name | `String` | The name of the primary key/ID attribute used to uniquely identify records in the data asset. |
| record\_id\_sample | `Array of Strings` | A sample of up to 100 failed record IDs. This sample aids in immediate investigation and debugging. |
***
### **User Actionability**
* To check DQ status: Users should query the External Destination Table, filtering by `data_asset_id` and `scan_timestamp`.
* To debug failures: Users can use the `record_id_sample` along with the `record_id_attribute_name` to look up the failing records directly in the source data asset for diagnosis.
***
## Supported Destinations
DQ scan results can be written to the following external storage destinations:
* **Amazon S3**
* **Azure File Storage**
* **Google Cloud Storage (GCS)**
## Configuring Reporting
Reporting is configured via API. Please refer to the [DQ Reporting APIs](/telmai/api-reference/dq-reporting-apis) page for the full reference.
# Telmai MCP Server
## Overview
Telmai's Model Context Protocol (MCP) server enables seamless integration with AI assistants like Claude, allowing you to interact with your data quality platform directly through conversational interfaces. This integration provides access to Telmai's comprehensive set of tools for monitoring, alerting, and data quality management.
### What is MCP?
The Model Context Protocol (MCP) is a standard that allows AI applications to securely connect to external services and data sources. Telmai's MCP server exposes your data quality platform's capabilities as tools that can be invoked conversationally through compatible AI assistants.
## Getting Started
#### Prerequisites
Before connecting to Telmai's MCP server, ensure you have:
* Access to Claude Desktop, Claude.ai, or a compatible MCP client (e.g., VSCode with Cline)
* Valid Telmai-Okta credentials
* MCP Server Details:
* OAuth client credentials (Client ID and Client Secret) for your application
* MCP URL for your environment
{% hint style="warning" %}
**Important**:
1. You must be added to the associated application in Okta to complete OAuth authentication
2. MCP server URL and client credentials must be obtained from your Telmai deployment owner
{% endhint %}
### Connecting to Claude
#### Method 1: Claude Desktop or Claude.ai
Claude supports secure OAuth-based authentication, providing the most seamless integration experience.
**Step 1: Access Connector Settings**
1. Open your Claude Desktop application or visit [claude.ai](https://claude.ai/)
2. Click your **Profile Name** or **Avatar** (usually located on the left sidebar or top corner)
3. Navigate to **Settings**
4. Look for the **Connectors** or **Integrations** section
**Step 2: Add the Custom Connector**
1. In the Connectors section, click **Add custom connector**
2. A configuration dialog will appear with the following fields:
* **Name**: Enter a descriptive name (e.g., "Telmai MCP")
* **URL**: Enter your Telmai MCP server URL (e.g., `https://mcp.yourdomain.telmai.dev/yourorg/mcp`)
3. Click **Advanced Details** to expand additional options
4. Enter your OAuth credentials:
* **Client ID**: Your Telmai OAuth application client ID
* **Client Secret**: Your Telmai OAuth application client secret
**Step 3: Initiate the OAuth Flow**
1. Click **Connect** for the newly added server
2. Claude will recognize that OAuth authentication is required
3. Your default web browser will automatically open to the Telmai-Okta login screen
4. Enter your **Telmai-Okta credentials** and sign in
5. Review and approve the requested permissions
**Step 4: Verify the Connection**
1. After successful authorization, your browser will redirect back to Claude
2. Return to the **Connectors** section in Claude settings
3. Your Telmai MCP server should now display as **Connected** with a green indicator
**Step 5: Start Using Telmai**
1. Start a new chat in Claude
2. You can now ask questions about your data quality, create monitors, review alerts, and more
3. Claude will automatically use Telmai's MCP tools when relevant to your queries
**Example prompts to try:**
* "Show me my recent incidents"
* "What assets are available in my production project?"
* "Create a completeness monitor for my customer table"
* "Get the profiling data for my sales dataset"
***
### Connecting with VSCode Cline
Cline is a VSCode extension that supports MCP servers but requires bearer token authentication instead of OAuth.
> **Note**: Bearer tokens expire after a set period (typically 1 hour). You'll need to regenerate and update the token when it expires.
#### Step 1: Generate Bearer Token
Run the following command in your terminal to generate an access token:
```bash
CLIENT_ID=""
CLIENT_SECRET=""
TOKEN_RESPONSE=$(curl -s -X POST "https:///oauth2/default/v1/token" \
-u "${CLIENT_ID}:${CLIENT_SECRET}" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&scope=mcp:access")
echo $(echo "$TOKEN_RESPONSE" | jq -r '.access_token')
```
**Prerequisites:**
* Replace `` and `` with your actual credentials
* Ensure `curl` and `jq` are installed on your system
The command will output your bearer token. Copy this token for the next step.
#### Step 2: Configure MCP Server in Cline
1. Open VSCode with the Cline extension installed
2. Click the **MCP server settings** button in the Cline interface
3. Select **Remote Servers**
4. Click **Add server details**
5. Provide the following information:
* **Name**: `telmai` (or any descriptive name)
* **URL**: Your Telmai MCP server URL (e.g., `https://mcp.dev.telm.ai/yourorg/mcp`)
#### Step 3: Add Authorization Header
1. Click **Edit Configuration** to open the MCP settings JSON file
2. Locate the server configuration you just added
3. Add a `headers` section with your bearer token:
```json
{
"mcpServers": {
"telmai": {
"url": "https://mcp.yourexampledomain.telm.ai/commonstock/mcp",
"type": "streamableHttp",
"disabled": false,
"autoApprove": [],
"headers": {
"Authorization": "Bearer eyJraWQiOiJ..."
}
}
}
}
```
4. Save the configuration file
#### Step 4: Start Using Telmai in Cline
1. Start a new Cline conversation in VSCode
2. Cline can now access Telmai's MCP tools
3. Try asking questions about your data quality or requesting specific operations
**Example prompts:**
* "List all my data assets in Telmai"
* "Show me open incidents from the last week"
* "What monitors are configured for my warehouse connection?"
#### Token Refresh
When your bearer token expires, you'll need to:
1. Generate a new token using the curl command from Step 1
2. Update the `Authorization` header in your Cline configuration
3. Restart Cline or reload the MCP server connection
***
## Available Tools
Once connected, your AI assistant has access to the following Telmai capabilities:
* Asset Management
* Connection Management
* Monitors Management
* Incidents & Alerts
* Data Quality Metrics
* Integration & Workflow
* Project & User Management
***
## Security Considerations
#### OAuth Authentication (Recommended)
* Uses industry-standard OAuth 2.0 protocol
* Tokens are securely managed by Claude
* Automatic token refresh
* Immediate revocation through Okta admin console
* Audit trail of access events
#### Bearer Token Authentication
* Tokens expire after a set period (typically 1 hour)
* Tokens should be treated as passwords—never commit them to version control
* Regenerate tokens regularly
* Revoke compromised tokens immediately through Okta
#### Best Practices
1. **Never share your Client Secret or Bearer Tokens**
2. **Use separate OAuth applications** for production and development environments
3. **Regularly audit** connected applications in your Okta dashboard
4. **Revoke access** for applications you no longer use
***
## Troubleshooting
#### OAuth Connection Issues
**Problem**: Browser doesn't open during OAuth flow
* **Solution**: Check your default browser settings and ensure pop-ups are allowed for Claude
**Problem**: "User not authorized" error
* **Solution**: Verify that your user account has been added to the Telmai application in Okta
**Problem**: Connection appears successful but tools don't work
* **Solution**: Restart Claude and try reconnecting. Check that your OAuth client has the correct scopes configured
#### Bearer Token Issues
**Problem**: "Unauthorized" errors when using tools
* **Solution**: Your token has likely expired. Generate a new bearer token and update your configuration
**Problem**: Token generation command fails
* **Solution**: Verify your Client ID and Client Secret are correct. Ensure `curl` and `jq` are installed
#### General Issues
**Problem**: Can't see Telmai's tools in Claude
* **Solution**: Verify the connection shows as "Connected" in settings. Start a new conversation to refresh available tools
**Problem**: Tools return "not found" errors
* **Solution**: Check that you're using the correct MCP server URL for your organization
# Authentication and Access Control
Logging in to Telmai is a simple process. You can sign in using an email and password or through a Single Sign-On (SSO) provider if your organization has it configured.
## Sign in to Telmai
### General Login Steps
1. Go to the Login Page: Navigate to the Telmai login page.
2. Enter Credentials: Use your registered email address and password.
3. Click "Sign In": Click the sign-in button to access your Telmai account.
### Single Sign-On (SSO) Login Steps
Many organizations use SSO for a more secure and streamlined login experience. If your company uses an identity provider (IdP) like Okta, you'll use this method to log in.
1. Go to the Login Page: Navigate to the Telmai login page.
2. Enter Credentials: Use your registered email address.
3. If SSO is configured for your account, you will be navigated to the IDP option. If multiple SSOs are configured, you will need to select the desired one.
4. You will be redirected to your organization's identity provider login page.
5. Enter your Company Credentials: Use your standard company username and password to authenticate.
6. Redirect to Telmai: After successful authentication with your IdP, you'll be redirected back to the Telmai Account.
Please click [here](/telmai/authentication-and-access-control/sso-configuration) to learn more about setting up your SSO configuration.
### Sign in Issues
Potential issues you may face when signing in:
* Account locked out
* User not added to SSO group
* User not added to Telmai account
## Managing Users
Telmai allows you to create, edit, and delete users via the Manage users UI menu:
### Role-based Access
Telmai supports project-scoped permissions. Tenant admins are able to modify these permissions accordingly:
| Role | Add/Modify Users | Add, Edit or Delete Source | Scan Source / Schedule Scans | View scan results |
| ------------ | ---------------- | -------------------------- | ---------------------------- | ----------------- |
| Tenant Admin | x | x | x | x |
| Editor | | x | x | x |
| Operator | | | x | x |
| Viewer | | | | x |
To modify user roles,
1. Click “**Manage Tenant Users**” under the user menu
2. Click on the user you would like to modify permissions for
3. “**Project Permissions**” table with different roles
4. Select appropriate roles
5. Click Save
# SSO Configuration
This document provides step-by-step instructions on how to configure Single Sign-On (SSO) on Telmai. Follow the steps below to set up SSO seamlessly.
### Step 1: Navigate to SSO Settings
1. Log in to the Telmai.
2. Go to **Administration**.
3. Click on **Single Sign-On (SSO) Settings**.
### Step 2: Configure SSO Details
1. Enter a unique **Login URL Suffix**
* This will be used to list IDPs available for the user
2. Provide client credentials:
* **Client ID**: Enter the ID provided by your IdP.
* **Client Secret**: Enter the secret key provided by your IdP.
3. Provide SSO endpoints or click **Provide Issuer URL**:
* You can enter the below details manually:
1. **Issuer**: Enter the Issuer URL from your IdP.
2. **Authorization**: Enter the Authorization endpoint.
3. **Token**: Enter the Token endpoint.
4. **JWKS**: Enter the JWKS (JSON Web Key Set) URL.
5. **User Info**: Enter the User Info endpoint.
* **OR** Click on **Provide Issuer URL** to open a modal dialog.
1. Enter the **Issuer URL** from your IdP and click **Retrieve Endpoints**.
4. Click **Save** to store the configuration.
5. You can use the toggle button to Enable/Disable the setting. You can also **update** or **delete** the setting at anytime.
***
### Step 3: SSO Testing and Verification
1. Navigate to the login page
* Example:
2. Click on **Sign in with IDP.**
3. You will be redirected to Issuer URL. Enter the required details and click Sign In.
If authentication is successful, you should be redirected to the Telmai dashboard.
# Microsoft Entra IDP Setup
This page describes how to setup Microsoft Entra ID as IDP
### Register an OIDC App in Microsoft Entra ID
Here's how to register an application in Microsoft Entra ID for OpenID Connect (OIDC):
1. **Log in to Azure Portal:** Navigate to and sign in with your Azure account.
2. **Navigate to App Registrations:**\
Go to **Microsoft Entra ID** > **App registrations** > **+ New registration**.
3. **Register the Application:**
* **Name:** Enter a descriptive name for your application (e.g., `Telmai-OIDC-App`).
* **Supported account types:** Choose the option that aligns with your organization's policies. Typically, **"Accounts in this organizational directory only"** is selected.
* **Redirect URI:** Enter the redirect URI provided by Telmai. This will likely follow the format: `https://.okta.com/oauth2/v1/authorize/callback`. **Please confirm the exact redirect URL with Telmai.**
4. **Post Registration Configuration:** Once the app is created, open its overview page.
* **Retrieve Application (client) ID and Directory (tenant) ID:** Copy and securely store these IDs. You will need them later.
* **Create a Client Secret:**
* Go to **Certificates & Secrets** > **+ New client secret**.
* Add a description for the secret and choose an expiration timeframe.
* Click **Add**.
* **Copy the Value** of the newly created client secret immediately. This value will not be shown again.
5. **Construct the Well-known Configuration URL:** Use the following format, replacing `` with the Directory (tenant) ID you copied earlier:
```
https://login.microsoftonline.com//v2.0/.well-known/openid-configuration
```
6. **Open the open-id configuration URL in browser and retrieve the following from the open-id configuration**
1. Issuer
2. authorization\_endpoint
3. token\_endpoint
4. jwks\_uri
5. userinfo\_endpoint
7. **Assign API Permissions to the App:**
* Under **API permissions**, click **+ Add a permission**.
* Select **Microsoft Graph** > **Delegated permissions**.
* Search for and add the following permissions:
* `openid`
* `profile`
* `offline_access`
* Click **Add permissions**.
* Finally, click **Grant admin consent for your tenant**.
8. **Change Assignment Settings**
1. Open the registered application (From Step 3) from Enterprise applications and navigate to Manage->Properties.
1. Under Assignment Required? - Selection "Yes" as the option
2. Save the settings
# Auth0 Setup
This guide outlines the steps to register an application in Auth0 for use as an OpenID Connect (OIDC) provider.
### Register an OIDC App in Auth0
#### 1. Log in to Auth0 Dashboard
Go to [https://manage.auth0.com](https://manage.auth0.com/) and sign in to your Auth0 tenant.
#### 2. Navigate to Applications
From the dashboard, go to **Applications > Applications** and click **+ Create Application**.
#### 3. Register the Application
Complete the following fields in the "Create Application" dialog:
* **Name:** Enter a descriptive name for your application (e.g., `Telmai-OIDC-App`).
* **Application Type:** Choose `Regular Web Applications`.
Click **Create**.
#### 4. Configure the Application
After creating the application, configure its settings:
* **Callback URL:** Enter the redirect URI provided by Telmai. This will likely follow the format: `https://.okta.com/oauth2/v1/authorize/callback`. **Please confirm the exact redirect URL with Telmai.**
* Under **Advanced Settings -> OAuth**:
* **JSON Web Token (JWT) Signature Algorithm:** Choose `RS256`.
* **Grant Types:** Select `Implicit`, `Authorization Code`, `Refresh Token`, and `Client Credentials`.
Click **Save Changes**.
#### 5. Enable Supported Connections
Under the **Connections** tab of the application:
* Enable the appropriate database, enterprise, or social login providers that your users will utilize.
* Ensure users are provisioned or available under the selected connection.
Click **Save Changes**.
#### 6. Post-Registration Configuration
Once saved, copy the following details from your application's settings, as you will need them for Telmai's configuration:
* **Client ID**
* **Client Secret**
* **Endpoints -> OAuth -> All the OAuth Endpoints**
***
# Okta SSO Setup
There's currently no APIs exposed for enabling SSO. To setup SSO, you will need to do the followinng
## Create Okta Group
1. Navigate to Directory Groups, and click button to Add Group
2. Specify the group name appropriately. E.g - “Telmai App Users Group”
3. Assign users who will need access to Telmai to this group
## Create Okta Application
1. Under Okta Applications, Select "Create App Integration"
2. Select the following options
* Sign-in Method OIDC: OpenID Connect
* Application Type: Web Application
3. Fill in the App Integration Details as follows:
* Application Name: Name of the Application. E.g Telmai Integration App
* Sign-in redirect URIs (provided by Telmai): Ex:
* Scenario 1 : If the okta tenant name is dev123456.okta.com, the URL would be
* Scenario 2 : If the okta tenant is mapped to a domain (ex: acmecompany.okta.com), the url would be
* Assignments: Controlled access "Limit access to selected groups"
* Select the group created previously
4. Save application
## Setup Application
Setup your Okta application and retrieve the following details
* Client ID
* Client Secret
* Open-Id configuration (https\://\/.well-known/openid-configuration)
{% hint style="info" %}
You will need to share details with Telmai team to retrieve IDP ID.
Telmai team will enable SSO accordingly
{% endhint %}
# Account Activation Instructions
**🎉 Welcome to Telmai!**
We're excited to have you on board. Your account has been successfully activated and you're ready to start your Telmai journey.
✅ **Next Steps to Get Started:**
🔐 **Accessing Telmai:**
* If you're using a **Private VPC deployment**, please contact your administrator to obtain the login URL.
* If you're using the **SaaS-based Telmai instance**, reach out to Telmai Support to receive your login URL.
🧭 **Explore Telmai Capabilities:**
* 🔗 [Connect your data sources](https://docs.telm.ai/telmai/connect-to-data)
* 📊 [Monitor your data](https://docs.telm.ai/telmai/monitoring-data)
* 🔌 [Integrate with other systems](https://docs.telm.ai/telmai/integrations)
* 🔎 [Profile your data](https://docs.telm.ai/telmai/profiling-data)
# Virtual Private Cloud (VPC) deployment
Telmai’s Virtual Private Cloud (VPC) deployment ensures that all Telmai services are hosted securely within your own cloud environment. The architecture integrates seamlessly with your existing infrastructure, enabling data-in-place monitoring while maintaining compliance with your organization’s security, governance, and privacy policies.
{% hint style="info" %}
To initiate a VPC deployment, contact your Telmai account manager. They will provide the Helm and Terraform scripts, container registry access, and guide you through the deployment process.
{% endhint %}
### Key Architectural Components
A typical Telmai VPC deployment includes the following cloud-native components, which are selected for their scalability, reliability, and portability across major cloud providers.
| Component | Purpose and Function | Cloud Service Examples |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Kubernetes (K8s) Cluster | Hosts Telmai services. Allows for scaling based on usage, simplified operations (auto-recovery), and ease of upgrades with no downtime. | Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS) |
| Elastic Search (ES) | Stores data profiling data. Enables fast data retrieval via API and UI. | Managed or hosted ES on K8s |
| Spark Engine | Handles the analysis of large datasets at scale with optimized cost and performance. Used to run Telmai data scan jobs. | Azure DataBricks, Databricks or Dataproc (GCP), Databricks or EMR (AWS) |
| Postgres | Stores results and configurations. | Azure Database, Cloud SQL (GCP), AWS RDS |
| Key Vault | Used to store encryption keys. | Azure Key Vault, Secret Manager (GCP/AWS) |
| Container Repository | Stores Telmai binaries. Used for deploying Telmai images. | Azure Container, GCP Artifact Registry, AWS ECR |
| Authentication Backend | Manages authentication, often utilizing existing services, i.e Okta. | Okta |
### Deployment Workflow
Telmai provides **Helm** and **Terraform** scripts to automate the whole deployment lifecycle.
1. **Infrastructure Setup**\
Provision cloud resources (K8s cluster, databases, vaults, etc.) within your VPC.
2. **Setup Validation**\
Validate connectivity, component availability, and Kubernetes configuration.
3. **Telmai Installation**\
Deploy Telmai Docker images, configure services, and initialize the application.
You can deploy Telmai from Telmai’s registry or from a customer-controlled container repository, depending on your organization’s security policy.
### Data Handling and Retention
Telmai handles several categories of data, each subject to specific definitions and guaranteed retention policies. **Telmai guarantees safe and complete deletion of data after the required retention period.**
#### Data Categories
| Data Type | Description | Sensitivity |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| Customer Data | Data (sensitive and non-sensitive) that Telmai monitors. This may be the original records or data decomposed into individual values for monitoring. | Varies (Sensitive or Non-sensitive) |
| Derived Data | Results (numbers) from statistical calculations performed on Customer Data. | Non-sensitive |
| Data Metrics | Derivatives used for analyzing trends in data. Examples include percentage of complete records or number of records. | Non-sensitive |
| Metadata | Meta information about data, such as data source names, attribute names, and create/update dates. | Non-sensitive |
| Sensitive User Data | Personally Identifiable Information (PII), such as usernames and passwords. | Sensitive |
| Non-Sensitive User Data | User identification and roles . | Non-sensitive |
#### Retention Policies
Telmai maintains strict, defined periods for the retention of data based on its classification:
| Data Type | Retention Policy | Notes |
| -------------------------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| Customer Data | Can be configured to be purged as soon as metrics are calculated (typically 1 hour). | If no explicit retention policy is set by the customer, this data is stored for up to 30 days and then permanently deleted . |
| Derived Data | Stored for up to 30 days. |
If no explicit retention policy is set by the customer, this data is stored for up to 30 days and then permanently deleted .
|
| Data Metrics | Stored for up to 360 days. | Used for long-term trend analysis. |
| Metadata and User Data (Sensitive & Non-Sensitive) | Stored indefinitely. | This data is retained until explicitly requested for deletion. |
# Overview
Telmai offers APIs that allow users to:
* Configure connections and assets
* Trigger scan
* Manage users
* Retrieve result data
### Authentication
Authentication can be specified through:
* User name & password
* API Key
* API token
Please refer to [Authentication API ](/telmai/api-reference/authentication-api)for more details
### Available controllers
Telmai provides the following controllers for APIs:
Controller
Controller Endpoint
Purpose
Authentication API {auth_endpoint}
https://{telmai_endpoint}/api/auth
Authentication and token creation
Configuration API {config_endpoint}
https://{telmai_endpoint}/api/backend/
Configuring connections and assets
Analytics API {analytics_endpoint}
https://{telmai_endpoint}/api/search
Metrics access
Data API {data_endpoint}
https://{telmai_endpoint}/api/data
Upload management
These controllers can be used for the corresponding Telmai instance:
`https://{telmai_endpoint}/{controller_endpoint}/{function}`
### Headers
The following headers are always required when calling Telmai APIs
| Name | Type | Description |
| ------------------------------------------------ | ------ | ------------------------------------------------------------- |
| Content-type | string | application/json |
| Authentication\* | string | Bearer \. Access token from Authentication API |
# IP Allowlist
This page describes the set of IP addresses used by Telmai. The list of IPs needs to be whitelisted at client's environment to allow access to their datalake
## env1.telm.ai
```
34.82.72.116
35.247.60.101
35.247.73.61
35.227.135.231
34.168.182.169
35.230.25.208
34.168.158.190
34.19.76.218
```
# Authentication
## Authenticate with Telmai
This API authenticates user credentials and returns an access token
`POST` `{auth_endpoint}/auth/token`
#### **Parameters**
| **Name** | **Type** | **Description** |
| --------------- | -------- | ----------------------------------------------------------------- |
| `clientId` | string | Client identifier for the application |
| `grantType` | string | Type of authentication. Use `"password"` or `"refresh_token"`. |
| `username` | string | Username for the user |
| `password` | string | Password for the user. Required if grant\_type is "password" |
| `refreshToken` | string | Refresh token string. Required if grant\_type is "refresh\_token" |
| `code` | string | Authorization code (used for `authorization_code` grant type) |
| `redirectUrl` | string | URL to redirect to after authorization |
| `transactionId` | string | Unique ID for tracking the request |
| `tenant` | string | Identifier for the tenant |
#### Headers
| Name | Type | Description |
| ---------------------------------------------- | ------ | --------------------------------- |
| Content-type\* | string | application/x-www-form-urlencoded |
{% tabs %}
{% tab title="curl" %}
`curl -X 'POST'`\
`'{auth_endpoint}/auth/token?clientId=string&grantType=password&username=string&password=string&refreshToken=string&code=string&redirectUri=string&transactionId=string&tenant=string'`\
`-H 'accept: */*'`\
`-d ''`
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="200" %}
```json
{
"expires": 3600,
"access_token": "string",
"refresh_token": "string",
"token_type": "string"
}
```
{% endtab %}
{% tab title="401" %}
```
{"description": "Unauthorized"}
```
{% endtab %}
{% tab title="403" %}
{"description": "Forbidden"}
{% endtab %}
{% tab title="404" %}
```
{"description": "Not Found"}
```
{% endtab %}
{% endtabs %}
# API Keys
An alternate way to access Telmai APIs using API Key.
All Telmai APIs can be executed either with OAuth access token or API keys. Use APIs listed below to create a key. Use this key in the header of the Telmai APIs instead of the Authorization bearer token like this:
```
X-TLM-Key:{API Key}
```
Execute Authenticate API to retrieve access token for Authorization header used below
Tenant Id can be retrieved from the URL.
## Retrieve all API Keys for specific tenant
Retrieves the list of API Keys for the given tenant.
`GET` `{auth_endpoint}/auth/{tenant}/apiKeys`
**Query Parameters**
| Name | Type | Description |
| --------------------------------------------------------------------- | ------ | ------------------------ |
| tenant\* | string | Unique Tenant identifier |
{% tabs %}
{% tab title="First Tab" %}
`curl -X 'GET' \`\
`'{auth_endpoint}/auth//apiKeys'`\
`-H 'accept: */*'`
{% endtab %}
{% endtabs %}
**Responses**
{% tabs %}
{% tab title="200" %}
```
{
"keys": [
{
"id": "string",
"name": "string"
}
]
}
```
{% endtab %}
{% tab title="401" %}
`Unauthorized`
{% endtab %}
{% tab title="403" %}
`Forbidden`
{% endtab %}
{% tab title="404" %}
`Not Found`
{% endtab %}
{% endtabs %}
## Create an API Key
Creates a new API key associated with the specified tenant.
`POST` `{auth_endpoint}/auth/{tenant}/apiKeys`
**Path Parameters**
| Name | Type | Description |
| ---------------------------------------- | ------ | --------------------------------- |
| tenant\* | string | Unique identifier for the tenant. |
#### Request Body
| Name | Type | Description |
| -------------------------------------- | ------ | ------------------- |
| name\* | String | Name of the API Key |
{% tabs %}
{% tab title="curl" %}
`curl -X 'POST' \`\
`'{auth_endpoint}/auth//apiKeys' \`\
`-H 'accept: */*' \`\
`-H 'Content-Type: application/json' \`\
`-d '{`\
`"name": ""`\
`}'`
{% endtab %}
{% endtabs %}
**Responses**
{% tabs %}
{% tab title="200" %}
`{`\
`"id": "string",`\
`"name": "string",`\
`"key": "string"`\
`}`
{% endtab %}
{% tab title="401" %}
`Unauthorized`
{% endtab %}
{% tab title="403" %}
`Forbidden`
{% endtab %}
{% tab title="404" %}
`Not Found`
{% endtab %}
{% endtabs %}
## Get details of specific API Key
Fetches the API key information using a unique key ID for a specific tenant.
`GET` `{auth_endpoint}/auth/{tenant}/apiKeys/{keyId}`
**Path Parameters**
| Name | Type | Required | Description |
| ------ | ------ | -------- | ----------------- |
| tenant | string | Yes | Tenant identifier |
| keyId | string | Yes | Unique API Key ID |
{% tabs %}
{% tab title="curl" %}
`curl -X 'GET'`\
`'/auth//apiKeys/'`\
`-H 'accept: */*'`
{% endtab %}
{% endtabs %}
**Responses**
{% tabs %}
{% tab title="200" %}
`{`\
`"id": "string",`\
`"name": "string",`\
`}`
{% endtab %}
{% tab title="401" %}
`Unauthorized`
{% endtab %}
{% tab title="403" %}
`Forbidden`
{% endtab %}
{% tab title="404" %}
`Not Found`
{% endtab %}
{% endtabs %}
## Delete an API Key
Removes the API Key with the specified `keyId` for the given `tenant`.
**Parameters:**
| Name | Type | Description |
| ---------------------------------------- | -------- | ----------------------------------------------- |
| tenant\* | `string` | The identifier of the tenant. |
| keyId\* | `string` | The unique identifier of the API Key to delete. |
{% tabs %}
{% tab title="curl" %}
`curl -X 'DELETE'`\
`'{auth_endpoint}/auth//apiKeys/'`\
`-H 'accept: */*'`
{% endtab %}
{% endtabs %}
**Responses**
{% tabs %}
{% tab title="200" %}
`{`\
`"message": "string"`\
`}`
{% endtab %}
{% tab title="401" %}
`Unauthorized`
{% endtab %}
{% tab title="403" %}
`Forbidden`
{% endtab %}
{% tab title="404" %}
`Not Found`
{% endtab %}
{% endtabs %}
# Google Service Account
Tenant Id can be retrieved from the URL.
## Get SA Account
Retrieves service account details for a specific tenant.
`GET` `/{tenant}/configuration/service_account`
#### **Request Parameters**
| **Parameter** | **Type** | **Description** |
| ------------- | -------- | ---------------- |
| `tenant` | string | Unique Tenant ID |
{% tabs %}
{% tab title="curl" %}
`curl -X 'GET'`\
`'//configuration/service_account'`\
`-H 'accept: */*'`
{% endtab %}
{% endtabs %}
{% tabs %}
{% tab title="200" %}
`{`\
`"tenant_service_account": {`\
`"name": "string",`\
`"email": "string"`\
`},`\
`"cloud_transfer_service_account": "string"`\
`}`
{% endtab %}
{% tab title="401" %}
`Unauthorized`
{% endtab %}
{% tab title="403" %}
`Forbidden`
{% endtab %}
{% tab title="404" %}
`Not Found`
{% endtab %}
{% endtabs %}
# Connections
This document outlines the API endpoints and structures for creating new connections, which can be either project-scoped or global-scoped.
## Create Connection
You can create a new connection using a `POST` request.
**Project-scoped:** `POST` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/connections`
**Global-scoped:** `POST` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/connections`
### Request Body
The request body defines the properties of the new connection.
```json
{
"name": "MyRedshift",
"type": "REDSHIFT",
"credential": {
"type": "SIMPLE",
"payload": {
"username": "test",
"password": "test"
}
},
"payload": {
"endpoint_prefix": "tlm-redshift02"
}
}
```
**Fields:**
* `name` (string): A user-defined name for the connection.
* `type` (string): The type of the connection (e.g., `REDSHIFT`, `GCS`, `AZURE`, `BIGQUERY`, `DELTALAKE`, `GENERIC_JDBC`, `HIVE_ICEBERG`, `SNOWFLAKE`). This determines the structure of the `payload` field.
* `credential` (object): Contains the authentication details for the connection.
* `type` (string): The type of credential (e.g., `SIMPLE`, `GCP`, `AWS`, `TOKEN`, `AZURE_SAS`, `GENERIC_MAP`).
* `payload` (object): The specific credential details, which vary based on the `credential.type`. Refer to the [**Credentials**](#credentials) documentation for detailed `payload` formats.
* `payload` (object): The connection-specific configuration. The format of this object depends on the `type` of the connection. Refer to the "[API Payload Formats by Type](#payload)" table for examples.
### Response
The response confirms the creation of the connection and provides its details, including a unique `id`.
```json
{
"id": 3111,
"project_id": 0,
"name": "MyRedshift",
"scope": "PROJECT", // Can be "GLOBAL" for global-scoped connections
"created_by": "john_doe@telm.ai",
"created_at": "2025-05-23T11:03:01.000Z", // ISO format in UTC
"type": "REDSHIFT",
"payload": {
"endpoint_prefix": "tlm-redshift02"
},
"db_type": "HANA101"
}
```
**Fields:**
* `id` (integer): The unique identifier of the newly created connection.
* `project_id` (integer): The ID of the project if it's a project-scoped connection; otherwise, it might be `0` or null for global-scoped.
* `name` (string): The name of the connection.
* `scope` (string): Indicates whether the connection is `PROJECT`-scoped or `GLOBAL`-scoped.
* `created_by` (string): The email of the user who created the connection.
* `created_at` (string): The timestamp when the connection was created in ISO format (UTC).
* `type` (string): The type of the connection.
* `payload` (object): The connection-specific configuration as provided in the request.
* `db_type` (string): The database type, derived from the connection.
#### Correspondence between Connection and Credential Types
The `credential.type` you provide in the request body should correspond to the `type` of the connection you are creating:
* `GCS` -> `GCP`
* `S3` -> `AWS`
* `BIGQUERY` -> `GCP`
* `AZURE` -> `AZURE_SAS`
* `SNOWFLAKE` -> `SIMPLE`
* `DELTALAKE` -> `TOKEN`
* `REDSHIFT` -> `SIMPLE`
* `GENERIC_JDBC` -> `GENERIC_MAP`
* `HIVE_ICEBERG` -> depends on `location_type` within its `payload` (e.g., `GCP`, `AWS`, `AZURE_SAS`).
### Payload
Payload object describes the connection detail and is dependent on connection type
## Update Connection
You can update an existing connection using a `PUT` request.
**Project-scoped:** `PUT` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/connections/{connectionId}`
**Connection by Id:** `PUT` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/connections/{connectionId}`
### Request Body
The request body defines the properties to update for the connection. All fields are optional, and only those provided will be updated. This request body uses the same parameters defined in create request
## List Connections
You can retrieve the list of connections using a `GET` request.
**Project-scoped:** `GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/connections`
**All Connections:** `GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/connections`
**Response**
The response is an array of connection objects, each containing detailed information about a connection.
```json
[
{
"id": 3111,
"project_id": 0,
"name": "MyRedshiftConnection",
"scope": "GLOBAL",
"created_by": "john_doe@telm.ai",
"created_at": "2025-05-23T11:03:01.000Z",
"type": "REDSHIFT",
"payload": {
"endpoint_prefix": "tlm-redshift02",
"database": "production_db",
"port": 5439
},
"db_type": "HANA101"
},
{
"id": 3112,
"project_id": 123,
"name": "MyProjectGCSConnection",
"scope": "PROJECT",
"created_by": "john.doe@telm.ai",
"created_at": "2025-05-24T10:00:00.000Z",
"type": "GCS",
"payload": {
"bucket": "telmai_project_storage"
},
"db_type": null
}
]
```
**Fields for Each Connection Object:**
* `id` (integer): The unique identifier of the connection.
* `project_id` (integer): The ID of the project the connection belongs to. For global connections, this will typically be `0`.
* `name` (string): The user-defined name of the connection.
* `scope` (string): Indicates whether the connection is `PROJECT`-scoped or `GLOBAL`-scoped.
* `created_by` (string): The email of the user who created the connection.
* `created_at` (string): The timestamp when the connection was created, in ISO 8601 format (UTC).
* `type` (string): The type of the connection (e.g., `REDSHIFT`, `GCS`, `AZURE`, `BIGQUERY`, `DELTALAKE`, `GENERIC_JDBC`, `HIVE_ICEBERG`, `SNOWFLAKE`).
* `payload` (object): The connection-specific configuration. The structure of this object varies based on the `type` of the connection. Refer to the "Connection Payload Formats by Type" section in the main Connection API documentation for detailed examples.
* `db_type` (string, optional): The specific database type associated with the connection, if applicable (e.g., "HANA101" for a `GENERIC_JDBC` connection or derived from other types). Can be `null` if not relevant.
## Get Connection Details
You can retrieve the details of an existing connection using a `GET` request.
**Project-scoped:** `GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/connections/{connectionId}`
**General reference:** `GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/connections/{connectionId}`
#### Request
This API endpoint uses a `GET` method and does not require a request body.
#### Response
The response provides the full details of the requested connection.
```json
{
"id": 3111,
"project_id": 0,
"name": "MyRedshift",
"scope": "PROJECT", // Can be "GLOBAL" in case of tenant-level endpoint
"created_by": "john_doe@telm.ai",
"created_at": "2025-05-23T11:03:01.000Z", // ISO format in UTC
"type": "REDSHIFT",
"payload": {
"endpoint_prefix": "tlm-redshift02"
},
"db_type": "HANA101"
}
```
**Fields:**
* `id` (integer): The unique identifier of the connection.
* `project_id` (integer): The ID of the project if it's a project-scoped connection; otherwise, it might be `0` or null for global-scoped.
* `name` (string): The name of the connection.
* `scope` (string): Indicates whether the connection is `PROJECT`-scoped or `GLOBAL`-scoped.
* `created_by` (string): The email of the user who created the connection.
* `created_at` (string): The timestamp when the connection was created in ISO format (UTC).
* `type` (string): The type of the connection.
* `payload` (object): The connection-specific configuration. The format of this object depends on the `type` of the connection. Refer to the "API Payload Formats by Type" table for examples.
* `db_type` (string): The database type, derived from the connection.
# Assets
This document outlines the API endpoints and structures for managing assets within projects.
## Create a New Asset
You can create a new asset using a `POST` request.
`POST` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/assets`
#### Request Body
The request body defines the properties of the new asset, using `DELTALAKE` (Databricks) as an example.
```json
{
"name": "MyDeltaLakeAsset",
"description": "a test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"id_attributes": ["order_id", "item_id"],
"separator": ",",
"sample_fraction": 10,
"skip_investigator": "false",
"delta_only": "false",
"timestamp_attribute": "event_timestamp",
"limit": 500,
"filter": {"region":["NA","EU"]},
"segments": {"category":["electronics","clothing"]},
"only_light_scan": "true",
"resolve_wildcards_to_folders": "true"
},
"payload": {
"catalog": "main_catalog",
"schema": "sales_data",
"table": "orders"
},
"connection_id": 4567
}
```
**Fields:**
* `name` (string): A user-defined name for the asset.
* `description` (string, optional): A description for the asset.
* `type` (string): The type of the asset (same as connection types, e.g., `DELTALAKE`, `REDSHIFT`, `GCP`, `BIGQUERY`, `SNOWFLAKE`, etc.). This determines the structure of the `payload` field.
* `options` (object, optional): A set of optional configuration settings for the asset.
* `id_attributes` (array of strings, optional): Attributes to be used as identifiers.
* `separator` (string, optional): Separator character for delimited files.
* `sample_fraction` (integer, optional): Fraction of data to sample (e.g., 25 for 25%).
* `skip_investigator` (string, optional): "true" to skip investigator, "false" otherwise.
* `delta_only` (string, optional): "true" to process only delta changes, "false" otherwise.
* `timestamp_attribute` (string, optional): The attribute to use for timestamp.
* `limit` (integer, optional): Maximum number of records to process.
* `filter` (object, optional): A map for filtering data, where keys are attributes and values are arrays of allowed values.
* `segments` (object, optional): A map for defining data segments, where keys are attributes and values are arrays of segment values.
* `only_light_scan` (string, optional): "true" for light scan only (supported for `SNOWFLAKE`, `DELTALAKE`, `BIGQUERY`).
* `resolve_wildcards_to_folders` (string, optional): "true" to resolve wildcards to folders.
* `payload` (object): The asset-specific configuration. The format of this object depends on the `type` of the asset. Refer to the "Asset Payload Formats by Type" section below for examples.
* `connection_id` (integer): The ID of the connection associated with this asset.
#### Response
The response confirms the creation of the asset and provides its details, including a unique `id`.
```json
{
"id": 1872,
"project_id": 0,
"name": "MyDeltaLakeAsset",
"description": "a test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"id_attributes": ["order_id", "item_id"],
"separator": ",",
"sample_fraction": 10,
"skip_investigator": "false",
"delta_only": "false",
"timestamp_attribute": "event_timestamp",
"limit": 500,
"filter": {"region":["NA","EU"]},
"segments": {"category":["electronics","clothing"]},
"only_light_scan": "true",
"resolve_wildcards_to_folders": "true"
},
"payload": {
"catalog": "main_catalog",
"schema": "sales_data",
"table": "orders"
},
"created_by": "john_doe@telm.ai",
"created_at": "2025-06-26T14:30:00.000Z",
"connection_id": 4567,
"db_type": null
}
```
**Fields:**
* `id` (integer): The unique identifier of the newly created asset.
* `project_id` (integer): The ID of the project the asset belongs to.
* `name` (string): The name of the asset.
* `description` (string): The description of the asset.
* `type` (string): The type of the asset.
* `options` (object): The configuration options for the asset.
* `payload` (object): The asset-specific configuration.
* `created_by` (string): The email of the user who created the asset.
* `created_at` (string): The timestamp when the asset was created in ISO format (UTC).
* `connection_id` (integer): The ID of the associated connection.
* `db_type` (string): The database type, derived from the asset's connection.
## Scan Schema
Once an asset is created, you can load it's schema by using a `POST` request.
`POST` `https://{telmai_endpoint}/api/data/{tenant}/assets/{assetId}/schema`
## Update Asset
You can update an existing asset using a `PUT` request.
`PUT` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/assets/{assetId}`
#### Request Body
The request body defines the properties to update for the asset, using `DELTALAKE` (Databricks) as an example. All fields are optional, and only those provided will be updated.
```json
{
"name": "MyUpdatedDeltaLakeAsset",
"description": "an updated test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"sample_fraction": 20,
"delta_only": "true"
},
"payload": {
"catalog": "analytics_catalog",
"schema": "marketing_data",
"table": "campaigns"
},
"connection_id": 4568
}
```
**Fields:**
* `name` (string, optional): A new user-defined name for the asset.
* `description` (string, optional): A new description for the asset.
* `type` (string, optional): The new type of the asset. If changed, the `payload` structure must conform to the new type.
* `options` (object, optional): New configuration settings for the asset.
* `payload` (object, optional): New asset-specific configuration. The format of this object depends on the `type` of the asset.
* `connection_id` (integer, optional): The ID of a different connection to associate with this asset.
#### Response
The response confirms the update of the asset and provides its latest details.
```json
{
"id": 1872,
"project_id": 0,
"name": "MyUpdatedDeltaLakeAsset",
"description": "an updated test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"id_attributes": ["order_id", "item_id"],
"separator": ",",
"sample_fraction": 20,
"skip_investigator": "false",
"delta_only": "true",
"timestamp_attribute": "event_timestamp",
"limit": 500,
"filter": {"region":["NA","EU"]},
"segments": {"category":["electronics","clothing"]},
"only_light_scan": "true",
"resolve_wildcards_to_folders": "true"
},
"payload": {
"catalog": "analytics_catalog",
"schema": "marketing_data",
"table": "campaigns"
},
"created_by": "john_doe@telm.ai",
"created_at": "2025-06-26T14:30:00.000Z",
"connection_id": 4568,
"db_type": null
}
```
**Fields:** (Same as Create Asset response, reflecting updated values)
## List Assets
You can retrieve a list of assets.
**Retrieve All:** Please use this endpoint to retrieve all assets you have access to
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/assets?with_attributes=true&with_schedules=true&with_lineage=true`
**Project-scoped:** Please use this endpoint to retrieve all assets you have access to in a given project
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/assets?with_attributes=true&with_schedules=true&with_lineage=true`
#### Query Parameters
* `with_attributes` (boolean, optional): If `true`, includes a list of attributes for each asset.
* `with_schedules` (boolean, optional): If `true`, includes scheduled upload information for each asset.
* `with_lineage` (boolean, optional): If `true`, includes parent and child asset IDs for lineage tracking.
#### Request
This API endpoint uses a `GET` method and does not require a request body.
#### Response
The response is an array of asset objects.
```json
[
{
"id": 1872,
"project_id": 0,
"name": "MyDeltaLakeAsset",
"description": "a test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"id_attributes": ["order_id", "item_id"],
"separator": ",",
"sample_fraction": 10,
"skip_investigator": "false",
"delta_only": "false",
"timestamp_attribute": "event_timestamp",
"limit": 500,
"filter": {"region":["NA","EU"]},
"segments": {"category":["electronics","clothing"]},
"only_light_scan": "true",
"resolve_wildcards_to_folders": "true"
},
"payload": {
"catalog": "main_catalog",
"schema": "sales_data",
"table": "orders"
},
"created_by": "alexander.chelyadin@telm.ai",
"created_at": "2025-06-26T14:30:00.000Z",
"connection_id": 4567,
"db_type": null,
"attributes": [ /* ... list of attributes ... */ ],
"scheduled_upload": { /* ... scheduled upload object ... */ },
"parents": ["T9sACQAATBg"],
"children": ["cycdCAAAUjw"]
}
]
```
**Fields:**
* All fields from the `Create Asset` response.
* `attributes` (array, optional): A list of attributes associated with the asset. Included if `with_attributes=true`.
* `scheduled_upload` (object, optional): Details about any scheduled uploads for the asset. Included if `with_schedules=true`.
* `parents` (array of strings, optional): List of IDs of parent assets for lineage. Included if `with_lineage=true`.
* `children` (array of strings, optional): List of IDs of child assets for lineage. Included if `with_lineage=true`.
## Get Asset
You can retrieve the details of a specific asset using a `GET` request.
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/assets/{assetId}`
#### Request
This API endpoint uses a `GET` method and does not require a request body.
#### Response
The response provides the full details of the requested asset.
```json
{
"id": 1872,
"project_id": 0,
"name": "MyDeltaLakeAsset",
"description": "a test asset for Delta Lake",
"type": "DELTALAKE",
"options": {
"id_attributes": ["order_id", "item_id"],
"separator": ",",
"sample_fraction": 10,
"skip_investigator": "false",
"delta_only": "false",
"timestamp_attribute": "event_timestamp",
"limit": 500,
"filter": {"region":["NA","EU"]},
"segments": {"category":["electronics","clothing"]},
"only_light_scan": "true",
"resolve_wildcards_to_folders": "true"
},
"payload": {
"catalog": "main_catalog",
"schema": "sales_data",
"table": "orders"
},
"created_by": "john_doe@telm.ai",
"created_at": "2025-06-26T14:30:00.000Z",
"connection_id": 4567,
"db_type": null,
"attributes": [ /* ... list of attributes ... */ ],
"scheduled_upload": { /* ... scheduled upload object ... */ },
"parents": ["T9sACQAATBg"],
"children": ["cycdCAAAUjw"]
}
```
**Fields:** (Same as List Assets response for a single asset)
## Delete Asset
You can delete an existing asset using a `DELETE` request.
`DELETE` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{projectId}/assets/{assetId}`
#### Request
This API endpoint uses a `DELETE` method and does not require a request body.
#### Response
The response confirms the deletion of the asset.
```json
{
"message": "Asset deleted successfully"
}
```
## Move Assets from Project to Project
You can move assets between projects using a `POST` request.
`POST` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/projects/{targetProjectId}/assets/move`
#### Request Body
The request body specifies the source project, the assets to move, and an optional target connection.
```json
{
"source_project_id": 0,
"source_asset_ids": ["T9sACQAATBg", "cycdCAAAUjw"],
"target_connection_id": 4567
}
```
**Fields:**
* `source_project_id` (integer): The ID of the project from which the assets will be moved.
* `source_asset_ids` (array of strings): A list of asset IDs to move.
* `target_connection_id` (integer, optional): The ID of a connection in the target project to associate with the moved assets. If connections used by sources are `GLOBAL`, this may not be defined, and existing global connections will be preserved.\
**Note:** Only connection of the same type can be used
#### Response
The response confirms the successful movement of assets.
```json
{
"message": "Assets moved successfully"
}
```
**Note:** The user should be a `PROJECT_EDITOR` in both the source and destination projects to perform this operation.
## Asset Payload Formats by Type
The `payload` structure for assets varies depending on the asset `type`.
* `GCP`, `S3`, `AZURE`:
```json
{
"path": "folder1/data.parquet",
"file_type": "CSV"
}
```
* **`BIGQUERY`:**
```json
{
"dataset": "test_dataset",
"result_dataset": "test_dataset", // optional
"table": "table1",
"result_table": "out_table", // optional
"query": "select * from table1" // optional
}
```
* **`SNOWFLAKE`, `REDSHIFT`:**
```json
{
"database": "db1",
"schema": "test_schema",
"table": "table1",
"query": "select * from table1" // optional
}
```
* **`DELTALAKE`:**
```json
{
"catalog": "cat1",
"schema": "test_schema",
"table": "table1",
"query": "select * from table1" // optional
}
```
* **`GENERIC_JDBC`:**
```json
{
"table": "table1",
"query": "select * from table1" // optional
}
```
* **`HIVE_ICEBERG`:**
```json
{
"database": "db1",
"table": "table1"
}
```
# Asset Groups
This document outlines the API endpoints and structures for managing source groups within projects. Source groups allow you to organize multiple assets under a common configuration.
## Create a New Source Group
You can create a new source group using a `POST` request.
`POST` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{projectId}/source_groups`
#### Request Body
The request body defines the properties of the new source group.
```json
{
"name": "MyRedshiftTables",
"description": "some test group",
"type": "SNOWFLAKE",
"connection_id": 1241,
"assets": [
{"name": "asset1", "table_or_file_name": "table1", "description": "desc1"}
],
"payload": {
"database": "SF_DB1",
"schema": "test_schema1"
},
"options": {
"only_light_scan": true
}
}
```
**Fields:**
* `name` (string): A user-defined name for the source group.
* `description` (string, optional): A description for the source group.
* `type` (string): The type of the source group (e.g., `SNOWFLAKE`, `REDSHIFT`, `BIGQUERY`, `DELTALAKE`). This determines the structure of the `payload` field.
* `connection_id` (integer): The ID of the connection associated with this source group.
* `assets` (array of objects, optional): A list of assets to include in the group.
* `name` (string): The name of the asset within the group.
* `table_or_file_name` (string): The table or file name for the asset.
* `description` (string, optional): A description for the individual asset.
* `payload` (object): The source group-specific configuration. The format of this object depends on the `type` of the source group. Refer to the "Source Group Payload Formats by Type" section below for examples.
* `options` (object, optional): Optional configuration settings for the source group.
* `only_light_scan` (boolean, optional): Set to `true` for light scan only (supported for `SNOWFLAKE`, `DELTALAKE`, `BIGQUERY`).
#### Response
The response confirms the creation of the source group and provides IDs for the group and newly created assets.
```json
{
"group_id": 1871,
"assets": [
{"id": "2fqjyu1yjwfn0", "analyze_job_id": "a3e480838f57440d3a457cdf11aab45a8"}
]
}
```
**Fields:**
* `group_id` (integer): The unique identifier of the newly created source group.
* `assets` (array of objects): A list of objects for the assets created within this group.
* `id` (string): The unique identifier of the asset.
* `analyze_job_id` (string): The ID of the schema analysis job initiated for this asset.
## Edit Source Group
You can update an existing source group using a `PUT` request.
`PUT` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{projectId}/source_groups/{sourceGroupId}`
#### Request Body
The request body defines the properties to update for the source group. All fields are optional, and only those provided will be updated.
```json
{
"name": "MyUpdatedRedshiftTables",
"description": "an updated test group",
"type": "SNOWFLAKE",
"connection_id": 1241,
"assets_to_add": [
{"name": "new_asset", "table_or_file_name": "new_table", "description": "new asset desc"}
],
"assets_to_delete": ["2fqjyu1yjwfn0"],
"payload": {
"database": "SF_DB1_Updated",
"schema": "test_schema1_Updated"
},
"options": {
"only_light_scan": false
}
}
```
**Fields:**
* `name` (string, optional): A new user-defined name for the source group.
* `description` (string, optional): A new description for the source group.
* `type` (string, optional): The new type of the source group. If changed, the `payload` structure must conform to the new type.
* `connection_id` (integer, optional): The ID of a different connection to associate with this source group.
* `assets_to_add` (array of objects, optional): A list of new assets to add to the group.
* `name` (string): The name of the asset within the group.
* `table_or_file_name` (string): The table or file name for the asset.
* `description` (string, optional): A description for the individual asset.
* `assets_to_delete` (array of strings, optional): A list of asset IDs to remove from the group.
* `payload` (object, optional): New source group-specific configuration. The format of this object depends on the `type` of the source group.
* `options` (object, optional): New optional configuration settings for the source group.
* `only_light_scan` (boolean, optional): Set to `true` for light scan only (supported for `SNOWFLAKE`, `DELTALAKE`, `BIGQUERY`).
#### Response
The response confirms the update of the source group and provides IDs for any newly added assets.
```json
{
"assets": [
{"id": "new_asset_id", "analyze_job_id": "new_analyze_job_id"}
]
}
```
**Fields:**
* `assets` (array of objects): A list of objects for assets that were newly added during this update.
* `id` (string): The unique identifier of the new asset.
* `analyze_job_id` (string): The ID of the schema analysis job initiated for this new asset.
## Move Groups to Another Project
You can move source groups between projects using a `POST` request.
`POST` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{targetProjectId}/source_groups/move`
#### Request Body
The request body specifies the source project, the source groups to move, and an optional target connection.
```json
{
"source_project_id": 0,
"source_group_ids": [12, 11, 4125],
"target_connection_id": 1241
}
```
**Fields:**
* `source_project_id` (integer): The ID of the project from which the source groups will be moved.
* `source_group_ids` (array of integers): A list of source group IDs to move.
* `target_connection_id` (integer, optional): The ID of a connection in the target project to associate with the moved source groups.
#### Response
The response confirms the successful movement of source groups.
```json
{
"message": "Source groups moved successfully"
}
```
## List Source Groups
You can retrieve a list of source groups.
**Project-scoped:** `GET` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{projectId}/source_groups`
**All:** `GET` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/source_groups`
#### Request
This API endpoint uses a `GET` method and does not require a request body.
#### Response
The response is an array of source group objects.
```json
[
{
"id": 141,
"name": "MyRedshiftTables",
"description": "some test group",
"type": "SNOWFLAKE",
"connection_id": 3111,
"db_type": null,
"project_id": 0,
"scheduled_upload": "* * * * *"
}
]
```
**Fields:**
* `id` (integer): The unique identifier of the source group.
* `name` (string): The name of the source group.
* `description` (string): The description of the source group.
* `type` (string): The type of the source group.
* `connection_id` (integer): The ID of the associated connection.
* `db_type` (string): The database type, derived from the group's connection.
* `project_id` (integer): The ID of the project the source group belongs to.
* `scheduled_upload` (string): The schedule for uploads (e.g., cron expression).
## Delete Source Group
You can delete an existing source group using a `DELETE` request.
`DELETE` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{projectId}/source_groups/{sourceGroupId}`
#### Request
This API endpoint uses a `DELETE` method and does not require a request body.
#### Response
The response confirms the deletion of the source group.
```json
{
"message": "Source group deleted successfully"
}
```
## List a Source Group's Assets
You can retrieve a list of asset names belonging to a specific source group.
`POST` `https://{telmai_endpoint}/api/backend/v2/{tenant}/configuration/projects/{projectId}/source_groups/{sourceGroupId}/list`
#### Request
This API endpoint uses a `POST` method and does not require a request body.
#### Response
The response is an array of strings, where each string is the name of an asset within the source group.
```json
["asset_name_1", "asset_name_2", "..."]
```
### Source Group Payload Formats by Type
The `payload` structure for source groups varies depending on the group `type`.
* **`SNOWFLAKE`, `REDSHIFT`:**
```json
{
"database": "SF_DB1",
"schema": "test_schema1"
}
```
* **`BIGQUERY`:**
```json
{
"dataset": "BQ_DB1"
}
```
* **`DELTALAKE`:**
```json
{
"catalog": "test_catalog",
"schema": "test_schema1"
}
```
# Monitors
The Monitors API allows you to programmatically manage data quality monitors for specific assets. You can create, update, list, and delete monitors to track metrics, predefined data quality indicators, or custom business rules.
***
## Create a New Monitor
`POST` `{config_endpoint}/{tenant}/configuration/assets/{assetId}/monitors`
Use this endpoint to create a new monitor. The structure of the `monitor` object changes based on the `type` selected.
#### Request Body Schema
The request body is a JSON object containing the high-level configuration for the monitor.
| `type` | String | Yes | The category of monitor. Options: `PREDEFINED_METRIC`, `METRIC`, or `RULE`. |
| -------------- | ------- | --- | --------------------------------------------------------------------------- |
| `name` | String | Yes | A unique, descriptive name for the monitor. |
| `enabled` | Boolean | Yes | Whether the monitor is active and running. |
| `description` | String | No | Additional context about the monitor's purpose. |
| `impact` | String | No | Severity level of an incident. Options: `LOW`, `MEDIUM`, `HIGH`. |
| `monitor` | Object | Yes | The specific logic/expression (schema varies by type). See below. |
| `threshold` | Object | Yes | Defines when an alert is triggered (ML or Static). |
| `ticketing` | Object | No | Integration settings for external tools like JIRA. |
| `notification` | Object | No | Settings for alert delivery (e.g., email, Slack). |
#### The "monitor" Object Detail
The internal structure of the `monitor` field depends on the `type` defined in the root of the request:
**Type: PREDEFINED\_METRIC**
Used for out-of-the-box Telmai metrics.
```json
"monitor": {
"predefined_metric": "COMPLETENESS",
"attributes": ["ACCOUNT_ID"]
}
```
**Type: METRIC**
Used for custom SQL-based calculations across an asset.
```json
"monitor": {
"expression": "avg(`LATEST_YEAR_ANNUAL_RETURN`) group by `COMPANY_CATEGORY`",
"options": { "ignore_nulls": true },
"raw_sql": false
}
```
**Type: RULE**
Used for row-level validations. Can be defined via a custom DSL expression or a Template ID.
```json
// Example using DSL Expression
"monitor": {
"expression": "validate a1 expect is_date_time",
"options": { "ignore_nulls": false, "empty_values": [] },
"variables": [{ "name": "a1", "type": "REFERENCE", "value": ["DUE_DATE"] }]
}
// Example using Template
"monitor": {
"template_id": 46,
"variables": [{ "name": "a1", "type": "REFERENCE", "value": ["DUE_DATE"] }]
}
```
### Threshold Object
The `threshold` object determines the logic used to trigger an incident. Telmai supports both Machine Learning (ML) based anomaly detection, relative and static thresholding.
| `threshold_type` | String |
|
| ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `threshold1` | Number | The lower bound for static or relative thresholds |
| `threshold2` | Number | The upper bound for static or relative thresholds |
| `extrapolation_strategy` | String | How to handle missing data points. Options: `IGNORE`, `ZERO`, `AVERAGE`. |
| `history_limit` | Integer | The number of historical data points the ML model or relative should consider (20 for default). |
### Notification Object
The `notification` object defines whether alerts should be pushed to external communication channels.
| `send_notifications` | Boolean | If `true`, notifications will be triggered when an incident is created. |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------ |
| `notification_channel_names` | Array | List of pre-configured channel names (e.g., `["Slack-Data-Engineering", "Email-Alerts"]`). |
### Ticketing Object
The `ticketing` object manages the automatic creation of tasks in external project management tools like JIRA.
| `type` | String | The ticketing system type. Currently supports `JIRA` only |
| -------------------- | ------- | ----------------------------------------------------------------------- |
| `create_ticket` | Boolean | If `true`, a ticket is automatically generated upon incident detection. |
| `ticket_template_id` | Integer | The ID of the specific template to use for the ticket structure. |
### Example Request Body
```json
{
"type": "METRIC", // Options: PREDEFINED_METRIC, METRIC, RULE
"enabled": true,
"name": "metricPol2",
"description": "Monitor description here",
"impact": "LOW",
"tags": null,
"monitor": {
"expression": "avg(`LATEST_YEAR_ANNUAL_RETURN`) group by `COMPANY_CATEGORY`",
"options": {
"ignore_nulls": true
},
"raw_sql": false
},
"threshold": {
"threshold_type": "ML",
"threshold1": null,
"threshold2": null,
"extrapolation_strategy": "IGNORE",
"history_limit": 0
},
"ticketing": {
"type": "JIRA",
"ticket_template_id": null,
"create_ticket": null
},
"notification": {
"send_notifications": false,
"notification_channel_names": null
}
}
```
{% hint style="warning" %}
**IMPORTANT**
Immutable Field: The monitor type cannot be changed after creation. The request body for an update is identical to the creation body but must omit the type field.
{% endhint %}
***
## Update Monitor
`PUT` `{config_endpoint}/{tenant}/configuration/assets/{assetId}/monitors/{monitorId}`
Updates an existing monitor's configuration. Payload will be same as the [Create New Monitor payload](#create-a-new-monitor) with exception to `type` as it's immutable.
***
## List Monitors
`GET` `{config_endpoint}/{tenant}/configuration/assets/{assetId}/monitors`
Returns an array of all monitors associated with the specified asset.
#### Response Object Definition
Each item in the returned array is a Monitor Object containing:
* **Identification**: `id` (unique integer), `name`, and `description`.
* **Configuration**: The full `monitor` logic, `type` (METRIC, RULE, etc.), and `enabled` status.
* **Incident Management:** `threshold`, `ticketing`, and `notification` settings.
* **Audit Metadata:** `created_by` (email) and `created_at` (ISO timestamp).
**Response Example:**
```json
[
{
"id": 3419,
"type": "RULE",
"enabled": true,
"name": "corrPol2",
"monitor": { ... },
"created_by": "user@telmai.ai",
"created_at": "2025-11-07T07:27:05.730Z"
}
]
```
***
## Get Monitor Details
`GET` `{config_endpoint}/{tenant}/configuration/assets/{assetId}/monitors/{monitorId}`
Retrieves the full configuration, metadata, and audit trail (created/modified) for a specific monitor.
### Response Object Definition
This returns a comprehensive Monitor Object which includes all fields found in the "List" response, plus additional audit fields:
* `modified_by`: The email of the last user to update the monitor.
* `modified_at`: The ISO timestamp of the last modification.
* `impact`: The severity level assigned to the monitor (e.g., LOW, MEDIUM, HIGH).
* `tags`: Any user-defined metadata tags associated with the monitor.
***
## Delete Monitor
`DELETE` `{config_endpoint}/{tenant}/configuration/assets/{assetId}/monitors/{monitorId}`
Permanently removes a monitor from the asset configuration.
**Response:**
```json
{
"message": "Monitor successfully deleted"
}
```
***
## Error Codes
The API uses standard HTTP status codes to indicate the success or failure of requests.
| **Status Code** | **Description** | **Possible Causes** |
| ------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `400 Bad Request` | The request was invalid or cannot be served. | Missing required fields, invalid SQL/DSL expression, or attempting to change the `type` in a PUT request. |
| `401 Unauthorized` | Authentication failed. | Invalid or expired API token. |
| `403 Forbidden` | Permissions error. | The user does not have access to the specified tenant or asset. |
| `404 Not Found` | Resource not found. | The `assetId` or `monitorId` does not exist. |
Also see: [Replay Monitor Scan API](/telmai/api-reference/upload-data-api/replay-monitor-scan)
# Incidents
This documentation describes the API endpoints for retrieving and managing incident data.
## List Incidents
This endpoint retrieves a list of incidents, which are aggregations of multiple alerts related to an asset.
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/incidents`
**Query Parameters (Filters)**
The request supports the following query parameters for filtering the incident list:
| Parameter | Type | Description | Example |
| ------------------ | ---------------------------------- | ------------------------------------------------------------------------ | -------------------------- |
| `project_ids` | List of Integers (comma-separated) | Filters by a list of Project IDs. | `0,24` |
| `connection_types` | List of Strings (comma-separated) | Filters by asset connection types (e.g., GCS, BIGQUERY). | `GCS,BIGQUERY` |
| `alert_policy_ids` | List of Strings (comma-separated) | Filters by Alert Policy IDs. (IDs can be strings for prebuilt policies). | `4,5` |
| `severities` | List of Strings (comma-separated) | Filters by severity (LOW, MEDIUM, HIGH). | `HIGH,MEDIUM` |
| `impacts` | List of Strings (comma-separated) | Filters by impact (LOW, MEDIUM, HIGH). | `HIGH` |
| `connection_ids` | List of Strings (comma-separated) | Filters by Connection IDs. | `conn-123,conn-456` |
| `from_time` | String (ISO 8601 Timestamp) | Range start timestamp for filtering. | `2025-07-15T12:14:54.123Z` |
| `to_time` | String (ISO 8601 Timestamp) | Range end timestamp for filtering. | `2025-07-15T12:14:54.123Z` |
| `text_to_search` | String | A search string applied to policy names and source names. | `test_policy` |
Example Request:
```
GET: https://{host:port}/api/backend/{tenant}/incidents?project_ids=0,24
```
**Response Body (JSON Array)**
Returns an array of incident objects.
```json
[
{
"id": 1311,
"project_id": 0,
"asset_id": "5x0e34yk1re4",
"last_upload_timestamp": "2025-06-30T14:12:51.871Z",
"created_time": "2025-06-30T14:12:51.871Z",
"severity": "HIGH",
"impact": "HIGH",
"status": "OPEN", // Can be one of OPEN / CLOSED
"ticket_id": 2411,
"alert_policy_id": "4",
"alert_policy_name": "test_policy",
"description": "...",
"tags": ["T1", "PROD"]
},
...
]
```
***
## Get Incident Details
Retrieves the detailed information for a specific incident.
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/incidents/{incidentId}`
**Path Parameter**
| Parameter | Type | Description |
| ------------ | ------- | ------------------------------ |
| `incidentId` | Integer | The unique ID of the incident. |
Example Request:
```
GET: https://{telmai_endpoint}/api/backend/{tenant}/incidents/1311
```
**Response Body (JSON Object)**
In addition to the fields in the list response, this endpoint returns detailed metrics and a history of state changes.
```json
{
"id": 1311,
"project_id": 0,
"asset_id": "5x0e34yk1re4",
"last_upload_timestamp": "2025-06-30T14:12:51.871Z",
"severity": "HIGH",
"impact": "HIGH",
"status": "OPEN", // Can be one of OPEN / CLOSED
"ticket_id": 2411,
"alert_policy_id": "4",
"alert_policy_name": "test_policy",
"description": "...",
"metrics": [
{
"metric_name": "contains_pii",
"metric_aggregation": "ratio",
"display_metric_name": "PII %",
"segment": null,
"rule": null, // If populated, it indicates a virtual attribute for a correctness policy
"attribute": null
},
...
],
"tags": ["T1", "PROD"],
"history": [
{
"timestamp": "2025-06-30T14:12:51.871Z",
"state": "NEW", // Can be one of NEW / ONGOING / RESOLVED / REOPENED
"alert_refs": [
{"metric_time":"2025-06-30T14:12:51.871Z", "violation_name":"FEATURE.s132j7ja1"},
...
]
}
]
}
```
***
## Incidents Distribution
This endpoint retrieves a day-by-day distribution of incident counts, categorized by severity and impact.
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/incidents/distribution`
Filters: This endpoint supports the same filter set as the Incidents List endpoint.
**Response Body (JSON Object)**
```json
{
"incidents": [
{
"date": "2025-07-22",
"counts": {
"LOW": 1,
"MEDIUM": 4,
"HIGH": 6,
"": 8 // Incidents without a specified severity/impact
},
"counts_by_impact": {
"LOW": 1,
"MEDIUM": 4,
"HIGH": 6
}
},
...
]
}
```
***
## Incidents Summary
Retrieves overall summary statistics for incidents.
`GET` `https://{host:port}/api/backend/{tenant}/incidents/summary`
**Response Body (JSON Object)**
| Field | Type | Description |
| ------------------------------- | ------- | ----------------------------------------------------- |
| `total` | Integer | Total number of incidents. |
| `open` | Integer | Total number of OPEN incidents. |
| `closed` | Integer | Total number of CLOSED incidents. |
| `mean_time_to_resolve` | Integer | Average time to resolve incidents (in hours). |
| `previous_mean_time_to_resolve` | Integer | Previous period's average time to resolve (in hours). |
| `open_by_tag` | Object | Count of OPEN incidents grouped by tag. |
| `closed_by_tag` | Object | Count of CLOSED incidents grouped by tag. |
```json
{
"total": 25,
"open": 6,
"closed": 19,
"mean_time_to_resolve": 122,
"previous_mean_time_to_resolve": 117,
"open_by_tag": {
"T1": 3,
"prod": 12
},
"closed_by_tag": {
"T1": 12,
"prod": 67
}
}
```
***
## Retrieving Alerts for an Incident
An Incident is an aggregation over time and refers to multiple uploads and alert objects, whereas an Alert refers to a specific upload and violation. To retrieve the individual alerts that compose an incident, you must use the `alert_refs` from the incident's history.
## POST Retrieve Alerts by Filters
`POST` `https://{host:port}/api/backend/{tenant}/configuration/alerts/by_filters`
**Request Body (JSON Object)**
The request body is constructed using data extracted from the incident's `asset_id` and the `alert_refs` array (found within the `history` of the detailed incident response).
Example:
If an incident has an `asset_id` of `"5x0e34yk1re4"` and the following `alert_refs`:
```json
"alert_refs": [
{"metric_time":"2025-06-30T14:12:51.871Z", "violation_name":"FEATURE.s132j7ja1"},
{"metric_time":"2025-06-30T14:13:55.325Z", "violation_name":"DATA.a531"}
]
```
The request body should be:
```json
{
"sources": ["5x0e34yk1re4"], // The asset_id from the incident
"upload_dates": ["2025-06-30T14:12:51.871Z", "2025-06-30T14:13:55.325Z"], // metric_time from alert_refs
"violation_names": ["FEATURE.s132j7ja1", "DATA.a531"] // violation_name from alert_refs
}
```
**Alert Object Structure**
The endpoint returns an array of detailed Alert objects.
```json
{
"type": "SCHEMA_CHANGE",
"source": "SNOWFLAKE.SALES_DB.PUBLIC.CUSTOMER_ORDERS",
"attribute": "SHIPPING_ADDRESS",
"job_id": "job-f8e2-4a1b-9c3d-5e6f7a8b9c0d",
"create_time": "2023-10-27T10:30:00.123Z",
"save_time": "2023-10-27T10:30:05.456Z",
"violation_data": {},
"violation_name": "Customer Orders Schema Drift",
"notification_channels": ["critical-alerts-slack", "data-team-email"],
"is_notification_enabled": true,
"metric_time": "2023-10-27T10:25:00.000Z",
"is_notified": false,
"metric_value": 1.0,
"policy_name": "Monitor Production Schemas",
"segment": "USA",
"priority": "HIGH",
"description": {
"HIGH": [
"The data type for column 'SHIPPING_ADDRESS' was changed from VARCHAR(255) to VARCHAR(512)."
]
},
"source_name": "CUSTOMER_ORDERS",
"source_type": "SNOWFLAKE",
"project_name": "E-commerce Analytics",
"tags": ["production", "schema", "customer-data"]
}
```
# Trigger Scan
{% hint style="warning" %}
Below APIs have been deprecated
`/api/data/{tenant}/sources/{source}/batch_data`
`/api/data/v2/{tenant}/sources/{source}/batch_data`
`/api/data/{tenant}/batch_data`
`/api/data/v2/{tenant}/batch_data`
{% endhint %}
## Trigger data scan
`POST` `https://{telmai_endpoint}/api/data/{tenant}/assets/{assetId}/batch_data`
Location specific request body parameters are defined below.
#### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | ----------------- |
| `tenant`\* | string | Tenant identifier |
| `assetId`\* | string | Asset identifier |
{% tabs %}
{% tab title="200 Data upload started" %}
```
{
"message": string,
"job_id": string
}
```
{% endtab %}
{% endtabs %}
# Replay Monitor Scan
This endpoint allows you to programmatically initiate a replay for a specific SQL monitor.
`POST` `https://{telmai_endpoint}/api/data/{tenant}/assets/{asset_id}/batch_data/replay`
#### Request Body
The request supports the following parameters in the JSON body:
| Parameter | Type | Description | Example |
| ------------ | --------- | --------------------------------------------------- | ------- |
| `monitor_id` | `Integer` | Unique ID of the SQL monitor to replay. | `5225` |
| `depth` | `Integer` | Number of historical scans to replay (Range: 1–20). | `5` |
**Example Request:**
```json
{
"monitor_id": 5225,
"depth": 5
}
```
# Track Scan Job
APIs to track status of upload
When upload job is triggered via API, the response contains a section with the job id:
```
{
"message": "Upload task started. Please track the job status to check progress",
"job_id": "a08aa2a2592a541e194db9d01e9d0e457"
}
```
Note, Tenant Id can be retrieved from the URL.
## Get jobs for a specific tenant and source
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/sources/{source}/jobs`
#### Path Parameters
| Name | Type | Description |
| ---------------------------------------- | ------ | ----------- |
| source\* | string | Source Id |
| tenant\* | string | Tenant Id |
#### Query Parameters
| Name | Type | Description |
| ------------ | ------- | ----------- |
| most\_recent | Boolean | true/false |
{% tabs %}
{% tab title="200 " %}
```
[
{
"details": "string",
"id": "string",
"run_time": {
"nano": 0,
"negative": true,
"seconds": 0,
"units": [
{
"dateBased": true,
"durationEstimated": true,
"timeBased": true
}
],
"zero": true
},
"start_time": "2022-05-24T23:50:19.809Z",
"status": "string"
}
]
```
{% endtab %}
{% endtabs %}
# Check for Alerts
APIs to check alert status of a given job
## Get Alerts for a Specific Job
`POST` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/sources/{sourceID}/alerts?job_id={job_id}`
Returns the alerts generated for the specified scan job. Use the `job_id` returned by the [Trigger Scan](/telmai/api-reference/upload-data-api) endpoint.
#### Path Parameters
| Name | Type | Description |
| -------------------------------------------- | ------ | ----------------- |
| `tenant`\* | string | Tenant identifier |
| `sourceID`\* | string | Source (asset) ID |
#### Query Parameters
| Name | Type | Description |
| ------------------------------------------ | ------ | ------------------------------- |
| `job_id`\* | string | Job ID returned by Trigger Scan |
{% tabs %}
{% tab title="200" %}
```json
[
{
"type": "",
"source": "",
"description": "",
"job_id": "",
"create_time": "",
"save_time": "",
"is_notification_enabled": true,
"metric_time": "",
"metric_value": "",
"policy_name": "",
"priority": "",
"source_name": "",
"source_type": ""
}
]
```
{% endtab %}
{% endtabs %}
# Spark Job Config
The Spark Job Config API allows you to programmatically manage compute cluster configuration for Telmai's Spark-based data quality jobs. Supports Dataproc (GCP), EMR (AWS), and Databricks configurations.
***
## Get Spark Job Config
`GET` `/configuration/tenants/{tenant}/spark-job-config`
Returns the current Spark job configuration for the specified tenant. Returns an empty object `{}` if no configuration has been saved yet.
***
## Save or Update Spark Job Config
`PUT` `/configuration/tenants/{tenant}/spark-job-config`
Creates or updates the Spark job configuration for a tenant. Each top-level block (`spark_properties`, `dataproc_config`, `emr_config`, `databricks_config`) is stored and replaced independently.
{% hint style="info" %}
**Partial Updates** Only the blocks you include in the request body are updated. Omitting a block leaves its existing values unchanged. To explicitly clear a block, send it as `null` (e.g., `"dataproc_config": null`)
{% endhint %}
**Request Body Schema**
| Field | Type | Required | Description |
| ------------------- | ------ | -------- | --------------------------------------------------------------------------------- |
| `spark_properties` | Object | No | Key-value pairs of Spark configuration properties (e.g., executor memory, cores). |
| `dataproc_config` | Object | No | GCP Dataproc cluster configuration. See schema below. |
| `emr_config` | Object | No | AWS EMR cluster configuration. See schema below. |
| `databricks_config` | Object | No | Databricks cluster configuration. See schema below. |
**`dataproc_config` Schema**
| Field | Type | Required | Description |
| ------------------------------ | ------- | -------- | ------------------------------------------------------------------ |
| `master_node_type` | String | No | Machine type for the Dataproc master node (e.g., `n1-standard-4`). |
| `worker_node_type` | String | No | Machine type for Dataproc worker nodes (e.g., `n1-standard-8`). |
| `spark_master_disk_size_in_gb` | Integer | No | Boot disk size in GB for the master node. |
| `spark_worker_disk_size_in_gb` | Integer | No | Boot disk size in GB for worker nodes. |
| `autoscale_policy` | String | No | Full resource path to a GCP autoscaling policy. |
**`emr_config` Schema**
| Field | Type | Required | Description |
| ------------------------ | ------- | -------- | --------------------------------------------------------------------- |
| `master_instance_type` | String | No | EC2 instance type for the EMR master node (e.g., `m5.xlarge`). |
| `worker_instance_type` | String | No | EC2 instance type for EMR worker nodes (e.g., `m5.2xlarge`). |
| `worker_instance_option` | String | No | Purchasing option for worker instances. Options: `ON_DEMAND`, `SPOT`. |
| `min_instance_count` | Integer | No | Minimum number of worker instances for autoscaling. |
| `max_instance_count` | Integer | No | Maximum number of worker instances for autoscaling. |
**`databricks_config` Schema**
| Field | Type | Required | Description |
| ------------------------- | ------- | -------- | ------------------------------------------------------- |
| `spark_version` | String | No | Databricks Runtime version (e.g., `13.3.x-scala2.12`). |
| `node_type` | String | No | VM type for worker nodes (e.g., `Standard_DS3_v2`). |
| `instance_pool_id` | String | No | ID of the Databricks instance pool for worker nodes. |
| `driver_instance_pool_id` | String | No | ID of the Databricks instance pool for the driver node. |
| `policy_id` | String | No | ID of the Databricks cluster policy to apply. |
| `min_instance_count` | Integer | No | Minimum number of worker nodes for autoscaling. |
| `max_instance_count` | Integer | No | Maximum number of worker nodes for autoscaling. |
{% hint style="warning" %}
The spark properties specified should be supported by the configured `node_type`. Example: you can't have 10G driver memory for a 6GB node type.
{% endhint %}
***
#### Example: Save Dataproc Config
```json
{
"spark_properties": {
"spark.executor.memory": "4g",
"spark.executor.cores": "2"
},
"dataproc_config": {
"master_node_type": "n1-standard-4",
"worker_node_type": "n1-standard-8",
"spark_master_disk_size_in_gb": 100,
"spark_worker_disk_size_in_gb": 500,
"autoscale_policy": "projects/my-project/regions/us-central1/autoscalingPolicies/my-policy"
}
}
```
#### Example: Save EMR Config
```json
{
"spark_properties": {
"spark.executor.memory": "8g"
},
"emr_config": {
"master_instance_type": "m5.xlarge",
"worker_instance_type": "m5.2xlarge",
"worker_instance_option": "SPOT",
"min_instance_count": 2,
"max_instance_count": 10
}
}
```
#### Example: Save Databricks Config
```json
{
"databricks_config": {
"spark_version": "13.3.x-scala2.12",
"node_type": "Standard_DS3_v2",
"instance_pool_id": "0123-456789-pool-abc",
"driver_instance_pool_id": "0123-456789-pool-def",
"policy_id": "ABCD1234",
"min_instance_count": 1,
"max_instance_count": 5
}
}
```
#### Example: Partial Update
Update only a single field without affecting other config values:
```json
{
"dataproc_config": {
"worker_node_type": "n1-highmem-8"
}
}
```
***
## Delete Spark Job Config
`DELETE` `/configuration/tenants/{tenant}/spark-job-config`
Permanently removes the Spark job configuration for the specified tenant.
{% hint style="warning" %}
**IMPORTANT** This action deletes the entire configuration for the tenant. Subsequent GET requests will return `{}` until a new configuration is saved.
{% endhint %}
***
### Error Codes
| Status Code | Description | Possible Causes |
| ------------------ | -------------------------------------------- | -------------------------------------------------------------------------- |
| `400 Bad Request` | The request was invalid or cannot be served. | Malformed JSON body or invalid field values (e.g., non-integer disk size). |
| `401 Unauthorized` | Authentication failed. | Invalid or expired API token. |
| `403 Forbidden` | Permissions error. | The caller does not have admin access to the specified tenant. |
| `404 Not Found` | Resource not found. | The specified `tenant` or `tenantId` does not exist. |
# Ticket Integration
This section provides a guide to the API endpoints for managing and integrating ticket templates. These templates allow you to define the structure and content of tickets created in external systems, such as Jira.
***
## Template Management
## Create a Template
Use this endpoint to create a new ticket template.
`POST` `https://{apiUrl}/{tenant}/configuration/integrations/tickets/template`
```json
{
"name": "template-1",
"description": "Template for new feature stories",
"type": "JIRA",
"props": {
"jira_project_id": "10000",
"jira_issue_type_id": "10001",
"custom_fields": {
"summary": "template desc"
}
}
}
```
Fields:
* `name` (string): A unique name for the template.
* `description` (string, optional): A brief description of the template's purpose.
* `type` (string): Ticket type (currently only `JIRA` is supported)
* `props` (object): Corresponding properties for ticket based on type
#### Jira Prop Fields
* `jira_project_id` (string): The ID of the Jira project to associate with this template.
* `jira_issue_type_id` (string): The ID of the Jira issue type (e.g., "Story," "Bug") for this template.
* `custom_fields` (object): A key-value pair of custom fields to be included in the ticket.
## Update a Template
Use this endpoint to update an existing template. The template ID is included in the URL path.
`PUT` `https://{apiUrl}/{tenant}/configuration/integrations/tickets/template/1`
```json
{
"name": "template-1",
"description": "Updated description",
"type": "JIRA",
"props": {
"jira_project_id": "10000",
"jira_issue_type_id": "10001",
"custom_fields": {
"summary": "updated template desc"
}
}
}
```
Fields:
* `name` (string): The new name for the template.
* `description` (string, optional): The new description for the template.
* `jira_project_id` (string): The new Jira project ID.
* `jira_issue_type_id` (string): The new Jira issue type ID.
* `custom_fields` (object): The updated custom fields.
## Get a Specific Template
Retrieve the details of a single template using its ID.
`GET` `https://{apiUrl}/{tenant}/configuration/integrations/tickets/template/1`
```json
{
"id": 1,
"name": "template-1",
"description": "Updated description",
"type": "JIRA",
"props": {
"jira_project_id": "10000",
"jira_issue_type_id": "10001",
"custom_fields": {
"summary": "updated template desc"
}
}
}
```
## Get All Templates
Retrieve a list of all available ticket templates.
`GET` `https://{apiUrl}/{tenant}/configuration/integrations/tickets/template`
```json
[
{
"id": 2,
"name": "template-2",
"description": null,
"type": "JIRA",
"props": {
"jira_project_id": "10000",
"jira_issue_type_id": "10001",
"custom_fields": {
"customfield_10014": "TEL-4825"
}
}
},
{
"id": 1,
"name": "template-1",
"description": "Updated description",
"type": "JIRA",
"props": {
"jira_project_id": "10000",
"jira_issue_type_id": "10001",
"custom_fields": {
"summary": "updated template desc"
}
}
}
]
```
***
# Alation Integration
To integrate Alation with Telmai, you'll use a set of REST API endpoints to manage the connection, automatically or manually create data mappings, and perform other configuration tasks. These API calls enable Telmai to interact with your Alation data catalog for seamless data governance.
## Setup
These endpoints allow you to configure, retrieve, and remove the core Alation integration with Telmai. All requests require your `backend_service_host` and `tenant` identifier.
## Add or Update Integration
Use this endpoint to establish the connection between Telmai and Alation or to update an existing one. You'll need your Alation URL, a user ID, and a refresh API token.
`PUT` `{backend_service_host}/{tenant}/configuration/integrations/alation`
#### Request Body
```json
{
"url": "https://.alationcloud.com/",
"user_id": 81,
"refresh_api_token": "xxxxxx..."
}
```
#### Response
```json
{
"url": "https://.alationcloud.com/",
"user_id": 81
}
```
***
## Get Integration Properties
Retrieve the current configuration details for your Alation integration. This call doesn't require a request body.
`GET` `{backend_service_host}/{tenant}/configuration/integrations/alation`
#### Response
```json
{
"url": "https://.alationcloud.com/",
"user_id": 81
}
```
***
## Remove Integration
Permanently delete the Alation integration from your Telmai environment.
`DELETE` `{backend_service_host}/{tenant}/configuration/integrations/alation`
#### Response
```json
{
"message": "Alation integration configuration deleted successfully"
}
```
***
## Mappings
These endpoints handle the creation and management of mappings, which link Telmai data assets to specific tables in your Alation data catalog.
## Start Automatic Mappings
This endpoint initiates a background job to automatically create mappings for your assets. You can optionally choose to overwrite existing mappings.
`POST` `{backend_service_host}/{tenant}/configuration/mappings/alation/auto`
Request Parameters:
* `should_overwrite`: `true` or `false` (default is `false`).
#### Response
```json
{
"message": "Creating Alation mappings started",
"job_id": "f583cea0-640d-4c50-9153-e187f92a982e"
}
```
The response provides a `job_id` that you can use to track the status of the mapping job.
***
## Get Mapping Job Result
Check the status and results of an ongoing or completed automatic mapping job using its `job_id`.
`GET` `{backend_service_host}/{tenant}/configuration/mappings/alation/auto`
Request Parameters:
* `job_id`: The `job_id` from the automatic mapping request.
#### Response
```json
{
"status": "COMPLETED", // Can also be IN_PROGRESS or FAILED
"error_message": null,
"result": [
{
"asset_id": "35xftecujtsjd",
"data_source_id": 251,
"data_source_name": "telmai_datasource",
"schema_id": 46,
"schema_name": "adventureworks.SalesLT",
"table_id": 6121,
"table_name": "CustomerAddress"
},
...
]
}
```
***
## List All Mappings
Retrieve a list of all existing Alation mappings configured in Telmai.
`GET` `{backend_service_host}/{tenant}/configuration/mappings/alation`
#### Response
```json
[
{
"asset_id": "35xftecujtsjd",
"data_source_id": 251,
"data_source_name": "telmai_datasource",
"schema_id": 46,
"schema_name": "adventureworks.SalesLT",
"table_id": 6121,
"table_name": "CustomerAddress"
},
...
]
```
***
## Delete a Mapping
Remove a specific mapping by providing its Telmai asset ID.
`DELETE` `{backend_service_host}/{tenant}/configuration/mappings/alation`
Request Parameters:
* `asset_id`: The ID of the Telmai asset to unmap.
#### Response
```json
{
"message": "Alation mapping deleted successfully"
}
```
***
## Create a New Mapping Manually
Manually create a new mapping between a Telmai asset and an Alation table. You must provide the Telmai asset ID and the Alation table ID.
`POST` `{backend_service_host}/{tenant}/configuration/mappings/alation`
#### Request Body
```json
{
"asset_id": "35xftecujtsjd",
"alation_table_id": 6121
}
```
#### Response
```json
{
"asset_id": "35xftecujtsjd",
"data_source_id": 251,
"data_source_name": "telmai_datasource",
"schema_id": 46,
"schema_name": "adventureworks.SalesLT",
"table_id": 6121,
"table_name": "CustomerAddress"
}
```
# DQ Reporting
This section details the API endpoints used to configure and enable the Centralized DQ Monitor Scan Reporting feature. This configuration dictates where the reports are written.
## Update Configuration
Use the `PUT` endpoint to enable the reporting feature and specify the external destination table location.
`PUT` `https://{config_endpoint}/{tenant_id}/configuration/dq_reporting`
**Request Body**
The request body specifies the connection and path for the external destination table.
```json
{
"enabled": true,
"path": "path/to/output",
"connection_id": 60529
}
```
| **Field** | **Type** | **Description** | **Notes** |
| --------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `Boolean` | Set to `true` to activate the centralized reporting feature. | If `false`, no reports will be generated. |
| `path` | `String` |
The relative path within the storage bucket/container where the report data will be written. The path MUST exist in storage
| Example: `"dq_test"` will result in reports being written to `s3a:///dq_test` or `gs:///dq_test`. |
| `connection_id` | `Integer` | The ID of the globally configured storage connection to use as the destination. | This must be a Global Connection configured for cloud storage (Azure Storage, GCS, or S3). Azure Storage is a critical testing focus. |
**Prerequisites**
* The specified `connection_id` must point to an existing Azure Storage, GCS, or S3 connection configured within the system.
* The system must have write permissions to the specified storage location (`connection_id` + `path`).
***
## Get Configuration
`GET` `https://{config_endpoint}/{tenant_id}/configuration/dq_reporting`
Returns the current DQ reporting configuration for a tenant.
***
## Delete Configuration
`DELETE` `https://{config_endpoint}/{tenant_id}/configuration/dq_reporting`
Removes the DQ reporting configuration from the tenant.
***
# DQ Score
## Get DQ Score Options
`GET` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/assets/{assetId}/dq_score`
This endpoint retrieves the current Data Quality Score configuration for a specific asset.
### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | ----------------- |
| `tenant`\* | string | Tenant identifier |
| `assetId`\* | string | Asset identifier |
### Response
**200** DQ Score configuration retrieved successfully
```json
{
"dq_score_config": {
"enabled": true,
"weights": {
"completeness": 0.25,
"accuracy": 0.25,
"consistency": 0.25,
"timeliness": 0.25
}
}
}
```
## Update DQ Score Options
`PUT` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/assets/{assetId}/dq_score`
This endpoint updates the Data Quality Score configuration for a specific asset.
### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | ----------------- |
| `tenant`\* | string | Tenant identifier |
| `assetId`\* | string | Asset identifier |
### Request Body
```json
{
"enabled": true,
"weights": {
"completeness": 0.3,
"accuracy": 0.3,
"consistency": 0.2,
"timeliness": 0.2
}
}
```
### Response
**200** DQ Score configuration updated successfully
```json
{
"dq_score_config": {
"enabled": true,
"weights": {
"completeness": 0.3,
"accuracy": 0.3,
"consistency": 0.2,
"timeliness": 0.2
}
}
}
```
## Delete DQ Score Options
`DELETE` `https://{telmai_endpoint}/api/backend/{tenant}/configuration/assets/{assetId}/dq_score`
This endpoint deletes the DQ Score configuration for a specific asset, reverting to default settings.
### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | ----------------- |
| `tenant`\* | string | Tenant identifier |
| `assetId`\* | string | Asset identifier |
### Response
**200** DQ Score configuration deleted successfully
```json
{
"success": true,
"message": "DQ Score configuration deleted successfully"
}
```
# Source APIs \[Deprecated]
APIs to manage a Telmai source
{% hint style="danger" %}
These APIs have been deprecated
{% endhint %}
Tenant Id can be retrieved from the UI as listed [here](broken://pages/-MkOLrNp2rbjIcVtIX6f).
## Create a source
`POST` `https://env1.telm.ai/api/backend/{tenant}/configuration/sources`
Create a new source in the tenant
#### Path Parameters
| Name | Type | Description |
| ---------------------------------------- | ------ | ----------- |
| tenant\* | string | Tenant Id |
#### Query Parameters
| Name | Type | Description |
| -------------------------------------- | ------ | --------------------- |
| name\* | string | Name of source |
| description | string | Description of source |
#### Headers
| Name | Type | Description |
| ------------------------------------------------ | ------ | ----------------------------------------------------------------------- |
| Content-type | string | application/json |
| Authentication\* | string | Bearer \. Access token retrieved from Authentication API |
{% tabs %}
{% tab title="200 Returns a json representation of the Source Object" %}
```
{
id = 'id',
name = 'name',
description = 'description',
attributes = [],
"last_upload_job_id": null
}
```
{% endtab %}
{% endtabs %}
## List all sources for this tenant
`GET` `https://env1.telm.ai/api/backend/{tenant}/configuration/source`
Api to list all sources in a tenant
#### Path Parameters
| Name | Type | Description |
| ---------------------------------------- | ------ | ----------- |
| tenant\* | string | Tenant Id |
#### Headers
| Name | Type | Description |
| ------------------------------------------------ | ------ | -------------------------------------------------------------------------- |
| Content-type | string | application/json |
| Authentication\* | string | Bearer \. Access token is retrieved from Authentication API |
{% tabs %}
{% tab title="200 A list of source objects" %}
```
[
{
id = 'id',
name = 'name',
description = 'description',
attributes = [
{
"id": "",
"name": "",
"monitored": ,
"status": "",
"model_last_updated": ""
},
{
"id": "",
"name": "",
"monitored": ,
"status": "",
"model_last_updated": ""
}
],
"last_upload_job_id": "",
"scheduled_upload":
}
]
```
{% endtab %}
{% endtabs %}
## Update a source
`PUT` `https://env1.telm.ai/api/backend/{tenant}/configuration/sources`
Create a new source in the tenant
#### Path Parameters
| Name | Type | Description |
| ---------------------------------------- | ------ | ----------- |
| tenant\* | string | Tenant Id |
#### Query Parameters
| Name | Type | Description |
| -------------------------------------- | ------ | --------------------- |
| name\* | string | Name of source |
| description | string | Description of source |
#### Headers
| Name | Type | Description |
| ------------------------------------------------ | ------ | ----------------------------------------------------------------------- |
| Content-type | string | application/json |
| Authentication\* | string | Bearer \. Access token retrieved from Authentication API |
{% tabs %}
{% tab title="200 Returns a json representation of the Source Object" %}
```
{
id = 'id',
name = 'name',
description = 'description',
attributes = [],
"last_upload_job_id": null
}
```
{% endtab %}
{% endtabs %}
## Get details of a specific source
`GET` `https://env1.telm.ai/api/backend/{tenant}/configuration/sources/`
#### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | ----------------------------- |
| source-id\* | string | Id of source to be retrieved. |
| tenant\* | string | Tenant id |
#### Headers
| Name | Type | Description |
| ----------------------------------------------- | ------ | ----------------------------------------------- |
| Content-type | string | application/json |
| Authorization\* | string | Bearer \ |
{% tabs %}
{% tab title="200 Source details" %}
```
{
"id": "",
"name": "",
"description": "",
"attributes": [],
"last_upload_job_id": null,
"scheduled_upload": null
}
```
{% endtab %}
{% endtabs %}
## Delete Source
`DELETE` `https://env1.telm.ai/api/backend/{tenant}/configuration/sources/`
Delete a specific source
#### Path Parameters
| Name | Type | Description |
| ------------------------------------------- | ------ | -------------------------------------------------------------------------------- |
| source-id\* | string | Id of source to be deleted. This can be retrieved by looking up a source by name |
| tenant\* | string | Name of tenant |
#### Headers
| Name | Type | Description |
| ----------------------------------------------- | ------ | ----------------------------------------------- |
| Content-type | string | application/json |
| Authorization\* | string | Bearer \ |
{% tabs %}
{% tab title="200 Source deleted successfully" %}
```
{
"message": "Source was deleted successfully"
}
```
{% endtab %}
{% endtabs %}
# Copy of Trigger Scan
## Trigger data scan
`POST` `https://{telmai_endpoint}/api/data/v2/{tenant}/sources/{source_id}/batch_data`
Location specific request body parameters are defined below.
#### Path Parameters
| Name | Type | Description |
| -------------------------------------------- | ------ | ------------------------------ |
| source\_id\* | string | Id of source to upload data to |
| tenant\* | string | Name of tenant |
#### Headers
| Name | Type | Description |
| ------------------------------------------------ | ------ | ------------------------------------------------------------- |
| Content-type | string | application/json |
| Authentication\* | string | Bearer \. Access token from Authentication API |
{% tabs %}
{% tab title="200 Data upload started" %}
```
{
"message": string,
"job_id": string
}
```
{% endtab %}
{% endtabs %}
# Amazon S3 Request data
### Request Body for Amazon S3
```
{
"type": string,
"id_attribute": string,
"payload": {
"bucket": string,
"path": string ,
"security": {
"aws_key": string,
"aws_secret": string,
"region": string
},
"read_options": {
"separator": string
}
}
}
```
| Field | Type | Value |
| ------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| type | string | Required. "S3" |
| id\_attribute | string | Optional. Name of the column in data that represents the identifier of the row. Read more [here](broken://pages/-MkctvuxvvtTvQH_pk0j) |
| bucket | string | Required. Name of S3 bucket |
| path | string | Required. Full path of file inside the bucket. Read more [here](broken://pages/-Mkcua8il6MsKqGOJ33C) |
| aws\_key | string | Required. AWS access key |
| aws\_secret | string | Required. AWS secret key |
| region | string | Required. AWS region |
| read\_options | json | Provide this only for CSV input if there is a separator like “,” or “\t”. For parquet and json, omit read\_options and separator |
| separator | string |
For CSV only,
‘,’ for CSV
|
# Azure Request data
### Request Body for Azure
```
{
"type":string,
"id_attribute": string,
"payload": {
"bucket":string,
"path":string,
"read_options": {
"separator": string
},
"security": {
"sas_key": "string",
"storage_account": "string"
}
}
}
```
| type | string | Required. "azure" |
| ---------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| id\_attribute | string | Optional. Name of the column in data that represents the identifier of the row. Read more [here](broken://pages/-MkctvuxvvtTvQH_pk0j) |
| bucket | string | Required. Name of the bucket |
| path | string | Required. Full path of file inside the bucket. Read more [here](broken://pages/-Mkcua8il6MsKqGOJ33C) |
| read\_options | json | Provide this only for CSV input if there is a separator like “,” or “\t”. For parquet and json, omit read\_options and separator |
| separator | string |