User-defined Monitors

User-defined monitors are policies that define the expected state or behaviour of your data. When data deviates from these policies, alerts are automatically triggered to notify your team. Monitors help you proactively manage data quality by continuously evaluating metrics, rules, and custom queries against your data assets.

When creating a user-defined monitor, you must attach it to an asset, and define the following components:

1. Metric: A quantifiable measure of data health (e.g., row count, null percentage, data freshness).

2. Threshold: A predefined value or range that, when exceeded, triggers an alert. Thresholds can be:

   1. Automatic: ML-based thresholds that analyze historical datapoints to determine dynamic boundaries

   2. Relative: Percentage-based boundaries that calculate moving averages from historical data

   3. Absolute: Constant, fixed boundaries

3. Notification endpoints: Which endpoints to notify when the monitor is alerting (Optional)

---

Monitor Types

Telmai supports three primary monitor types, each designed for different data quality scenarios:

1. Built-in Metric

2. User-Defined Metric

3. Record Validation Rule

Built-in Metric

Monitor built-in metrics managed by the Telmai platform.

Use cases:

• Track predefined data quality metrics (row count, null percentage, data freshness, completeness, etc.)

• Monitor standard data health indicators across your datasets

• Leverage out-of-the-box metrics without custom configuration

Example:

• Monitor record_count to ensure tables are being populated

• Check freshness to detect data delays

User-Defined Metric

Define and monitor custom metrics using Telmai expressions or push-down custom SQL.

Use cases:

• Create custom calculations and aggregations

• Monitor business-specific KPIs

• Define complex metric logic using SQL queries

Example expression:

SUM(salary)/COUNT(*)

Example custom SQL:

SELECT emp_salary, emp_Region FROM `employee_table` WHERE emp_Age > 60

Record Validation Rule

Define record validation checks and monitor the number of records passing these checks.

Use cases:

• Validate individual records against business rules

• Ensure data integrity at the row level

• Check data completeness and correctness across records

• Monitor the percentage of valid records over time

Example expression:

validate part_date expect is_date

The monitor tracks the percentage of records that pass the validation rule, and alerts are triggered when this percentage falls outside acceptable thresholds.

Click here to learn more about using and creating Data Quality rules.

---

Monitor Properties

Each monitor has the following properties:

Property	Description	Required
Monitor Name	Must be unique per asset (e.g., "Freshness Monitor" on sales_data is different from "Freshness Monitor" on customer_data)	Yes
Monitor ID	System-generated unique identifier	Auto-generated
Monitor Type	Built-in Metric, User-Defined Metric, or Record Validation Rule (cannot be changed after creation)	Yes
Description	Brief description for additional context	Optional
Monitor Tags	Tags for organizing and categorizing monitors	Optional
Impact	Severity level of the monitor (Critical, High, Medium, Low)	Optional
Data Quality Metric	For Built-in Metrics: the specific metric to monitor	Conditional
Attributes	For metrics that support scope: specific attributes to monitor	Conditional
Threshold Configuration	Automatic, Acceptable Drift %, or Acceptable Range	Yes
Creator/Editor	User who created or last modified the monitor	Auto-tracked
Creation/Update Time	Timestamp of creation and last update	Auto-tracked
History	Version history of monitor changes	Auto-tracked

Note: Monitor type can not be changed once set. To change the type, you must create a new monitor.

Related Documentation

• Built-in Metrics

• User-defined Metrics

• Record Validation Rules

• Monitors Management

• Incident Management