Skip to main content

DWA (Data Warehouse Automation) – v3 Release Notes

Version: 3.2 Supersedes: release/v25.6.2

The 3.2 engine serialises target jobs, records every quality check individually, and tags every statement it sends with the job that issued it. See What changes in 3.2 — the Data quality and Traceability sections cover the reasoning behind these changes.

🚀 Features

  • Target job serialisation. Redis lock-based job serialisation (lock:<loadstep>) with fallback pending queues prevents concurrent target-execution conflicts.
  • Quality scheduling and tracing. The QPI quality-check engine was rebuilt with cadence-aware scheduling (daily, hourly, monthly, adhoc), independent execution tracking per check, native query tagging (QUERY_TAG, sqlcommenter) and OpenLineage parent-job event emissions.
  • Per-check execution records. Each check is keyed by (qpi, runDttm) and records its own state and result through its own start and finish calls — the batch status previously reflected only the last check in the batch. runDttm is taken from the message and written back onto it, so a retried batch reuses the same run key and start/finish stay idempotent. On completion the API rolls MDQpi.LastRunTm forward.
  • A failing test is distinguished from a broken check. A check whose SQL ran but whose test case failed is recorded Completed with resultState=fail and reported; a check whose SQL raised is recorded Failed with the real error and traceback, and the batch is retried. validate() now raises on any failure instead of returning a partial result.
  • Orchestration and hygiene. File triggers were upgraded with Redis SHA256 deduplication, and age-aware container cleanup passes were added for stuck and exited worker tasks.

🐛 Bug Fixes

  • Execution and result handling. setBatch empty-result-set handling now drops zero-identity DML counter rows, and database driver settings were made optional to prevent startup crashes on non-ODBC targets.
  • Task sequencing and gating. Load-order gating (identify_tasks_to_inititate) was rewritten to block dependent tasks when prior scheduled orders are incomplete.
  • State recovery. Restart handling was fixed, and residual state and information keys are cleared before re-queueing tasks to prevent stale error text propagating into the next run.
  • Queue stability. Redis queue operations were hardened with StrictRedis keepalives and health checks, and container cleanup logging noise was silenced.

Upgrade notes​

  1. Move every worker to REDIS. The SQS and Azure Storage Queue backends are removed; a deployment left on one falls through to a null queue silently.
  2. Query tagging needs no configuration. A static QUERY_TAG or APP value already set is preserved, not discarded. The tag lands in QUERY_TAG on Snowflake, user_agent_entry on Databricks, application_name on PostgreSQL and the ODBC APP on SQL Server, Synapse and Fabric, with a leading sqlcommenter comment on all targets.
  3. Quality definitions are no longer written by the worker — that endpoint is reserved for editing them.