Skip to main content

DWA (Data Warehouse Automation) – v2 Release Notes

Version: 2025.6.1
Release Date: June 16, 2025

This release introduces comprehensive support for Microsoft Fabric, enabling the full automation suite to run on Microsoft's latest analytics platform. It also includes a significant new OpenLineage integration for enhanced data traceability and a new JWT authentication method for improved, unified security across APIs.

SQL Execution Engine​

A summary of all changes made to the execution engine.

🚀 Features​

  • Added direct support for Microsoft Fabric workloads, including connectivity using Service Principal authentication for secure, automated access.
  • Introduced OpenLineage integration to capture start and end events for all executions, providing robust job monitoring and data lineage tracking.

🐛 Bug Fixes​

  • Improved error reporting by removing a control mechanism that could hide underlying database errors during execution.

🔧 Maintenance​

  • Standardised the naming of the environment configuration file from mq.settings.env to mq.env.

SQL Generator API​

A summary of all changes made to the SQL generation logic and templates.

🚀 Features​

  • Implemented full SQL generation compatibility for Microsoft Fabric, including a new fabric dialect and updated logic for all core data loading patterns.
  • Deployed a new, secure JWT OAUTH2.0 authentication method, allowing for a coordinated and unified user base across all APIs.
  • Added new "Order by" parameters to the AttributeLoader, Publisher, and RelationLoader patterns for more control over data processing output.

🐛 Bug Fixes​

  • Corrected the generation of full ADLS URLs for OPENROWSET commands.
  • Fixed an issue with incorrect characters appearing in logging columns for stage loads.
  • Corrected a minor naming issue in generated statistics.
  • Modified Snowflake data type conversions to be more tolerant of empty JSON strings, preventing errors in the publisher.

🔧 Maintenance​

  • All SQL patterns were refactored to use temporary tables (#temp), improving reliability and alignment with SQL Server best practices.
  • The relationship loading pattern was updated to align more closely with the behaviour of Synapse and SQL Server.
  • Refactored internal context handling within SQL templates for cleaner and more maintainable code.