Skip to main content

Active Metadata Engine (AME) – v2 Release Notes

Version: 25.10 Release Date: December 1, 2025

This release delivers major improvements across the Service API and SQL Generator API, significantly expanding the platform's metadata intelligence and automation capabilities. The Service API introduces advanced field categorisation, refined ingest-flow control, new workload-management endpoints, and a broad set of enhancements to lineage, auditing, scheduling, and source-file metadata handling. Meanwhile, the SQL Generator API receives powerful new SQL-generation logic—including list-split inheritance, improved formatting, CROSS APPLY support, and advanced multi-file output handling—paired with numerous reliability and compatibility fixes. Together, these updates deepen lineage accuracy, strengthen ingest governance, and modernize SQL generation for complex hierarchical data structures.

Service API​

🚀 Features

  • Introduced FieldCategorization model with new endpoints to fetch, create, and update field domain categorisations, including full version tracking.
  • Added full support for ingest claimed logic and data classifications, improving ingest flow control.
  • Implemented a new startWorkload API and connection logic, enabling explicit workload initiation and state management.
  • Enhanced logging and publisher functionality with new log-by-date endpoints and additional attributes in log tables.
  • Added support for SplitToTable at the source-file level, including new endpoints and refined validation.
  • Expanded OpenLineage functionality with improved stage patterns, trusted dataset handling, namespace fixes, PDQ view integration, and detailed/attribute-level lineage options.
  • Added ability to rerun file and custom ingest source types.
  • Introduced a new API for rescheduling ingest flows with improved date handling (useAsValidFrom, mapping updates).
  • Added new attributes and metadata fields across SourceFiles and related endpoints (listLevels, targetNormalization, additional columns, updated stage schemas).
  • Added audit endpoints for publisher workflows and improved original message logging.

🐛 Bug Fixes

  • Corrected logic for workload claiming, SQL status comparisons, and transaction consistency.
  • Fixed several SQL issues (datatype conversions, substring args, coalesce handling, wrong-table inserts, missing columns in merge statements).
  • Fixed SplitToTable and SourceFile validation issues, including skip-file logic and corrected column names.
  • Corrected multiple namespace issues in OpenLineage (trusted datasets, stage patterns, uppercase handling, relationship schemas).
  • Repaired endpoint regressions for post_lastfiltervalue, including corrected URLs, validation, and handling empty filter columns.
  • Fixed various alignment, typo, and default-value issues across ingest and publisher APIs.
  • Ensured f_ prefixes are not incorrectly defaulted on field names.

🔧 Maintenance

  • Refactored SQL queries across scheduler, ingest, and workload logic for clarity and performance (including support for never schedule types).
  • Cleaned unused functions, improved comments, and streamlined logic in OpenLineage and relationship mapping APIs.
  • Updated logging schemas with new columns, renamed fields, and improved error message handling.
  • Improved handling of DLS namespace parameters and enhanced SourceFiles metadata representation.
  • General cleanup: removed stray print statements, updated validations, merged master branches, and aligned versioning (API v3.2).

SQL Generator API​

🚀 Features

  • Added list-split with child inclusion, enabling parent-level SQL generation that automatically includes child levels while discarding deeper descendants.
  • Introduced a new SQL formatter and improved key propagation logic across nested structures.
  • Implemented CROSS APPLY support for Microsoft SQL publishing patterns.
  • Added support for splitting a single input file into multiple SQL outputs, improving modularity and performance.
  • Introduced faster join strategies for publisher workflows, optimised for smaller subsets.
  • Added default key columns to prevent SQL failures and improved checksum handling at correct hierarchy levels.

🐛 Bug Fixes

  • Fixed corrupted comment handling and ensured comments are returned as proper strings rather than text lists.
  • Resolved issues related to hierarchical file loading (path mismatches, aliasing, incorrect field paths, and . vs -> misuse).
  • Corrected multiple privacy and build-order issues, improving SQL template generation stability.
  • Fixed missing blob file paths, token leakage, and incorrect SQL where conditions for Synapse.
  • Addressed backwards compatibility failure for DEFAULTCOLUMNS by introducing separate templates.
  • Fixed numerous typos, casing issues, and alignment problems affecting SQL generation consistency.

🔧 Maintenance

  • Cleaned up repository routes and removed obsolete endpoints.
  • Added richer error messages and improved debugging output for join optimisation.
  • Updated SQL generator to use the correct API version and improved version naming for temporary output files.
  • Applied consistent casing strategies to final SQL output and improved default column path definitions.
  • Improved splitting-to-table workflow: added unit tests, completed required logic, and optimised sample-row generation.