Skip to main content

DLS (Data Lake Service) – v3 Release Notes

Version: 3.2 Supersedes: release/v25.10.4

The 3.2 line reworks how a landed file is handed on. External storage triggers are gone, profiling runs by default after a file lands, and a failed audit log is now visible rather than silently absent. See What changes in 3.2 for the platform-wide picture, and the Landing section there for the message-flow detail.

🚀 Features

  • Pipeline handover and flow stability. External storage blob-creation triggers were replaced with in-process chunk publishing (queue:qpublisher) and direct completion messaging (queue:qfileready, queue:qmodifyfile, queue:qprofile). This removes the external dependency from the handover path and accelerates downstream handover velocity.
  • Profiling runs by default. After a successful organise with an accepted audit log, the file is posted to the modify queue and handed on, and additionally to the profile queue when targetData.enableProfiler is 1 — the default when the field is absent. Set it to 0 to opt a source out.
  • Transaction and upload resilience. Strict per-file transaction atomicity is enforced across all file levels, and chunk streaming is zlib-compressed to eliminate memory growth during high-volume uploads.
  • Governance and audit trail. Audit metadata is pre-persisted before the file upload; audit logging was upgraded to millisecond precision with store-relative paths; and failed messages are routed to automatic parking queues (<zone>-parked).
  • Security and credentials. Added Snowflake key-pair (JWT) authentication, dynamic credential resolution through AWS Secrets Manager and Azure Key Vault, and an offline key-verification tool.

🐛 Bug Fixes

  • Transaction and execution atomicity. Per-file transaction handling in DataPublisher was fixed so connection lifecycles execute within a single transaction, row-count reporting was corrected, and USE WAREHOUSE execution is now restricted strictly to Snowflake targets.
  • Queue and retry routing. Trashcan queue routing now directs failed messages to queue:trashcan, and the retry threshold logic in DataModifier was corrected to retries < 5.
  • Timestamp and path formatting. Event-time conversion errors were resolved using _to_local_naive() to prevent timezone-offset bugs, double slashes in constructed URLs were eliminated, and store-relative path formatting was standardised.
  • API payload and header integrity. Backend form-data rejection was resolved by explicitly setting Content-Type: application/json headers across DataProfiler and DataOrganizer.
  • Upload finalization. Uploads are now finalised explicitly (flush_data(close=True)) so blob notifications fire with accurate content lengths.

Upgrade notes​

  1. objectUrl is now a store-relative path, not a fully-qualified URL. Anything reading it must prepend the host itself. The .gz suffix is applied in exactly one place.
  2. queue:qfileready is gone from the external-trigger path — completion is messaged directly. Start the modify and profile consumers before deploying the file mover, or messages accumulate behind it.
  3. Monitor the <zone>-parked queues. Audit logging retries five times and then parks the message; parked messages are not retried automatically.
  4. A message with no objectUrl now reports failure, where it previously reported success.
  5. Audit logging is limited to the landing zone; timestamps carry milliseconds so files landing inside the same second can be ordered.