Skip to main content

DLS (Data Lake Service) – v2 Release Notes

Version: 2025.10.1 Release Date: December 01, 2025

This release delivers significant improvements across the Data Lake Service components, strengthening cross-cloud streaming, metadata handling, and publishing workflows. The DataOrganizer now features a unified, high-performance streaming architecture for both Azure and AWS, along with improved compression, enhanced multipart upload stability, and expanded support for Data Lake API operations. The EventListener receives important updates for source-file validation and API compatibility, while the DataPublisher introduces more reliable pipeline flow, publisher-level controls, and support for Microsoft Fabric. Together, these enhancements provide a faster, more robust, and more cloud-aligned data lake foundation.

DataOrganizer​

🚀 Features

  • Unified streaming architecture for Azure and AWS with consistent 8MB chunking for improved performance and lower memory usage.
  • Added new S3 streaming compatibility, including updated transfer configs and improved multipart upload handling.
  • Introduced enhanced streaming compression with a refactored StreamingGzipCompressor, supporting optional compression of uncompressed streams.
  • Migrated Azure uploads to the Data Lake API with append-and-flush streaming, enabling better event triggering and full Hierarchical Namespace support.
  • Improved SDK usage across AWS and Azure to better support unknown-length streams and continuous streaming without buffering whole files.

🐛 Bug Fixes

  • Fixed an issue where parallel S3 uploads corrupted compressed data.
  • Corrected stream position resets and truncation behaviours to avoid corruption during uploads.
  • Resolved issues in compression handling related to EOF, buffer flushing, and data boundaries.
  • Fixed path validation for raw destinations by removing leading slashes and enforcing trailing slashes.
  • Reverted previous chunk size regression to ensure stable compression behaviour.

🔧 Maintenance

  • Refactored streamUploader and streamDownloader logic to reduce memory usage and simplify upload paths.
  • Updated documentation and inline comments for clarity on streaming behaviour and API usage.
  • Removed redundant error-handling paths and streamlined upload workflows for Azure and AWS.
  • Improved overall stability by standardising compression and streaming logic across cloud providers.

EventListener​

🚀 Features

  • Added a new update source file endpoint, improving the ability to modify metadata for existing source files.
  • Updated the getSourceFiles read format to align with the latest API behaviour.
  • Advanced the component to API version 3.2 for improved compatibility and consistency across services.

🐛 Bug Fixes

  • Added validation to ensure that extracted filekey values from filenames are valid f-guids, preventing ingestion of malformed or incorrectly named files.

🔧 Maintenance

  • Updated CircleCI configuration to align with the new validation logic and ensure consistent build behaviour.

DataPublisher​

🚀 Features

  • Added logic to always run the root step, even when splitToTables is populated, ensuring consistent pipeline execution flow.
  • Introduced publisher-level handling, improving control and behaviour across different publishing modes.
  • Added support for Microsoft Fabric, expanding compatibility with modern data platforms.

🐛 Bug Fixes

  • Removed hard-coded clientType values to restore dynamic behaviour.
  • Multiple fixes addressing build errors, TypeErrors, typos, and minor logic issues throughout the publisher flow.
  • Restored previous endpoint behaviour to resolve incorrect routing in certain environments.

🔧 Maintenance

  • Pinned snowflake-connector-python<4.0 to avoid breaking changes in newer versions.
  • Improved logging, including restored older logging paths and new test logging for better debugging.
  • Reordered Docker Hub login steps in the CI process for more reliable builds.
  • Consolidated audit log wrappers by moving them to repository.py for cleaner structure.