Skip to main content

INGEST (Data Importer) – v3 Release Notes

Version: 3.2 Supersedes: release/v25.10.1

The 3.2 line broadens connector coverage, rewrites the CDC engine for large extracts, and formalises the workload execution lifecycle. See What changes in 3.2 for the platform-wide picture.

🚀 Features

  • Expanded connectors and streaming. Added a dedicated Message Queue exporter family (SQS, Azure Service Bus, Redis streams), a SharePoint Graph API exporter, a Quinyx SOAP exporter, and NAS bureau file processing.
  • High-throughput ETL performance. The CDC engine was rewritten with DuckDB disk spilling and automated ~2 GB part-file splitting, server-side database streaming cursors (SSCursor) were added, and chunked file downloads were enabled.
  • Partitioning and lifecycle. Dynamic partition discovery was introduced across S3, ADLS and SFTP file sources, time token substitution was standardised, and the workload execution lifecycle was formalised.

🐛 Bug Fixes

  • ETL error propagation. CDC error handling now propagates exceptions directly, failing the workload rather than silently falling back to an uncompressed full extract. An upload error part-way through a list now marks the workload state Error.
  • Connection protocols. OAuth 2.0 token handling was reworked to branch correctly on grant_type, SSL default verification was corrected, and custom connection configuration parsing was fixed.
  • File and partition operations. Timezone localisation in S3 partition searching was fixed, leading slashes on target paths are stripped to prevent empty root folders being created, and exponential backoff retries were added for ADLS uploads.
  • Workspace hygiene. The working directory is now cleaned up automatically at workload completion, while version tracking files are preserved.

Upgrade notes​

  1. CDC failures are no longer silent. A workload that previously fell back to an uncompressed full extract on a CDC error now fails. Check any monitoring that treated a completed full extract as a healthy CDC run.
  2. Target paths with a leading slash change shape. The leading slash is stripped, so the empty root folder previously created by such paths no longer appears.
  3. OBJECTSTOREURL replaces AZUREBLOBACCOUNTURL and is scheme-neutral. The old name still works; if both are set, the new one wins.