Skip to main content

INGEST (Data Importer) – v2 Release Notes

Version: 25.10 Release Date: December 1, 2025

This release brings substantial advancements to ingestion performance, scalability, and cloud interoperability. It introduces full DuckDB-powered Change Data Capture, new streaming download capabilities across all major storage platforms, and expanded importer/exporter coverage including API, ADLS Gen1/Gen2, and enhanced filesystem operations. Significant improvements in workload management, S3 partition handling, and JSON serialisation further strengthen ingestion reliability. Paired with extensive optimisations in memory usage, chunking, compression, and cloud event handling, this update delivers a markedly more efficient and resilient ingestion pipeline for large-scale, multi-cloud data processing.

Ingestion Engine​

🚀 Features

  • Added full Change Data Capture (CDC) capabilities using DuckDB, including efficient diffing, comparison, and file-based CDC workflows.
  • Introduced streaming download support for SFTP, ADLS, and S3 exporters, enabling low-memory large-file ingestion.
  • Implemented a new apiExporter with full OAuth 2.0, bearer token, basic auth, and certificate authentication support.
  • Added Azure Data Lake Gen1 & Gen2 importers, plus multiple enhancements to ADLS uploading with block-blob staging for reliability.
  • Introduced a dedicated fileSystemExporter with file browsing, timestamp/filename filtering, multi-path binary export, and compression support.
  • Added support for compressed intermediate files, improved gzip handling, and optimised compression performance.
  • Enabled partition-aware S3 browsing, including dynamic partition discovery and improved time-range searching.
  • Implemented start_workload and workload completion handling for more robust workload management.
  • Added 5GB output file support, aligning export behaviour with large dataset requirements.
  • Integrated orjson for fast JSON serialisation in exporters.
  • Added BlobCreationTrigger and other cloud-event improvements to enhance integration reliability.

🐛 Bug Fixes

  • Fixed indentation error in SftpExporter causing bad mapping behaviour.
  • Corrected file pattern matching and wildcard handling in fileSystemExporter.
  • Fixed execution order issues in export workflows.
  • Resolved multiple datatype conversion issues in BuildSQL, post_lastfiltervalue, and tableexport logic.
  • Repaired incremental fetch logic and fixed indexing issues in dbExporter.
  • Corrected filterColumn handling and datatype inference when empty.
  • Fixed path normalisation issues causing incorrect root folders in S3/Azure.
  • Resolved endpoint issues in post_lastfiltervalue, including corrected POST URL and reverting breaking changes.

🔧 Maintenance

  • Extensive DuckDB optimisations:
    • Improved memory limits and file size constraints.
    • Enabled disk spilling, disabled parallel execution, and tuned chunk sizes.
    • Added sequential operation support and memory-efficient row streaming via generators.
  • Major refactors across exporters to improve memory efficiency, chunking, and file writing:
    • Consolidated binary file logic.
    • Optimised CDC file handling and filename normalisation.
    • Added dynamic chunk size estimation based on DuckDB column types.
  • Improved temporary file handling and S3 performance through better size tracking and cleaning logic.
  • Enhanced logging across S3, filesystem, DB exporters, and ingestion workflows.
  • Updated ingester to support exporter-mapping configuration files.
  • Removed debug code and simplified processing paths to increase performance and reliability.