Data Operations Console – v3 Release Notes

Version: 3.2.a Supersedes: release/v25.10.1

This is a rebuild of the Data Operations Console rather than an incremental release. Almost every page was rewritten or newly added, role-based access and audit logging were introduced, direct SQL was largely replaced by backend APIs, and the navigation model, visual identity and container image all changed. Read the Upgrade notes at the end of this page before deploying.

A full page-by-page reference for the rebuilt console is available under DataOps Console.

🚀 Features

• Rebuilt application shell. Start.py is no longer a landing page but the shell itself. Pages moved from pages/ to app_pages/ and are registered explicitly through st.navigation, grouped by domain — Ingestion, Data Lake, Data Warehouse, Data Quality, Reference and System.
• One global date range and cache TTL, seeded in the shell (default: last 24 hours) and shared by every page, so a filter set in the sidebar survives page switches, reloads and direct URL entry.
• Global sidebar header and footer with a live health badge, plus ghost-free page transitions and cache invalidation on refresh, page re-entry and filter change — a page never shows the previous visit's data.
• New: role-based access and audit logging. Roles are derived from the reverse proxy's X-User-Groups header (admin / developer / reader, defaulting to reader), and every user-submitted action emits a structured JSON audit line — user, role, page, action, target, detail and timestamp — to the container logs. All mutating actions are admin-only.
• New: System Health page, now the default landing page. It aggregates runtime logs, runtime metrics, job logs, job metrics and job status from the monitoring APIs into a single view, with issue banners that filter the relevant log table and scroll to it.
• New: Data Quality / QPI, split across a read-only QPI Monitor and an admin-only QPI Administration page for creating, editing, scheduling, triggering, force-closing and retiring checks. A guided QPI builder generates worker-compliant probe SQL from three presets — threshold, freshness and comparison — in the customer warehouse's SQL dialect; generated commands parse back into their parts and stay editable, and hand-written SQL falls through to advanced mode.
• New: DLS Publisher Trace, tracing published files and tables end to end through dedicated publisher endpoints that return far less data than the previous query-driven approach, with a publisher re-trigger action and corrected status handling.
• New: DLS Contract Validation, validating landed data against its contract by reading Delta tables directly. Data contracts can now also be created and updated from the app, backed by new pydantic models for the V3b contract shape (levels, fields, domains, key ordering, sensitivity and profiling flags).
• New: DLS Data Profiler, with level and field editors that track edits in session state so in-progress work is not lost on rerun, and contract generation from profiled structures.
• New: DWA Task Orchestration, showing the dependency graph of warehouse tasks and allowing a task to be started directly, with an audit record for every trigger.
• New: Loading Statistics, adding interactive pivot tables over warehouse loading statistics with timezone-aware time-series views and drill-down to per-sourcefile detail.
• INGEST rerun, skip and restart moved to the v3 workload endpoints; warehouse task restart goes through the API and bulk skip is supported. Export row and file counts are extracted and visualised, and ingest history accepts a source-system filter.
• Incomplete work is never hidden. Failed and long-running tasks survive every filter and the record cap, so the ingest and warehouse pages no longer cover up problems.
• Reference pages reworked. Data Lineage was rewritten on graphviz and grown from a stub into a full lineage explorer; Data Model gained subject-area management (create and edit subject areas from the app); Documentation was rebuilt, including definition generation through the SQL generator API.
• API-first data access. Direct SQL was replaced by repository API calls across the operational pages, with new endpoints for system health, QPI, publisher, scheduling and orchestration, model and contracts, extra processors and statistics.
• New Simplitics brand palette applied throughout, with Lato/Raleway web fonts, Material Symbols icons, a matching Streamlit theme, and task-state colours reworked to the palette (failures kept in a universal deep red).

🐛 Bug Fixes

• Fixed the earlier ingest rerun workflow, which could fail and then crash on its own failure message.
• Terminating a stuck QPI run now completes the task instead of leaving it in Failed, where it would continue to count as incomplete and block the check.
• Corrected publisher status handling and metrics, and added graceful handling of empty results and missing zone names.
• Field renames in the profiler are reverted rather than silently dropped when rejected.
• Fixed "sticky" Gantt file selection, so a selected file no longer persists into an unrelated view.
• Repository failures now render as actionable error cards rather than blanking the page or surfacing a raw traceback.

🔧 Maintenance

• Every remaining v2 API call was migrated to v3. Most are pure aliases and behave identically; three needed more than a path change — getModels/getModelObject now pin their mode explicitly, getIngesters moved to a single global list call instead of a per-system fan-out, and getConnection moved to a v3 route whose payload differs from v2.
• The container image was rebuilt as a two-stage build on debian12-minimal, running as a non-root user, with a health check and an explicit bind address.
• Charts migrated to Altair; deprecated Streamlit width arguments replaced per current API conventions.
• Dependencies: added streamlit-pivot and pydantic; removed matplotlib, seaborn, streamlit-aggrid, streamlit-agraph and streamlit-extras; pinned pandas<3.
• Two further profiling pages — DLS Data Profiling and Discover New Fields — ship parked: present in the repository but not registered in the navigation.
• The Data Flow Overview page was removed, superseded by the improved Data Model and Data Lineage pages.
• Container-status pinging was removed; health is now derived from the monitoring APIs instead of a hard-coded container list.

New and changed settings

Setting	Change
BACKEND_OPTIONAL_PATH / GENERATOR_OPTIONAL_PATH	New. Appended to the built API URL, so the backend can sit behind a path-routed reverse proxy instead of its own port.
BACKEND_CERTIFICATE	New. Controls TLS verification against the backend; when unset, the matching warnings are suppressed instead of flooding the logs.
PUBLISHER_ENABLED	New. Defaults to enabled; set to 0/false/no to hide publisher metrics and sections.
KEEPDATAINTRUSTEDFORDAYS	New. Retention window used in trusted-zone views (default 3).
CACHE_TTL	New default (120 s) applied to all cached data queries.
SERVICE_LIST / CONTAINER_LIST	Removed. Container pinging is gone.

Upgrade notes

1. Backend requirement. The console now depends on the v3 sourcefile, ingest, schedule, statistics, publisher, QPI and system-health APIs. Deploy a backend that exposes them before upgrading.
2. A reverse proxy is now mandatory. Roles and the audit identity come from the X-User-Groups and X-User-Email headers set by Caddy. Exposing the container directly makes every visitor a reader with an unknown audit identity, and would let a caller forge those headers.
3. SERVICE_LIST is no longer read. Remove it from deployment configuration; container status is no longer displayed.
4. Bookmarked page URLs change — pages moved from pages/ to app_pages/ and are now routed by st.navigation. The Data Flow Overview page is gone.
5. Rebuild the image, do not reuse a cached layer set: the base image changed and the process now runs as a non-root user. Any mounted volume the app writes to must be writable by that user.
6. Set PUBLISHER_ENABLED=0 on installations without the publisher component, otherwise Publisher sections render with zero counts.
7. INGEST checklist figures shift. With getIngesters on the global v3 list route, the last-scheduled timestamp now shows the last attempted export rather than the last completed one, and unscheduled or zero-interval definitions are no longer filtered out — they appear in the checklist and count as zero expected executions.