Skip to main content

The daily pass

Run this pass every operational day, before the daily deadline. It checks that each automated step actually ran — and, more importantly, that nothing was silently skipped, which a failure count will never tell you.

The whole routine is done from the DataOps Console.

The operational pillars​

1. Centralised visibility and self-service management​

The DataOps Console serves as the primary control center, designed to simplify daily tasks and provide immediate insights into platform performance.

  • Runtime Slicing and Dicing: The console provides the ability to inspect runtime operators and statistics, offering detailed information about ongoing processes.
  • Failure Resolution (Idempotency): Tasks, particularly those managed by DWA, are designed to be restarted safely via the console, as they are idempotent (can be rerun without risk of duplication or corruption). Restarting failed DWA Loading Tasks is a simple fix available to operators.

2. Validating end-to-end data integrity​

Daily operations focus on verifying that data acquisition and subsequent processing phases maintain traceability and consistency throughout the system's architectural layers.

  • Source Acquisition Audits (INGEST): Operators use the INGEST Tasks views to confirm that the required data extraction agents have run as scheduled (Expected = Completed, Failed = 0) and that all expected file types have been delivered. The DLS Task Checklist confirms the second half of that: a green banner reading "All configured files have at least one delivery" means every configured file type arrived at least once in the selected window. A task that never started shows up here and nowhere else — a "no failures" metric will not reveal it.

  • Flow Consistency (DLS Trace): The DLS Trace view verifies data movement through the standard processing layers:

    Landing → Raw Archive → Trusted, with Profile alongside

    Successful flow is validated when the counts for Landing, Raw Archive, Trusted and Profile are identical. This confirms that the DLS Worker has performed archival, standardisation of format, and profiling. The Published count may sit slightly lower — a difference of up to 10–20 records is normal and not by itself a fault.

    Nothing is filtered out between the zones, so a shortfall means something is stuck, not that something was discarded — see What the platform records, and what it stops.

3. Orchestration and model integrity​

The DWA Loading Tasks view confirms that the received and processed files reached the integrated model, and that the metadata-driven generation produced the SQL and orchestration it was supposed to.

The check is:

  • Expected loading tasks and Completed loading tasks are identical
  • Failed is 0

Open the DWA Loading Tasks view, compare expected against completed, investigate anything failed straight away, and confirm the data is there in the target systems.

DataOps Console​

The DataOps console is the central point for monitoring and simple management of runtime operators and statistics. It groups its pages by the stage of the platform they cover — Ingestion, Data Lake, Data Warehouse, Data Quality, Reference and System — each with its own monitoring views, checklists and admin actions.

Page-by-page reference

This page describes the routine. For what each console page shows, what every metric means and which corrective actions it offers, see DataOps Console.

Daily monitoring checklist​

Follow this sequence every operational day. It is the same order used throughout these docs — services first, then the data, from the source outwards.

  1. ☑ System Health

    • Are the services up, and did anything fail or warn overnight?
    • Any row with status FAILED has to be restarted from the page that owns the work

  2. ☑ INGEST Tasks

    • Confirm that every expected run completed (Expected = Completed, Failed = 0)
    • Look for completed runs that returned zero rows — the classic silent failure
    • Confirm the green banner on the DLS Task Checklist: "All configured files have at least one delivery"

  3. ☑ DLS Trace

    • Verify count consistency across the zones (see the rule above)
    • Check for files that stalled at a zone, using the Last seen on filter

  4. ☑ DWA Loading Tasks

    • Expected and Completed must match, and Failed must be 0
    • Restart failed tasks — they are idempotent

  5. ☑ DWA Additional Tasks

    • Confirm that the chained post-processors ran

  6. ☑ QPI Monitor

    • Did any quality check report a failing result?
    • Keep Fail (the data is wrong) apart from Error (the check itself is broken)

Document anything you could not resolve, and follow up on it before the next pass.