[I] [Feature Request]: Replaying Workflow Engine, Replay Gates, and Pre-Flight Replay Evaluation (hop)

["mattcasters (via GitHub)" <gi...@apache.org>]

mattcasters opened a new issue, #8648:
URL: https://github.com/apache/hop/issues/8648

   ### What would you like to happen?
   
   ### Background & Context
   This proposal builds upon and synthesizes the ideas in:
   - #6797: *A safe single threaded pipeline engine* (disk-spooling rowsets and step resumption)
   - #6537: *Dev Cache Mode / Step Checkpoints* (avoiding slow upstream re-queries during pipeline development)
   
   ### The Real Problem: Operational Outages vs. Technical Retries
   Distributed processing frameworks like Apache Spark or Apache Flink address *technical infrastructure faults* (e.g., worker node failure, disk I/O errors, rescheduling a failed task on a healthy executor). However, in real-world production data engineering, the dominant causes of failure are *environmental and operational*:
   - An upstream source database undergoes maintenance or crashes for several hours.
   - A remote network connection drops.
   - A cloud API rate-limits or revokes tokens.
   
   In these operational failure scenarios, automated task retries fail repeatedly. Worse, when the source database or API recovers hours later, its state has mutated. If an extraction failed halfway through, an engine cannot simply "continue where it left off" from the live database—doing so would stitch together a temporally inconsistent "Frankenstein" dataset (e.g., 30% from 02:00 AM and 70% from 05:00 AM).
   
   Generic, automated resumption across arbitrary live data sources is fundamentally flawed. Instead, we need to **manage the chaos** by giving data engineers explicit tools to declare snapshot boundaries, contracts, and replay policies at both the pipeline and workflow orchestration levels.
   
   ---
   
   ### Architectural Proposal
   
   #### 1. Replay Gates (Sealing Gates) on Transforms and Actions
   Instead of guessing where data boundaries lie, users explicitly mark transforms or workflow actions as **Replay Gates**:
   - **GUI Context Actions**: Modeled after `plugins/misc/debug` (`TransformDebugGuiPlugin`), transforms and workflow actions provide a `@GuiContextAction`: *"Mark as Replay Gate (Seal Output)"*.
   - **Canvas Visual Decorator**: Modeled after `DrawTransformDebugLevelBeeExtensionPoint`, using `PipelinePainterTransform` and `WorkflowPainterAction` extension points to render an overlay badge (e.g. a gate/seal icon) over marked components.
   - **Attributes Map Persistence**: Settings are persisted cleanly in `pipelineMeta.getAttributesMap().get("replay-gate")` and `workflowMeta.getAttributesMap().get("replay-gate")`, keeping `.hpl` and `.hwf` definitions clean and backward-compatible.
   
   #### 2. Persistent Spooling RowSet & Resilient VFS Storage
   - Implement `PersistentSpoolingRowSet` backed by Apache Commons VFS (`HopVfs`).
   - When a pipeline hits a Replay Gate, rows are serialized (`rowMeta.writeData()`) with block compression (Snappy / ZSTD) to a designated VFS path.
   - **VFS Everywhere**: Spool paths can target local scratch (`file:///...`), Amazon S3 (`s3://...`), Apache Hadoop HDFS (`hdfs://...`), or Azure Data Lake (`azure://...`). If a Kubernetes pod or batch VM terminates mid-pipeline, the spooled data and tamper-evident manifest survive in S3/HDFS.
   - **Snapshot Manifest (`SnapshotManifest.json`)**: When the extraction reaches completion without error, the gate writes a manifest containing the `snapshotId`, row count, serialized `IRowMeta` schema, and SHA-256 checksum, marking the snapshot status as `SEALED`. If interrupted, it is marked `INTERRUPTED`.
   
   #### 3. Pre-Flight Replay Evaluator (Safety Gate)
   Before allowing an operator to replay a failed workflow, Hop runs a deterministic pre-flight evaluation:
   - **Upstream Gate Verification**: Verifies that all required Replay Gates prior to the failure point exist, are marked `SEALED`, and pass SHA-256 / row count validation.
   - **Interruption Guard**: If a source extract was interrupted mid-stream, downstream replay is **strictly blocked**, alerting the operator that the source data was partially extracted and must be re-staged from scratch.
   - **Divergence Detection**: Ensures pipeline schemas and workflow variables have not incompatibly drifted.
   
   #### 4. The Replaying Workflow Engine (`ReplayWorkflowEngine`)
   A dedicated workflow engine plugin (`@WorkflowEnginePlugin(id = "Replay")`):
   - Reads the previous execution record from the active `IExecutionInfoLocation`.
   - Bypasses verified, completed upstream actions while populating their recorded `ActionResult` and workflow variables.
   - Resumes execution directly at the designated action (or the first failed action).
   - For child pipelines, binds the remote VFS spooled snapshot so the pipeline runs against the sealed data without touching the source database.
   
   #### 5. Operations Experience in the Execution Perspective
   In the Hop GUI **Execution Perspective** (`WorkflowExecutionViewer`):
   - A new toolbar action: **"Replay Workflow..."**.
   - Opens a Replay Dialog showing:
     - A pre-flight checklist with status badges (green checkmarks for verified sealed snapshots, red warnings for interrupted extracts).
     - Selection of replay start point (e.g. *"Resume from first failed action"*, *"Re-run entire workflow"*, or custom selection).
     - Parameter / variable review and overrides.
   - Initiates the replay run and links it to the original failed execution for complete operational auditability.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Workflows
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org