[I] [Feature Request]: Data Quality Alerts in Execution Information (hop)

["CarlosJuncher03 (via GitHub)" <gi...@apache.org>]

CarlosJuncher03 opened a new issue, #8626:
URL: https://github.com/apache/hop/issues/8626

   ### What would you like to happen?
   
   I would like to propose a discussion about adding data quality alerts to Apache Hop Execution Information.
   
   Hop already stores valuable execution details such as status, logs, metrics, and profiled data. The idea would be to allow users to evaluate that information against user-defined rules and generate persistent alerts that can also be displayed in the Hop GUI.
   
   For example, a pipeline could define checks such as:
   
   - Null CPF values must not be greater than zero
   - Null name values must be below 1%
   - Rows sent to the “Error Map” hop must be below 500
   - Rows loaded to the target must represent at least 95% of input rows
   - Pipeline duration must be below 30 minutes
   
   At the end of an execution, Hop could evaluate these rules and save the results together with the Execution Information.
   
   Conceptual example:
   
   Execution: person_load #12345
   Status: Completed with alerts
   
   Rule: Null CPF values
   Actual value: 84
   Threshold: 0
   Severity: Failure
   
   Rule: Rows sent to Error Map
   Actual value: 2,184
   Threshold: 500
   Severity: Warning
   
   In addition to being linked to an execution, each alert could be stored in a dedicated structure in the configured Execution Information Location. This would make it possible to query and monitor alerts across multiple pipelines and workflows.
   
   The Hop GUI could expose these alerts through the existing notifications/problems area and provide a simple lifecycle:
   
   Open → Under investigation → Resolved / Ignored
   
   When opening an alert, users could see:
   
   - Related pipeline, workflow, and execution
   - Related transform or hop
   - Configured rule
   - Expected and actual values
   - Execution logs and metrics
   - Analysis comments
   - History of similar occurrences
   
   It could also be useful to define an action for each rule:
   
   - Record only
   - Raise an alert
   - Mark execution as warning
   - Mark execution as failed
   - Stop execution
   - Route rows to quarantine
   
   The intention is not to assume that every metric deviation is an error. For example, rejected rows can be expected in some data flows. Therefore, rules should explicitly identify which transform, hop, field, or execution metric is being monitored.
   
   Points for community discussion:
   
   Should this be part of the standard Execution Information model, or should it start as a plugin?
   
   What would be the best metadata model for data quality rules?
   
   Should alerts be stored in the same Execution Information backend or in a dedicated table/index?
   
   How could this integrate with existing data profiling, metrics, lineage, and logging capabilities?
   
   Which actions should be supported without making pipeline execution too complex?
   
   How can this be implemented with minimal performance impact for high-volume pipelines?
   
   I believe this could help evolve Hop from only recording execution details to also identifying, tracking, and auditing data quality deviations in a structured way.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Hop Gui, Component: Infrastructure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature Request]: Data Quality Alerts in Execution Information (hop)

["CarlosJuncher03 (via GitHub)" <gi...@apache.org>]

CarlosJuncher03 commented on issue #8626:
URL: https://github.com/apache/hop/issues/8626#issuecomment-5838129159

   I'm working with the execution data and was thinking about that point mentioned earlier. I wanted to know if this makes sense to the community. I think it would be a good feature, but I'm not sure if the way I envisioned it is the best approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org