You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:48:19 UTC
[I] Test using spark listeners that guards any changes to DAG [hudi]
hudi-bot opened a new issue, #15402:
URL: https://github.com/apache/hudi/issues/15402
For major write operations and table services, we should guard the DAG by making use of spark listeners. Some basic things can be validated such as no. of stages, tasks, executor runtime, records written.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4761
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test using spark listeners that guards any changes to DAG [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #15402:
URL: https://github.com/apache/hudi/issues/15402#issuecomment-5068088134
This issue was reviewed as part of the JIRA-migrated backlog triage.
Findings: This task (HUDI-4761) asked for tests that use Spark listeners to guard against unintended changes to the DAG for major write operations and table services. This was implemented in PR #8641 (HUDI-5980, merged 2023-05-21, commit `bf5715df326f`), which added `TestSparkDataSourceDAGExecution.scala` and `TestHoodieDeltaStreamerDAGExecution.java`. Both register a `SparkListener` hooking `onStageCompleted` to assert the number of stages triggered by write operations, guarding against repeated/unexpected DAG execution. Additional stage/parallelism listener guards also exist (e.g. `TestHoodieDatasetBulkInsertHelper`, `TestInsertTable4`).
Closing as completed.
If you believe additional operations still need dedicated DAG guards, please reopen with specifics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Test using spark listeners that guards any changes to DAG [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #15402: Test using spark listeners that guards any changes to DAG
URL: https://github.com/apache/hudi/issues/15402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org