You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "reele (via GitHub)" <gi...@apache.org> on 2025/02/26 02:32:57 UTC
[I] [Bug] [AbstractWorkflowStateAction] Duplicate publish same `TaskStartLifecycleEvent` in `triggerTasks` [dolphinscheduler]
reele opened a new issue, #17030:
URL: https://github.com/apache/dolphinscheduler/issues/17030
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
when a task has a lot of predecessors, and they all finished in short time, `TopologyLogicalTransitionEvent` may published multi times in same time, before the first `TaskStartLifecycleEvent` handled, there will be a lot TaskStartLifecycleEvent in queue to handling.
### What you expected to happen
...
### How to reproduce
a part of the log
```log
...
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L1*]_41t TaskSuccessLifecycleEvent{task=[*Task-L1*]_41t, endTime=Wed Feb 26 09:39:24 CST 2025, varPool='null'} with state RUNNING_EXECUTION
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]_2rxtaskState=SUCCESS}] with state: RUNNING_EXECUTION
******* here the WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent was called several times *******
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskStartLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]_2rxtaskState=SUCCESS}] with state: RUNNING_EXECUTION
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]taskState=SUCCESS}] with state: RUNNING_EXECUTION
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskStartLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]taskState=SUCCESS}] with state: RUNNING_EXECUTION
[WI-4025052][TI-0] - 2025-02-26 09:39:24.194 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[47] - Begin fire workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]_41ttaskState=SUCCESS}] with state: RUNNING_EXECUTION
[WI-4025052][TI-0] - 2025-02-26 09:39:24.195 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskStartLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.195 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.w.l.h.AbstractWorkflowLifecycleEventHandler:[52] - Fired workflow dup-wf-20250226093922435 LifecycleEvent[WorkflowTopologyLogicalTransitionWithTaskFinishLifecycleEvent{task=[*Task-L1*]_41ttaskState=SUCCESS}] with state: RUNNING_EXECUTION
******* here the TaskDispatchLifecycleEvent was published 3 times *******
[WI-4025052][TI-0] - 2025-02-26 09:39:24.199 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskDispatchLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.199 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskStartLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskDispatchLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskStartLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskDispatchLifecycleEvent{task=[*Task-L2*]}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskStartLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskDispatchLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskDispatchLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-4025052][TI-0] - 2025-02-26 09:39:24.200 INFO [ds-workflow-eventbus-worker-6] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskDispatchLifecycleEvent{task=[*Task-L2*]} with state SUBMITTED_SUCCESS
[WI-0][TI-0] - 2025-02-26 09:39:24.215 INFO [MasterRpcServer-methodInvoker-1] o.a.d.s.m.e.WorkflowEventBus:[41] - Publish event: TaskDispatchedLifecycleEvent{task=[*Task-L2*], executorHost='10.0.6.23:1234'}
[WI-4025052][TI-0] - 2025-02-26 09:39:24.307 INFO [ds-workflow-eventbus-worker-28] o.a.d.s.m.e.t.l.h.AbstractTaskLifecycleEventHandler:[47] - Fired task [*Task-L2*] TaskDispatchedLifecycleEvent{task=[*Task-L2*], executorHost='10.0.6.23:1234'} with state SUBMITTED_SUCCESS
...
```
### Anything else
i think there
https://github.com/apache/dolphinscheduler/blob/d8506925dd12c110b228884aaa5c49dad416aa93/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/engine/workflow/statemachine/AbstractWorkflowStateAction.java#L68-L75
should add a filter
``` java
.filter((successor) -> !workflowExecutionGraph.isTaskExecutionRunnableActive(successor))
```
to prevent to get the same `TaskExecutionRunnable` in following calls of `triggerTasks`
### Version
dev
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [AbstractWorkflowStateAction] triggerTasks should prevent duplicated publishing of TaskStartLifecycleEvent to same TaskExecutionRunnable [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17030:
URL: https://github.com/apache/dolphinscheduler/issues/17030#issuecomment-2684467245
cc @ruanwenjun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [AbstractWorkflowStateAction] triggerTasks should prevent duplicated publishing of TaskStartLifecycleEvent to same TaskExecutionRunnable [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed issue #17030: [Bug] [AbstractWorkflowStateAction] triggerTasks should prevent duplicated publishing of TaskStartLifecycleEvent to same TaskExecutionRunnable
URL: https://github.com/apache/dolphinscheduler/issues/17030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [AbstractWorkflowStateAction] triggerTasks should prevent duplicated publishing of TaskStartLifecycleEvent to same TaskExecutionRunnable [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #17030:
URL: https://github.com/apache/dolphinscheduler/issues/17030#issuecomment-2684788747
@reele Good catch, the second way seems better, are you welling to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [AbstractWorkflowStateAction] triggerTasks should prevent duplicated publishing of TaskStartLifecycleEvent to same TaskExecutionRunnable [dolphinscheduler]
Posted by "reele (via GitHub)" <gi...@apache.org>.
reele commented on issue #17030:
URL: https://github.com/apache/dolphinscheduler/issues/17030#issuecomment-2684800306
> [@reele](https://github.com/reele) Good catch, the second way seems better, are you welling to fix this?
ok, i will.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org