You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "LYL41011 (via GitHub)" <gi...@apache.org> on 2024/04/07 09:28:59 UTC
[I] [Bug] [DEPENDENT node] DAY DEPENDENT [dolphinscheduler]
LYL41011 opened a new issue, #15807:
URL: https://github.com/apache/dolphinscheduler/issues/15807
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
Production Issue: Both Workflow A and Workflow B run multiple times a day, once in the morning and once in the afternoon, pushing the latest data to production. Workflow B relies on Workflow A.
The situation for Workflow A is as follows:
The run at 2 o'clock succeeded at 4:08.
The run at 4 o'clock failed at 4:02.
The downstream process, Workflow B, is scheduled for 2:10 and 4:10.
When the downstream is scheduled at 4:10, the dependencies can pass, leading to data quality issues. (Because the 4 o'clock run of Workflow B should not be allowed to pass because of the successful 2 o'clock run, as it is intended to have the latest data from the 4 o'clock run)
After testing and reproduction, as shown in the diagram, Dolphin Scheduler considers the end time.
### What you expected to happen
When the downstream is scheduled at 4:10, the dependencies can not pass,should faild
### How to reproduce
启动一个A工作流，shell脚本如下，制造成功实例，并且结束时间会晚于后面启动的那个工作流
sleep 1000
echo "success"
更改A工作流，shell脚本改成如下，制造失败实例，再启动
sleep 10
exit -1
启动B工作流，依赖A工作流，B工作流会在1000秒之后成功
### Anything else
_No response_
### Version
3.1.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [DEPENDENT node] DAY DEPENDENT BUG [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15807:
URL: https://github.com/apache/dolphinscheduler/issues/15807#issuecomment-2101704744
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [DEPENDENT node] DAY DEPENDENT BUG [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15807:
URL: https://github.com/apache/dolphinscheduler/issues/15807#issuecomment-2113688410
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [DEPENDENT node] DAY DEPENDENT [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15807:
URL: https://github.com/apache/dolphinscheduler/issues/15807#issuecomment-2041385358
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
Production Issue: Both Workflow A and Workflow B run multiple times a day, once in the morning and once in the afternoon, pushing the latest data to production. Workflow B relies on Workflow A.
The situation for Workflow A is as follows:
The run at 2 o'clock succeeded at 4:08.
The run at 4 o'clock failed at 4:02.
The downstream process, Workflow B, is scheduled for 2:10 and 4:10.
When the downstream is scheduled at 4:10, the dependencies can pass, leading to data quality issues. (Because the 4 o'clock run of Workflow B should not be allowed to pass because of the successful 2 o'clock run, as it is intended to have the latest data from the 4 o'clock run)
After testing and reproduction, as shown in the diagram, Dolphin Scheduler considers the end time.
### What you expected to happen
When the downstream is scheduled at 4:10, the dependencies can not pass,should faild
### How to reproduce
启动一个A工作流，shell脚本如下，制造成功实例，并且结束时间会晚于后面启动的那个工作流
sleep 1000
echo "success"
更改A工作流，shell脚本改成如下，制造失败实例，再启动
sleep 10
exit -1
启动B工作流，依赖A工作流，B工作流会在1000秒之后成功
### Anything else
_No response_
### Version
3.1.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [DEPENDENT node] DAY DEPENDENT BUG [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15807: [Bug] [DEPENDENT node] DAY DEPENDENT BUG
URL: https://github.com/apache/dolphinscheduler/issues/15807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [DEPENDENT node] DAY DEPENDENT BUG [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #15807:
URL: https://github.com/apache/dolphinscheduler/issues/15807#issuecomment-2041720428
Currently the dependency judgment logic is to take the latest one within the same dependency cycle for judgment. Multiple calls per day should not rely on cycle configuration days. I think this is a usage issue rather than a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org