You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "fishhami (via GitHub)" <gi...@apache.org> on 2023/08/25 02:34:23 UTC
[GitHub] [dolphinscheduler] fishhami opened a new issue, #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
fishhami opened a new issue, #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
[version]
3.1.7
[description]
I created a task group, set the resource capacity to 2, and created 6 tasks at the same time.2 tasks will start to execute, and 4 tasks are in the status of submitted successfully. But when those two tasks are executed, the task group resources are in an idle state at this time, and the four tasks that were submitted successfully before are not scheduled for execution, and have always been submitted successfully. Then I create a new task, this task can obtain the task group resources and start executing。
### What you expected to happen
When the task group resources are free, the tasks in the queue should be scheduled and executed normally instead of being blocked.
### How to reproduce
<img width="1424" alt="image" src="https://github.com/apache/dolphinscheduler/assets/54088190/364f9690-fbe9-4dbb-9160-6bd1bee3e15b">
<img width="1105" alt="image" src="https://github.com/apache/dolphinscheduler/assets/54088190/1436b22f-a519-4f55-8d8b-9c09ce20a6cc">
From the above, we can see that the 'data-analysis' task group has idle resources, but there are still 4 tasks in the WAIT_QUEUE state, and they are not scheduled for execution.
I don't know if there is a queuing timeout? Because during the test, I found that if the task group can release resources quickly, the tasks in the queue can be scheduled for execution. But when the previous task takes a long time, it will appear that although the resources are finally released , the previously queued tasks will no longer continue to be scheduled for execution
### Anything else
_No response_
### Version
3.1.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "wh474718572 (via GitHub)" <gi...@apache.org>.
wh474718572 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1786695489
这个问题解决了吗楼主，我也遇到了同样的问题，期待你的回复
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] DaHanII commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "DaHanII (via GitHub)" <gi...@apache.org>.
DaHanII commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1728695259
I met this issue right now, but even if I follow the suggestion that changing in_queue value to 0 in t_ds_task_group_queue, it still seems doesn't work. Need help !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] fishhami commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "fishhami (via GitHub)" <gi...@apache.org>.
fishhami commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1692684586
<img width="1058" alt="image" src="https://github.com/apache/dolphinscheduler/assets/54088190/7405b997-30f9-4715-9664-a478ded99c5c">
ADD: After further investigation, I found that the in_queue value of the blocked task in the t_ds_task_group table is 1. When I manually set the in_queue value to 0 in the database, the task can be scheduled normally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "2421994771 (via GitHub)" <gi...@apache.org>.
2421994771 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1916152423
I had the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
URL: https://github.com/apache/dolphinscheduler/issues/14813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1692799030
I remember this bug should already be fixed at the dev branch, but I cannot find the pr. Could you please help to test on dev?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "2421994771 (via GitHub)" <gi...@apache.org>.
2421994771 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1920716898
When a high priority task in the wake-up stage, IN_QUEUE is set to 1 after any exception, resulting in IN_QUEUE is not set to 0, then will lead to subsequent all tasks with a lower priority than it can not be consumed, even if the queue is empty, at present I here is due to the IN_QUEUE is set to 1, send WAKE_UP message to the master when the communication error caused by this phenomenon.
1. Due to the communication error will lead to task wake-up failure, failure will never be reawakened
2. The wake-up failure will lead to the existence of status=-1, in_queue =1 records in the t_ds_task_group_queue, the record will lead to all tasks with a lower priority than the record priority can not access the queue resources
Temporary solution:
through the SQL query out of the t_ds_task_group_queue table associated with the workflow being executed 'status = -1 and in_queue = 1 records and update the in_queue value of these records to 0 so that they can be consumed again
英文不不太好，附上中文:
当一个高优先级的任务在唤醒阶段，IN_QUEUE置为1之后发生任何异常，导致IN_QUEUE没有被置为0，那么将导致后续所有优先级比它小的任务都无法被消费，即使队列是空的。 目前我这边是由于在IN_QUEUE置为1后，发送WAKE_UP消息给master时通信异常导致的这个现象。
目前任务组功能发现两个bug：
1.由于通信异常会导致任务唤醒失败，失败后永远不会被再次唤醒
2.唤醒失败导致t_ds_task_group_queue中存在 status=-1, in_queue =1 的记录,该记录将导致所有优先级小于此记录优先级的任务无法获取队列资源
临时解决办法:
通过SQL查询出与正在执行中的工作流关联的t_ds_task_group_queue表中 'status = -1 and in_queue = 1 的记录 并将这些记录in_queue值更新为0 使他们能够再次被消费
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1902460312
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1705914585
> > I remember this bug should already be fixed at the dev branch, but I cannot find the pr. Could you please help to test on dev?
> > @ruanwenjun
> > Thanks for your reply，I pulled the code of the dev branch to compile and deploy, but an error occurred during the process of upgrading the database schema, which caused dolphinscheduer to fail to start normally.
>
> <img alt="image" width="1092" src="https://user-images.githubusercontent.com/54088190/263944604-748bfe9a-00ae-46d1-91a8-ff402bac0af7.png">
> I don't know if you are sure that this bug has been fixed, or if you can find out which codes have been modified, I can update and compile directly from version 3.1.7
Right now, we don't support auto-upgrade to dev.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1773953621
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1784290176
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
URL: https://github.com/apache/dolphinscheduler/issues/14813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1890174434
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "wh474718572 (via GitHub)" <gi...@apache.org>.
wh474718572 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1786695822
这个问题解决了吗楼主，我也遇到了同样的问题，期待你的回复
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "wh474718572 (via GitHub)" <gi...@apache.org>.
wh474718572 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1786795994
Have you solved this problem yet? I have also encountered the same problem and look forward to your reply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] fishhami commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "fishhami (via GitHub)" <gi...@apache.org>.
fishhami commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1696963361
> I remember this bug should already be fixed at the dev branch, but I cannot find the pr. Could you please help to test on dev?
@ruanwenjun
Thanks for your reply，I pulled the code of the dev branch to compile and deploy, but an error occurred during the process of upgrading the database schema, which caused dolphinscheduer to fail to start normally.
<img width="1092" alt="image" src="https://github.com/apache/dolphinscheduler/assets/54088190/748bfe9a-00ae-46d1-91a8-ff402bac0af7">
I don't know if you are sure that this bug has been fixed, or if you can find out which codes have been modified, I can update and compile directly from version 3.1.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "BenTao996 (via GitHub)" <gi...@apache.org>.
BenTao996 commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1825467985
Is this sloved？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] hc-leo commented on issue #14813: [Bug] [task group]The task group queue is idle, but the queued tasks are not executed
Posted by "hc-leo (via GitHub)" <gi...@apache.org>.
hc-leo commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1705206912
I once met with this problem. To reproduce it, trying to start multiple tasks at the same time over task group limitation. After one task is finished, the successor task of it will be triggered and those waiting in queue at the first beginning will never be executed.
Looking for more responses for this issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [task group]The task group queue is idle, but the queued tasks are not executed [dolphinscheduler]
Posted by "himper (via GitHub)" <gi...@apache.org>.
himper commented on issue #14813:
URL: https://github.com/apache/dolphinscheduler/issues/14813#issuecomment-1970203676
I have this issue on 3.1.18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org