You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/03/17 03:26:38 UTC
[GitHub] [dolphinscheduler] lhjzmn opened a new issue #8951: [Feature][Master] Worker resources are exhausted and cause downtime
lhjzmn opened a new issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
After I upgraded from version 1.3.8 to 2.0.5, worker resources were exhausted due to too many tasks submitted by users。
### Use case
I looked through the source code and saw that in MasterConfig 1.3.8, there is a parameter masterExecTaskNum with a value of 20, that is, only 20 tasks can be submitted to the worker for execution at the same time. After upgrading to 2.0.5, although this parameter exists, it is not used. The master will rudely submit tasks to the workers for execution. I don't understand why it is designed this way.
This problem is fatal in production. I think the number of tasks running at the same time should be controlled, like version 1.3.8, to ensure that the pressure on the workers is not too high, even at the tenant level (ie each tenant can run at most N jobs at the same time)
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] lhjzmn commented on issue #8951: [Feature][Master] Worker resources are exhausted and cause downtime
Posted by GitBox <gi...@apache.org>.
lhjzmn commented on issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951#issuecomment-1070196210
@caishunfeng @lenboo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng closed issue #8951: [Bug][Master] Worker resources are exhausted and cause downtime
Posted by GitBox <gi...@apache.org>.
caishunfeng closed issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #8951: [Feature][Master] Worker resources are exhausted and cause downtime
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951#issuecomment-1070241847
Worker thread was controlled by ```worker.exec.threads``` in ```worker.properties```.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8951: [Feature][Master] Worker resources are exhausted and cause downtime
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951#issuecomment-1070204369
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #8951: [Bug][Master] Worker resources are exhausted and cause downtime
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #8951:
URL: https://github.com/apache/dolphinscheduler/issues/8951#issuecomment-1070356528
It is not a bug, and you can solve as @SbloodyS said. I will close this issue.
BTW, this issue is not a feature, I change the title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org