You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "Gallardot (via GitHub)" <gi...@apache.org> on 2023/10/23 11:58:34 UTC
[I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Gallardot opened a new issue, #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
During the discussion with @ruanwenjun on WeChat, we have come to the following conclusion: when deploying dolphinscheduler on k8s, the overload protection feature should be removed.
The reasons are as follows:
1. The judgment of obtaining overload is not very accurate, whether it is deployed on k8s or on the host.
2. When deployed on k8s, if we want to obtain the CPU usage or remaining memory of the container, there will be some ambiguity in the calculation. Should we use request or limit? Workers also execute scripts such as bash and python, should these be considered in the calculation?
3. K8S itself has resource limitation mechanisms, as well as KEDA, HPA and other capabilities, so this feature is not necessary.
4. To implement overload protection, controlling the number of workflows or tasks may be more appropriate.
WDYT? @EricGao888 @SbloodyS @Radeity @qingwli
We also found a few related issues:
#15049 #14968
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15066: [Improvement][K8S] Remove overload protection when deploying on k8s
URL: https://github.com/apache/dolphinscheduler/issues/15066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1832905592
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot closed issue #15066: [Improvement][K8S] Remove overload protection when deploying on k8s
URL: https://github.com/apache/dolphinscheduler/issues/15066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "qingwli (via GitHub)" <gi...@apache.org>.
qingwli commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1878158117
I agree when deploying dolphinscheduler on k8s, the overload protection feature should be removed.
K8s also have limit on namespace, we can use defalut resoure for all service in ds.
https://github.com/apache/dolphinscheduler/blob/e5a512803ec6f8728fe05f4a5433231ab1d13c7e/deploy/kubernetes/dolphinscheduler/values.yaml#L711-L718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "hdygxsj (via GitHub)" <gi...@apache.org>.
hdygxsj commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1878169027
> > Any progress？ @Radeity
>
> Seems like no one works on it currently, do you have any idea about it?
Our company is currently running on version 3.2.0, it resulted in the inability to schedule workflow normally and our implemented solution is modifying the source code to remove the overload protection feature, as well as utilizing the hpa.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1845651240
> Any progress？ @Radeity
Seems like no one works on it currently, do you have any idea about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "hdygxsj (via GitHub)" <gi...@apache.org>.
hdygxsj commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1845637654
Any progress？ @Radeity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1775375157
Hi @Gallardot
Good catch, the calculation of memory and cpu usage are all wrong in PR#14968. However, I think we should correct this calculation first, cuz we have to show correct metrics in monitor page. Maybe use **LIMIT** and consider all processes running on the worker/master pod sounds good to me.
> K8S itself has resource limitation mechanisms, as well as KEDA, HPA and other capabilities, so this feature is not necessary.
Agree with you that we can fully utilize these capabilities when deployed in K8S, but if we don't set KEDA/HPA by default, we have to keep current overload protection strategy with correct metrics to avoid OOM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "Gallardot (via GitHub)" <gi...@apache.org>.
Gallardot commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1914052374
Close this issue, because #15439 has been merged, providing a switch for enabling or disabling overload protection. Thanks to @ruanwenjun .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Improvement][K8S] Remove overload protection when deploying on k8s [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15066:
URL: https://github.com/apache/dolphinscheduler/issues/15066#issuecomment-1823687164
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org