You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/06/30 05:58:11 UTC
[GitHub] [dolphinscheduler] zhuxt2015 opened a new issue, #10690: [Feature][Api] Suggest support recommend timing configurations to users
zhuxt2015 opened a new issue, #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
When the user configuration completes the timing configuration, according to the timing configuration of all the workflows that have been online, estimate the number of tasks that will be started in minutes, if the task number threshold is exceeded, it is recommended that the user modify the timing configuration of the current workflow.
当用户配置完成定时配置时, 根据当前已经上线的所有工作流的定时配置, 计算出已经上线的工作流每分钟会被启动的task数量, 如果超出阈值, 则建议用户修改当前工作流的定时配置。
### Use case
Users often configure a large number of workflow' timing config at the same point in time, resulting in too concentrated task execute, high worker server load, which will lead worker server downtime
用户经常在同一时间点配置大量任务, 导致任务运行过于集中, 服务器负载过高, 出现服务器宕机问题
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuxt2015 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
zhuxt2015 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170810303
> Sounds pretty cool to me. If DS could provide users with suggestions on scheduled workflows / tasks number, it would definitely help.
>
> May I ask how to calculate the potential workload of all scheduled tasks to make the suggestions? Since the workload caused by different kinds of tasks could vary a lot.
only calculate the number of tasks, can not calculate workload of the task will be started
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuxt2015 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
zhuxt2015 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170843180
> > > Sounds pretty cool to me. If DS could provide users with suggestions on scheduled workflows / tasks number, it would definitely help.
> > > May I ask how to calculate the potential workload of all scheduled tasks to make the suggestions? Since the workload caused by different kinds of tasks could vary a lot.
> >
> >
> > only calculate the number of tasks, can not calculate workload of the task will be started
>
> For example, if DS could only handle 10 tasks at the same time and some user schedules 15 tasks, most of which take up few resources, it seems okay for the user to do so. How could we calculate the number to trade off between avoiding crashing the system and making the most use of it?
This can configure a weight for each type of task, then compare total task weight with the configured threshold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170822920
I think there are two sub-features in this issue:
- Firstly, calculate the number of tasks scheduled at the same time and show the statistics. This could be a good reminder for users.
- Secondly, give suggestions. This part could be a little bit complicated and need further discussions.
WDYT?@zhuxt2015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170814846
> > Sounds pretty cool to me. If DS could provide users with suggestions on scheduled workflows / tasks number, it would definitely help.
> > May I ask how to calculate the potential workload of all scheduled tasks to make the suggestions? Since the workload caused by different kinds of tasks could vary a lot.
>
> only calculate the number of tasks, can not calculate workload of the task will be started
For example, if DS could only handle 10 tasks at the same time and some user schedule 15 tasks, most of which take up little resources, it seems okay for the user to do so. How could we calculate the number to trade off between avoiding crashing the system and making the most use of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170807376
Sounds pretty cool to me. If DS could provide users with suggestions on schedule workflows / tasks number, it would definitely help.
May I ask how to calculate the potential workload of all scheduled tasks to make the suggestions? Since the workload caused by different kinds of tasks could vary a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170799078
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
When the user configuration completes the timing configuration, according to the timing configuration of all the workflows that have been online, estimate the number of tasks that will be started in minutes, if the task number threshold is exceeded, it is recommended that the user modify the timing configuration of the current workflow.
When the user configuration completes the timing configuration, according to the timing configuration of all the currently online workflows, the number of tasks that will be started per minute for the online workflow is calculated. If it exceeds the threshold, it is recommended that the user modify the timing configuration of the current workflow .
### Use case
Users often configure a large number of workflow' timing config at the same point in time, resulting in too concentrated task execute, high worker server load, which will lead worker server downtime
Users often configure a large number of tasks at the same time, resulting in too concentrated task operation, high server load, and server downtime problems
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170799233
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhuxt2015 commented on issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
Posted by GitBox <gi...@apache.org>.
zhuxt2015 commented on issue #10690:
URL: https://github.com/apache/dolphinscheduler/issues/10690#issuecomment-1170844658
> I think there are two sub-features in this issue:
>
> * Firstly, calculate the number of tasks scheduled at the same time and show the statistics. This could be a good reminder for users.
> * Secondly, give suggestions. This part could be a little bit complicated and need further discussions.
>
> WDYT?@zhuxt2015
Based on the task weight, I think the second one is also well solved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Api] Suggest support recommend timing configurations to users [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #10690: [Feature][Api] Suggest support recommend timing configurations to users
URL: https://github.com/apache/dolphinscheduler/issues/10690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org