You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/05/24 05:20:55 UTC
[GitHub] [dolphinscheduler] guodongym opened a new issue, #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster
guodongym opened a new issue, #10219:
URL: https://github.com/apache/dolphinscheduler/issues/10219
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
EMR currently only supports creating clusters and adding steps, but in real use scenarios there are resident clusters waiting to receive new tasks, so you need to support adding steps to existing clusters
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie closed issue #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster
Posted by GitBox <gi...@apache.org>.
zhongjiajie closed issue #10219: [Feature][EMR] EMR supports use <add-Steps> to add steps to an existing cluster
URL: https://github.com/apache/dolphinscheduler/issues/10219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] guodongym commented on issue #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster
Posted by GitBox <gi...@apache.org>.
guodongym commented on issue #10219:
URL: https://github.com/apache/dolphinscheduler/issues/10219#issuecomment-1142329208
## ui
Add task type selection button, switch RunJobFlow/AddJobFlowSteps
When selecting RunJobFlow, keep the original interface
When selecting AddJobFlowSteps, add a text box to enter the cluster name, and enter the parameter Json of Steps. This API AWS natively requires JobFlowId. The reason why the cluster name is required is that the cluster may be terminated and rebuilt during use. At this time, the JobFlowId will be found If you want to change the scheduled task that has already been created, you need to convert it into JobFlowId through the cluster name backend, so that you only need to ensure that the cluster name does not change without re-updating the scheduling parameters. Or do not handle this situation, let the user maintain the JobFlowId through a global variable.
## backend
● Add the logic of switching specific tasks according to task types in org.apache.dolphinscheduler.plugin.task.emr.EmrTaskChannel#createTask
● Create org.apache.dolphinscheduler.plugin.task.emr.EmrTask if the task type is RunJobFlow
● If the task type is AddJobFlowSteps then create org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask
● Newly implemented EmrAddStepsTask logic to meet new requirements
The above is the initial plan that I thought of, and everyone is welcome to make suggestions for improvement.
--------------
前端
增加任务类型选择按钮，切换 RunJobFlow/ AddJobFlowSteps
选择 RunJobFlow 时，保持原有界面
选择 AddJobFlowSteps 时，增加文本框输入 集群名 , 并且输入 Steps 的参数Json，该 API AWS 原生是需要 JobFlowId 的，之所以要求输入集群名是由于集群使用过程中可能会终止，重建，这个时候 JobFlowId 会发现变化，如果想要发生变化时已建好的调度任务不受影响，则需要通过 集群名 后台进行转换成 JobFlowId ，这样只需要保证集群名 不发生变化就可以不用重新更新调度参数。或者不对这种情况做处理，让用户通过全局变量来维护JobFlowId。
后端
● 在 org.apache.dolphinscheduler.plugin.task.emr.EmrTaskChannel#createTask 中增加按照 任务类型切换具体 Task 的逻辑
● 如果任务类型为 RunJobFlow 则创建 org.apache.dolphinscheduler.plugin.task.emr.EmrTask
● 如果任务类型为AddJobFlowSteps 则创建 org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask
● 新实现EmrAddStepsTask 逻辑来满足新的需求
以上是我想到的初步方案，欢迎大家提出改进意见
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10219:
URL: https://github.com/apache/dolphinscheduler/issues/10219#issuecomment-1135422022
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10219:
URL: https://github.com/apache/dolphinscheduler/issues/10219#issuecomment-1146734633
That's great. Looking forward to your contribution. @guodongym
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org