[GitHub] [dolphinscheduler] guodongym commented on issue #10219: [Feature][EMR] EMR supports use to add steps to an existing cluster

[GitBox <gi...@apache.org>]

guodongym commented on issue #10219:
URL: https://github.com/apache/dolphinscheduler/issues/10219#issuecomment-1142329208

   ## ui
   Add task type selection button, switch RunJobFlow/AddJobFlowSteps
   When selecting RunJobFlow, keep the original interface
   When selecting AddJobFlowSteps, add a text box to enter the cluster name, and enter the parameter Json of Steps. This API AWS natively requires JobFlowId. The reason why the cluster name is required is that the cluster may be terminated and rebuilt during use. At this time, the JobFlowId will be found If you want to change the scheduled task that has already been created, you need to convert it into JobFlowId through the cluster name backend, so that you only need to ensure that the cluster name does not change without re-updating the scheduling parameters. Or do not handle this situation, let the user maintain the JobFlowId through a global variable.
   ## backend
   ● Add the logic of switching specific tasks according to task types in org.apache.dolphinscheduler.plugin.task.emr.EmrTaskChannel#createTask
   ● Create org.apache.dolphinscheduler.plugin.task.emr.EmrTask if the task type is RunJobFlow
   ● If the task type is AddJobFlowSteps then create org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask
   ● Newly implemented EmrAddStepsTask logic to meet new requirements
   
   The above is the initial plan that I thought of, and everyone is welcome to make suggestions for improvement.
   
   --------------
   前端
   增加任务类型选择按钮，切换  RunJobFlow/ AddJobFlowSteps
   选择 RunJobFlow 时，保持原有界面
   选择 AddJobFlowSteps 时，增加文本框输入 集群名 , 并且输入 Steps 的参数Json，该 API AWS 原生是需要 JobFlowId 的，之所以要求输入集群名是由于集群使用过程中可能会终止，重建，这个时候 JobFlowId 会发现变化，如果想要发生变化时已建好的调度任务不受影响，则需要通过 集群名 后台进行转换成 JobFlowId ，这样只需要保证集群名 不发生变化就可以不用重新更新调度参数。或者不对这种情况做处理，让用户通过全局变量来维护JobFlowId。
   后端
   ● 在 org.apache.dolphinscheduler.plugin.task.emr.EmrTaskChannel#createTask 中增加按照 任务类型切换具体 Task 的逻辑
   ● 如果任务类型为 RunJobFlow 则创建  org.apache.dolphinscheduler.plugin.task.emr.EmrTask
   ● 如果任务类型为AddJobFlowSteps 则创建 org.apache.dolphinscheduler.plugin.task.emr.EmrAddStepsTask
   ● 新实现EmrAddStepsTask 逻辑来满足新的需求
   
   以上是我想到的初步方案，欢迎大家提出改进意见


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org