You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/07/01 07:02:40 UTC
[GitHub] [dolphinscheduler] Jiezhi opened a new issue #5731: [Feature][service] Submit actions to YARN like Oozie
Jiezhi opened a new issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731
**Describe the feature**
DS submit job to YARN like Oozie.
**Describe the solution you'd like**
> Oozie workflow actions start jobs in remote systems (i.e. Hadoop, Pig). Upon action completion, the remote systems callback Oozie to notify the action completion, at this point Oozie proceeds to the next action in the workflow.
Besides Sqoop or other MR jobs, Oozie would start jobs like normal Java program (not MR job) directly to YARN. Oozie supplys a Yarn ApplicationMaster layer to its jobs. In other words, any jobs running on Oozie would be a YARN application.
But DS run jobs just in local way (please correct me if I'm wrong)，a normal Java program would run in DS worker node, and the node resource is hard to manage like YARN.
May be DS can provide this option when start a job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
Jiezhi commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-871993213
> The worker just calls the task client to trigger the task, you can search for AbstractYarnTask, which is usually submitted to yarn. k8s will be supported later.
>
> Or, did I misunderstand what you meant?
Maybe AbstractYarnTask can solve my problem, does that mean I need to develop the plugin myself according to [plugin-development](https://dolphinscheduler.apache.org/en-us/development/plugin-development.html).
If there's no official plugin(which is this issue request), I can spend some time on this plugin development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi closed issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
Jiezhi closed issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-871986905
The worker just calls the task client to trigger the task, you can search for AbstractYarnTask, which is usually submitted to yarn. k8s will be supported later.
Or, did I misunderstand what you meant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-871980880
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-871995148
> > The worker just calls the task client to trigger the task, you can search for AbstractYarnTask, which is usually submitted to yarn. k8s will be supported later.
> > Or, did I misunderstand what you meant?
>
> Maybe AbstractYarnTask can solve my problem, does that mean I need to develop the plugin myself according to [plugin-development](https://dolphinscheduler.apache.org/en-us/development/plugin-development.html).
>
> If there's no official plugin(which is this issue request), I can spend some time on this plugin development.
What type of tasks do you need to submit to yarn?
Can't the existing task types fit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
Jiezhi commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-872008853
Ok, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] CalvinKirs commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-872002654
This you need to implement yourself, there is currently no Java type of task.
Later we will change to SPI form, so that it is easier to extend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Jiezhi commented on issue #5731: [Feature][service] Submit actions to YARN like Oozie
Posted by GitBox <gi...@apache.org>.
Jiezhi commented on issue #5731:
URL: https://github.com/apache/dolphinscheduler/issues/5731#issuecomment-872000238
> > > The worker just calls the task client to trigger the task, you can search for AbstractYarnTask, which is usually submitted to yarn. k8s will be supported later.
> > > Or, did I misunderstand what you meant?
> >
> >
> > Maybe AbstractYarnTask can solve my problem, does that mean I need to develop the plugin myself according to [plugin-development](https://dolphinscheduler.apache.org/en-us/development/plugin-development.html).
> > If there's no official plugin(which is this issue request), I can spend some time on this plugin development.
>
> What type of tasks do you need to submit to yarn?
> Can't the existing task types fit?
Just a normal Java task, say a helloworld program.
Because we used to deploy tasks to oozie, and some of our jobs are pure Java programs. And we are migrating Oozie workflow to DS now, while awkwardly found DS cannot submit those jobs to YARN directly like Oozie.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org