You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/18 05:49:55 UTC
[GitHub] [dolphinscheduler] liqingwang opened a new issue, #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
liqingwang opened a new issue, #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Hi All,
As we talked, we want to support Spark on K8s at the Spark task plug-in.
### Use case
We change the Spark task plug-in and add a button in `Deploy Mode` named K8s, and when you chose this button will show some inputs about namespace and Docker images, etc. And change the background code to support Spark on the K8s cluster.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1101116947
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1301812329
Hi @dpengpeng , we already designed and finished spark/flink on k8s using operator mode, and we have the plan to contribute to the community about spark on k8s, and we want to know the attitude of the community how about we contribute the flink on k8s to the community? @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156134989
Looks like k8s task plugin was not included in 3.0.0-beta-1-release? https://github.com/apache/dolphinscheduler/tree/3.0.0-beta-1-release/dolphinscheduler-task-plugin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1304546846
> Hi @dpengpeng , we already designed and finished spark/flink on k8s using operator mode, and we have the plan to contribute to the community about spark on k8s, and we want to know the attitude of the community how about we contribute the flink on k8s to the community? @caishunfeng
Hi @liqingwang really good news if ds can support spark on k8s, looking forward to your pr.
And I found the issue of flink on k8s #10904, we can discuss in that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
EricGao888 commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1119412398
Hello @liqingwang, this feature is pretty cool and I'm quite looking forward to it. May I ask is there any update on this issue? Thx~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156252434
> Oh, I'm so sorry for that. I have checked the code and 3.0.0-beta-1 doesn't contain the K8s commit. If you want to try this feature, you can run the dev branch. And I don't know when to release this feature, maybe version 3.0.1?
ok, thanks for the answer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156147093
Oh, I'm so sorry for that. I have checked the code and 3.0.0-beta-1 doesn't contain the K8s commit. If you want to try this feature, you can run the dev branch. And I don't know when to release this feature, maybe version 3.0.1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1153074367
Hello @liqingwang do you have any plans to support flink on k8s? Our project is migrating k8s recently, but dolphin has no way to detect k8s programs, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1302899689
related issue: #10904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156128554
its the same with workflow, all of this, i am so sorry , i cant find
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156048717
The K8s task is in the `Task Definition` tab.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dpengpeng commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
dpengpeng commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1300636322
@liqingwang Hi, can your design about spark/flink on k8s support spark/flink Operator mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1293075068
Hi @caishunfeng , Yes We have finished the feature and now we are upgrading our platform to 3.1.0 and front part rewrite, after that we will plan to open a pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1156010854
> Hi @ikiler, we already support the Fink job on ds at the local version but have not yet pushed the code to the community. And my colleague from our team pushed the k8s task and k8s namespace to the community, you can see this pr. #8252, #9425, #9303. And at ds 3.0.0-beta1, you can see these two feature.
I am using ds 3.0.0-beta1, but not found any card with k8s or like , only mesos in sea tunnel and k8s namespace manager in Security center， Maybe I did something wrong? or next beta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1120108911
Hi @EricGao888, We have finished the design and the code. But we are doing secondary development locally, version 2.0.5, with `dolphinscheduler-ui`. Also the `dolphinscheduler-ui` [will not be maintained soon](#9896 ). We will adapt and upgrade local to version 3.0.0-alpha this month, so you can see the pr of this feature next month.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1305023023
Hi @caishunfeng , sure, my co-workers will discuss it in that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1293068607
Hi, @liqingwang is there any update of this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1294368315
Hi @caishunfeng , I remember that last meeting we talk about the spark job on k8s, and our solution is using the operation to manage the spark k8s job, and some people have some opinions about this piece, we want to know what's their attitude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1153386725
Hi @ikiler, we already support the Fink job on ds at the local version but have not yet pushed the code to the community. And my colleague from our team pushed the k8s task and k8s namespace to the community, you can see this pr. #8252, #9425, #9303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #9550: [Feature][plug-in] Add Spark-K8s task in Spark task plug-in
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #9550:
URL: https://github.com/apache/dolphinscheduler/issues/9550#issuecomment-1295845640
> Hi @caishunfeng , I remember that last meeting we talk about the spark job on k8s, and our solution is using the operation to manage the spark k8s job, and some people have some opinions about this piece, we want to know what's their attitude.
@liqingwang can you add some design detail of this feature, and send a discussion email to `dev@dolphinscheduler.apache.org`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org