You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/12 09:50:16 UTC
[GitHub] [dolphinscheduler] ChenpiDog opened a new issue, #10904: [Feature][Flink] Flink on kubernetes
ChenpiDog opened a new issue, #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
I hope the next official version can support the scheduling of Flink on kubernetes.
### Use case
I hope the next official version can support the scheduling of Flink on kubernetes.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182729737
Hi @ChenpiDog can you add some detail description of it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ChenpiDog commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ChenpiDog commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1264670033
Apache StreamPark (StreamX) and Dinky (DLink) seem to realize the management of Flink on Kubernetes, and their methods can be referred to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1304547973
> I'd like to join this discussion.
>
> # init
> 1. write Dockerfile to workspace and add extra data
> 2. pull flink base image
> 3. build job image with current workspace
> 4. push job image to the docker registry
>
> # handle
> 1. check kubernetes has the same app name
> 2. start flink application mode through `ShellCommandExecutor`
> 3. waiting for kubernetes deployment ready
> 4. listening kubernetes event and process(`logHandler` `errorHandler`...)
>
> # cancel
> 1. cancel flink job through the flink client(use `KubernetesClient` exec, so we don't have to care about `kubernetes.rest-service.exposed.type`.)
>
> FlinkParameters for kubernetes need add the parameter(`extJarLibs`) , becausewe must import libs into the docker image when building it.
I think the k8s task can support it, right? see https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/task/kubernetes.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1181575018
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] litiliu commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
litiliu commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1306582252
hi @caishunfeng @ikiler , this is the design of our implementation based on flink-kubernetes-operator.
# Background
We are using [Flink Kubernetes Operator](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/) to run Flink batch jar job on k8s cluster.
And all files in resource are stored on S3.
# Summary design
- In flink-task plugin's frontend, namespace & cluster & image & compute resources need to be specified by user to run this job.
- In the ds-worker, the CRD represents this batch job was built and submitted to k8s cluster by KubernetesClient.
- After that CRD has been submitted, a watcher is registered in ds-worker to monitor the status of that batch job(error or finish or timeout).
After getting the final status of that job, that watcher will be closed.
- The resources(jar or config files) are downloaded from S3 by the initContainer. Before the mainContainer run.
# Detail design
- In flink-task plugin's frontend, user need to specify the namespace & cluster & image in which to run that job.
- In the ds-master, the resources(jar or config files)'s absolute path on S3 will be calculated and set to context in the BaseTaskProcessor.getTaskExecutionContext()
- In the ds-worker, the CRD of [FlinkDeployment](https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/FlinkDeployment.java)
including the initContainer was built and submit to k8s cluster by KubernetesClient.
- After that CRD has been submitted, a watcher is registered in ds-worker to monitor the status of that batch job(error or finish). A CountDownLatch is used to monitor job timeout.
# limitation
- Because we are using the "error" filed to monitor job status，only flink version >= 1.15 are supported, because of a flink bug, see [status no clear when deploying batch job with flink-k8s-operator
](https://lists.apache.org/thread/f9wr2l55p48w0b2q2mmtpojqjo2vzstm)
Tks for discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] RuiFG commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
RuiFG commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305848880
@ikiler Yeah, I didn't see it in doc, maybe have to implement `KubernetesWatcher` on DS.
For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] litiliu commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
litiliu commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305097208
hi, I'm the colleague of @liqingwang, We have implemented using flink-operator to submit flink-batch job in our production. I will write the detail about it in the following days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305715406
> > Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng
>
> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator https://github.com/apache/flink-kubernetes-operator cool, DS build image, uses operator through `KubernetesClient`
Hi @RuiFG I checked the official documentation of operator, but I feel that the key problem is still unsolved, that is how to get his status after the batch task ends, the operator only provides how to manage a running task, but for the batch task to get his information after the end is important.
Or did I miss some points in doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1306500074
> @ikiler Yeah, I didn't see it in doc, maybe have to implement `KubernetesWatcher` on DS. For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng
Well, by the way our implementation, we use flink native api to submit task ，and no need deploy anything else in k8s ,maybe using the operator would be better to submit,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1304547528
> Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng
+1, the operator way is good, can you add some design detail for this way? @dpengpeng
cc @liqingwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] RuiFG commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
RuiFG commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305627592
hi @caishunfeng, I don't think k8s task can support it. `K8sTask` uses `KubernetesClient` to execute tasks, we have to do more to be compatible with flink task(eg: `flink-config Config Map` `Flink kubernetes HA`), and `K8sTaskExecutor` currently only supports batch job, for Flink k8s task, it is better to monitor `pod`(Flink TaskManager).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ChenpiDog commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ChenpiDog commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182749229
@caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182865241
>
I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this [application mode](https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/deployment/resource-providers/native_kubernetes/#application-mode) Pod creation, operation, and destruction are all operated by flink itself , like this:
```
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image=custom-image-name \
local:///opt/flink/usrlib/my-flink-job.jar
```
flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182942664
> > I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this [application mode](https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/deployment/resource-providers/native_kubernetes/#application-mode) Pod creation, operation, and destruction are all operated by flink itself , like this:
> > ```
> > ./bin/flink run-application \
> > --target kubernetes-application \
> > -Dkubernetes.cluster-id=my-first-application-cluster \
> > -Dkubernetes.container.image=custom-image-name \
> > local:///opt/flink/usrlib/my-flink-job.jar
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab
>
> I get your point, they are different way. So it's better to expand flink task to support the k8s deployment modes. @ikiler Are you interested in expanding flink task? like #9577
I'd love to if I could, but I have no good idea to solve it now. we are still using placing the api in the jar and using the history-server api to monitor the app in ds, which is obviously a bad practice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ChenpiDog commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ChenpiDog commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182751216
My Flink is deployed with kubernetes, and my ETL is Flink SQL. I expect to use the dolphinscheduler to schedule or manage my Flink SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dpengpeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
dpengpeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1302906787
Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. @caishunfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] RuiFG commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
RuiFG commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305658741
> Hi, @RuiFG Flink and Spark can submit application by Flink/Spark Operator, and this way is so native k8s. In my opinion,I think we maybe clould use Operator mode,not ShellCommandExecutor. In this way,we can submit an application by a yaml file @caishunfeng
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
https://github.com/apache/flink-kubernetes-operator
cool, DS build image, uses operator through `KubernetesClient`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1306498366
> @ikiler Yeah, I didn't see it in doc, maybe have to implement `KubernetesWatcher` on DS. For DS, open source operators maybe have more management troubles. gcp spark k8s operators and flink kubernetes operators API are still beta, and they both have to deploy in k8s to make the task run, what did you think? @dpengpeng
Well, by the way our implementation, we use flink native api to submit task ，and no need deploy anything else in k8s ,maybe using the operator would be better to submit,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ikiler commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
ikiler commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182872954
> > @caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.
>
> Now ds has k8s task type in dev, I'm not sure it's what you need, but you can have a try.
I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this [application mode](https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/deployment/resource-providers/native_kubernetes/#application-mode) Pod creation, operation, and destruction are all operated by flink itself , like this:
```
./bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-first-application-cluster \
-Dkubernetes.container.image=custom-image-name \
local:///opt/flink/usrlib/my-flink-job.jar
```
flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182822544
> @caishunfeng For the task type of Flink, the dolphinscheduler supports scheduling Flink deployed in local mode or yarn mode, but it does not seem to support scheduling Flink deployed on kubernetes. I can't find the relevant instructions on the official website.
Now ds has k8s task type in dev, I'm not sure it's what you need, but you can have a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1182896358
> I build the dev , flink still uses yarn by default in application mode , I find the k8s task , but cannot be applied to flink , you can look at this [application mode](https://nightlies.apache.org/flink/flink-docs-release-1.15/zh/docs/deployment/resource-providers/native_kubernetes/#application-mode) Pod creation, operation, and destruction are all operated by flink itself , like this:
>
> ```
> ./bin/flink run-application \
> --target kubernetes-application \
> -Dkubernetes.cluster-id=my-first-application-cluster \
> -Dkubernetes.container.image=custom-image-name \
> local:///opt/flink/usrlib/my-flink-job.jar
> ```
>
> flink will create jobmanager, taskmanager, and copy config .........does many things, So ,simply specifying a image for a k8s task cannot solve the problem. we don't have any path to run a flink task through the k8s tab
I get your point, they are different way. So it's better to expand flink task to support the k8s deployment modes.
@ikiler Are you interested in expanding flink task? like #9577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] RuiFG commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
RuiFG commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1297970125
I'd like to join this discussion.
# init
1. write Dockerfile to workspace and add extra data
2. pull flink base image
3. build job image with current workspace
4. push job image to the docker registry
# handle
1. check kubernetes has the same app name
2. start flink application mode through `ShellCommandExecutor`
3. waiting for kubernetes deployment ready
4. listening kubernetes event and process(`logHandler` `errorHandler`...)
# cancel
1. cancel flink job through the flink client(use kubernetesClient exec, so we don't have to care about `kubernetes.rest-service.exposed.type`.)
FlinkParameters for kubernetes need add the parameter(`extJarLibs`) , becausewe must import libs into the docker image when building it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] liqingwang commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
liqingwang commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1305021343
cc @litiliu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Nicholaswang commented on issue #10904: [Feature][Flink] Flink on kubernetes
Posted by GitBox <gi...@apache.org>.
Nicholaswang commented on issue #10904:
URL: https://github.com/apache/dolphinscheduler/issues/10904#issuecomment-1347733037
Hi, any update for submitting spark tasks to kubernetes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org