You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/01/08 12:06:22 UTC
[GitHub] [dolphinscheduler] leo-lao opened a new issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
leo-lao opened a new issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Hi, Community members, I am curious about how is the schedule for the new registry on k8s environment,
I see you have discussed a lot on mails, but have not seen your implementations till now.
It is truly of great value if we are able to replace zookeeper with internal registry mechanism inside kubernetes.
I am interested in joining in this task, and have read similar design on `Spring Cloud Kubernetes` .
### Use case
_No response_
### Related issues
[mail discussion](https://lists.apache.org/thread/xb0zodqh81cfomkslkyb4t12l1tdyk15)
[closed issue one year ago](https://github.com/apache/dolphinscheduler/issues/3961)
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] edited a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
github-actions[bot] edited a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1001189371
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007971510
@dailidong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao closed issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao closed issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao edited a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao edited a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007978112
Recently I made the design for k8s registry:
## Objective
add one new registry for DS, and make it free of zookeeper dependency on k8s environments.
this registry should only be used when DS running on k8s
## Core Implementations
dependency: https://github.com/fabric8io/kubernetes-client
distributed lock: [fabric lock](https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/resourcelock
)
<table>
<tr>
<td> item
<td>zookeeper registry
<td>k8s registry
<tr>
<td> how to register self
<td> create one new node certain path
<td> do nothing, pods on ks8 environments will be discovered automatically
<tr>
<td> how to discover otherss
<td> use CuratorFramework client to connect and watch zk cluster
<td> use fabric client to connect and watch k8s cluster
<tr>
<td> how to implements distributed lock
<td> InterProcessMutex in curator project
<td> lock in fabric, according to [stack overflow](https://stackoverflow.com/questions/34409160/kubernetes-getting-cluster-locks-over-a-replication
)
<tr>
<td> how to implements HeartBeatTask
<td> periodically persist ephemeral node on specific zk path
<td> do nothing, K8s will check pod`s liveness automatically
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1003034523
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007978112
Recently I made the design for k8s registry:
## Objective
add one new registry for DS, and make it free of zookeeper dependency on k8s environments.
this registry should only be used when DS running on k8s
## Core Implementations
<table>
<tr>
<td> item
<td>zookeeper registry
<td>k8s registry
<tr>
<td> how to register self
<td> create one new node certain path
<td> do nothing, pods on ks8 environments will be discovered automatically
<tr>
<td> how to discover otherss
<td> use CuratorFramework client to connect and watch zk cluster
<td> use fabric client to connect and watch k8s cluster
<tr>
<td> how to implements distributed lock
<td> InterProcessMutex in curator project
<td> [lock in fabric](https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/resourcelock)
<tr>
<td> how to implements HeartBeatTask
<td> periodically persist ephemeral node on specific zk path
<td> do nothing, K8s will check pod`s liveness automatically
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1001189371
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao removed a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao removed a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007971510
@dailidong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1001189372
@CalvinKirs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao edited a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao edited a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1003034523
>
OK. I will try to make this recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao edited a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao edited a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007978112
Recently I made the design for k8s registry:
## Objective
add one new registry for DS, and make it free of zookeeper dependency on k8s environments.
this registry should only be used when DS running on k8s
## Core Implementations
dependency: https://github.com/fabric8io/kubernetes-client
distributed lock: https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/resourcelock
<table>
<tr>
<td> item
<td>zookeeper registry
<td>k8s registry
<tr>
<td> how to register self
<td> create one new node certain path
<td> do nothing, pods on ks8 environments will be discovered automatically
<tr>
<td> how to discover otherss
<td> use CuratorFramework client to connect and watch zk cluster
<td> use fabric client to connect and watch k8s cluster
<tr>
<td> how to implements distributed lock
<td> InterProcessMutex in curator project
<td> lock in fabric, according to https://stackoverflow.com/questions/34409160/kubernetes-getting-cluster-locks-over-a-replication
<tr>
<td> how to implements HeartBeatTask
<td> periodically persist ephemeral node on specific zk path
<td> do nothing, K8s will check pod`s liveness automatically
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong commented on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1003010345
@leo-lao I have assigned this task to you, please contact me if you meet any problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] leo-lao edited a comment on issue #7633: [Feature][Registry] How is the schedule for the new registry on k8s environment
Posted by GitBox <gi...@apache.org>.
leo-lao edited a comment on issue #7633:
URL: https://github.com/apache/dolphinscheduler/issues/7633#issuecomment-1007978112
Recently I made the design for k8s registry:
## Objective
add one new registry for DS, and make it free of zookeeper dependency on k8s environments.
this registry should only be used when DS running on k8s
## Core Implementations
dependency: https://github.com/fabric8io/kubernetes-client
distributed lock: [fabric lock](https://github.com/fabric8io/kubernetes-client/tree/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/extended/leaderelection/resourcelock
)
<table>
<tr>
<td> item
<td>zookeeper registry
<td>k8s registry
<tr>
<td> how to register self
<td> create one new node certain path
<td> do nothing, pods on ks8 environments will be discovered automatically
<tr>
<td> how to discover otherss
<td> use CuratorFramework client to connect and watch zk cluster
<td> use fabric client to connect and watch k8s cluster
<tr>
<td> how to implements distributed lock
<td> InterProcessMutex in curator project
<td> lock in fabric, according to [stack overflow](https://stackoverflow.com/questions/34409160/kubernetes-getting-cluster-locks-over-a-replication
)
<tr>
<td> how to implements HeartBeatTask
<td> periodically persist ephemeral node on specific zk path
<td> do nothing, K8s will check pod`s liveness automatically
</table>
@dailidong is there any one able to review this design for me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org