You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/08/05 03:21:56 UTC
[GitHub] [incubator-kyuubi] timothy65535 opened a new issue #894: Enhance kyuubi-zookeeper module
timothy65535 opened a new issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894
### 1. Describe the feature
Kyuubi use `TestingServer` as the EmbeddedZookeeper service in codebase, and require many args to init the service. These `electionPort`, `quorumPort`, `serverId` parameters only work in cluster mode, these ports are not bound to the host.
From hadoop and bookkeeper project, they use `ZooKeeperServer` as a built-in zk service, especially the bookeeper project.
- https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/server/services/MicroZookeeperService.java
- https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/shims/zk/ZooKeeperServerShimImpl.java
### 2. Describe the solution
Improve
- Remove `electionPort`, `quorumPort`, `serverId` which are nerver be used, hadoop and bookkeer do the same
- Replace `TestingServer` with `ZooKeeperServer`
- Remove `curator-test` dependency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
ulysses-you closed issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893196646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 edited a comment on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 edited a comment on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893201807
Do remove the testing server first.
> maybe an abstract service discover layer with adapters to K8s api-server, Zookeeper, Etcd make more sense to me.
Let's think about it, will let you know if have any ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893196646
Thanks for this proposal, replace `TestingServer` with `ZooKeeperServer` make sense to me.
> For Kyuubi quorum peer, it seems raft is a better choice(I not sure).
Do you mean implements a Raft component like KRaft in Kafka 3.0? Personally, I think Paxos or ZAB or Raft or Gossip is not really mattered here, maybe an abstract service discover layer with adapters to K8s `api-server`, `Zookeeper`, `Etcd` make more sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893135770
Removing TestingServer sounds good to me
> Remove electionPort, quorumPort, serverId which are nerver be used, hadoop and bookkeer do the same
Can we make each KyuubiServer served as a quorum peer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893238221
> There is a fork of Kyuubi which implemented K8sSessionManager and K8sSessionImpl, maybe we can learn something from it.
>
> https://zhuanlan.zhihu.com/p/345214051
> https://github.com/yilong2001/kyuubi
Nice !!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 edited a comment on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 edited a comment on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893201807
Do remove the testing server first.
> maybe an abstract service discover layer with adapters to K8s api-server, Zookeeper, Etcd make more sense to me.
Let's think about it, will let you know if have any ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893182251
Got @yaooqinn, do you have any ideas @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893138616
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893139151
Ok, we can keep optimizing this module when we find better solutions someday. we can remove the testing server here first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893201807
Will remove the testing server first.
> maybe an abstract service discover layer with adapters to K8s api-server, Zookeeper, Etcd make more sense to me.
Let's think about it, will let you know if have any ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893223166
There is a fork of Kyuubi which implemented K8sSessionManager and K8sSessionImpl, maybe we can learn something from it.
https://zhuanlan.zhihu.com/p/345214051
https://github.com/yilong2001/kyuubi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] timothy65535 commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
timothy65535 commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893138616
> Removing TestingServer sounds good to me
>
> > Remove electionPort, quorumPort, serverId which are nerver be used, hadoop and bookkeer do the same
>
> Can we make each KyuubiServer served as a quorum peer?
Maybe we don't need to do this, because `TestingServer` is not a full feature supported.
For zookeeper quorum peer, it is recommended to use a separate zookeeper cluster.
For kyuubi quorum peer, it seems raft is a better choice(I not sure).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #894: Enhance kyuubi-zookeeper module
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #894:
URL: https://github.com/apache/incubator-kyuubi/issues/894#issuecomment-893135770
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org