You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jordan Zimmerman <jo...@jordanzimmerman.com> on 2019/11/18 14:25:01 UTC
Any interest in a gRPC version of ZooKeeper
Hi Folks,
I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
Issues:
Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
I'm not sure how things like SASL and reconfig would work with gRPC
-Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I’ll ping Infra and see what they say
> On Nov 28, 2019, at 8:04 AM, Enrico Olivelli <eo...@gmail.com> wrote:
>
> Il gio 28 nov 2019, 14:02 Andor Molnar <an...@apache.org> ha scritto:
>
>> Unfortunately non-committers are having some difficulties joining that
>> channel.
>>
>> For example:
>> "@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can
>> invite them to a public channel.”
>>
>> Does anyone have an idea what does that mean?
>>
>
> No.
> But I am sure that we can invite people.
> Some ASF projects have a link on the website (Beam?)
>
> Enrico
>
>
>
>> Andor
>>
>>
>>
>>> On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com>
>> wrote:
>>>
>>> FYI
>>>
>>> We have an open discussion regarding replacing Jute, using gRPC and
>> related things in this sub channel on the ASF Slack board. All are welcome
>> to join in:
>>>
>>> https://the-asf.slack.com/archives/CQKS7A3FT <
>> https://the-asf.slack.com/archives/CQKS7A3FT>
>>>
>>> -Jordan
>>>
>>>> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>>>
>>>> Hi Folks,
>>>>
>>>> I've written a proof of concept implementation of a ServerCnxnFactory
>> that implements gRPC. The goal is to make it possible to easily write
>> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
>> able to write a Golang client easily. What's the interest level of
>> something like this? Let's discuss if it's worth pursuing. I'd be willing
>> to move this from proof of concept to production but I'll need help (1 or 2
>> co-developers).
>>>>
>>>> If you want to try it, I've pushed the Golang client and some
>> instructions here (let me know if you have any issues - I'm a go neophyte).
>> Note: "zookeeper/test.go" is the interesting file:
>>>>
>>>> https://github.com/Randgalt/zkgrpc <
>> https://github.com/Randgalt/zkgrpc>
>>>>
>>>> Here's the proof of concept on the ZK server side (the interesting
>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
>> RpcZooKeeperServer.java and zookeeper.proto):
>>>>
>>>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>>>
>>>> Issues:
>>>> Writing a client, even with gRPC, will require some work. Sessions have
>> to be maintained, watchers have to be maintained, etc.
>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
>> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
>> only records could be used so that the gRPC connection doesn't have to keep
>> marshalling/unmarshalling byte buffers
>>>> I don't know enough about the gRPC client/server implementations to
>> know if it will meet the needs of ZooKeeper. Anyone have experience here?
>>>> I haven't completely thought through how much work it will take to
>> write useful clients. As I've shown with the proof of concept simple ZK
>> CRUD db operations work well. I need to spend time writing a recipe such as
>> Leader Election to see how much work is required.
>>>> I'm not sure how things like SASL and reconfig would work with gRPC
>>>>
>>>> -Jordan
>>>
>>
>>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Enrico Olivelli <eo...@gmail.com>.
Il gio 28 nov 2019, 14:02 Andor Molnar <an...@apache.org> ha scritto:
> Unfortunately non-committers are having some difficulties joining that
> channel.
>
> For example:
> "@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can
> invite them to a public channel.”
>
> Does anyone have an idea what does that mean?
>
No.
But I am sure that we can invite people.
Some ASF projects have a link on the website (Beam?)
Enrico
> Andor
>
>
>
> > On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com>
> wrote:
> >
> > FYI
> >
> > We have an open discussion regarding replacing Jute, using gRPC and
> related things in this sub channel on the ASF Slack board. All are welcome
> to join in:
> >
> > https://the-asf.slack.com/archives/CQKS7A3FT <
> https://the-asf.slack.com/archives/CQKS7A3FT>
> >
> > -Jordan
> >
> >> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>
> >> Hi Folks,
> >>
> >> I've written a proof of concept implementation of a ServerCnxnFactory
> that implements gRPC. The goal is to make it possible to easily write
> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
> able to write a Golang client easily. What's the interest level of
> something like this? Let's discuss if it's worth pursuing. I'd be willing
> to move this from proof of concept to production but I'll need help (1 or 2
> co-developers).
> >>
> >> If you want to try it, I've pushed the Golang client and some
> instructions here (let me know if you have any issues - I'm a go neophyte).
> Note: "zookeeper/test.go" is the interesting file:
> >>
> >> https://github.com/Randgalt/zkgrpc <
> https://github.com/Randgalt/zkgrpc>
> >>
> >> Here's the proof of concept on the ZK server side (the interesting
> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> RpcZooKeeperServer.java and zookeeper.proto):
> >>
> >>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
> >>
> >> Issues:
> >> Writing a client, even with gRPC, will require some work. Sessions have
> to be maintained, watchers have to be maintained, etc.
> >> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
> only records could be used so that the gRPC connection doesn't have to keep
> marshalling/unmarshalling byte buffers
> >> I don't know enough about the gRPC client/server implementations to
> know if it will meet the needs of ZooKeeper. Anyone have experience here?
> >> I haven't completely thought through how much work it will take to
> write useful clients. As I've shown with the proof of concept simple ZK
> CRUD db operations work well. I need to spend time writing a recipe such as
> Leader Election to see how much work is required.
> >> I'm not sure how things like SASL and reconfig would work with gRPC
> >>
> >> -Jordan
> >
>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Andor Molnar <an...@apache.org>.
Unfortunately non-committers are having some difficulties joining that channel.
For example:
"@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can invite them to a public channel.”
Does anyone have an idea what does that mean?
Andor
> On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> FYI
>
> We have an open discussion regarding replacing Jute, using gRPC and related things in this sub channel on the ASF Slack board. All are welcome to join in:
>
> https://the-asf.slack.com/archives/CQKS7A3FT <https://the-asf.slack.com/archives/CQKS7A3FT>
>
> -Jordan
>
>> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>
>> Hi Folks,
>>
>> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>>
>> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>>
>> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>>
>> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>
>> Issues:
>> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
>> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
>> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
>> I'm not sure how things like SASL and reconfig would work with gRPC
>>
>> -Jordan
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
FYI
We have an open discussion regarding replacing Jute, using gRPC and related things in this sub channel on the ASF Slack board. All are welcome to join in:
https://the-asf.slack.com/archives/CQKS7A3FT <https://the-asf.slack.com/archives/CQKS7A3FT>
-Jordan
> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> Hi Folks,
>
> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>
> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>
> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>
> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> Issues:
> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
> I'm not sure how things like SASL and reconfig would work with gRPC
>
> -Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Michael Han <ha...@apache.org>.
>> Or, am I missing something you're seeing?
No, you were right. I misunderstood. Thanks for clarification!
On Sun, Nov 24, 2019 at 5:33 PM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> > It's still
> > difficult to write a client because there needs to be language specific
> > implementations for the rpc mapper that maps jute Record to protobuf.
>
> No. I'm sorry if it isn't clear. RecordMapper and all related classes
> (everything in this package:
> https://github.com/Randgalt/zookeeper/tree/wip-grpc/zookeeper-grpc/src/main/java/org/apache/zookeeper/grpc
> <
> https://github.com/Randgalt/zookeeper/tree/wip-grpc/zookeeper-grpc/src/main/java/org/apache/zookeeper/grpc>)
> is meant to be part of ZooKeeper server itself (ignore ClientTester and
> ServerTester - those are just dummy test classes). Clients do not need to
> implement this. The mapping between protobuf data structures and ZooKeeper
> internal datastructures will all be inside of ZooKeeper itself.
>
> Have a look at the quickie Go implementation I did. No mapping here. Just
> straight protobuf and gRPC.
>
> https://github.com/Randgalt/zkgrpc/tree/master/zookeeper <
> https://github.com/Randgalt/zkgrpc/tree/master/zookeeper>
>
> Or, am I missing something you're seeing?
>
> -Jordan
>
> > On Nov 24, 2019, at 4:40 PM, Michael Han <ha...@apache.org> wrote:
> >
> >>> That's 100% protobuf/gRPC
> >
> > Yes agree. Sorry, I should be probably more clear. What I meant
> > "serialization format" in this case is jute's Record, which is still used
> > in the POC code base. The wire serialization format is protobuf, and it's
> > converted to / from Record through the rpc mapper utility. It's still
> > difficult to write a client because there needs to be language specific
> > implementations for the rpc mapper that maps jute Record to protobuf.
> >
> > On Thu, Nov 21, 2019 at 6:56 AM Jordan Zimmerman <
> jordan@jordanzimmerman.com>
> > wrote:
> >
> >>> The proof of concept is still using jute as serialization format, which
> >>> makes write a client library harder. Use protobuf as serialization
> format
> >>> might achieve this goal the marshal / unmarshal code can be generated
> >>> automatically.
> >>
> >> No it doesn't. Have you seen "zookeeper.proto"? That's 100%
> protobuf/gRPC.
> >>
> >> -Jordan
> >>
>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
> It's still
> difficult to write a client because there needs to be language specific
> implementations for the rpc mapper that maps jute Record to protobuf.
No. I'm sorry if it isn't clear. RecordMapper and all related classes (everything in this package: https://github.com/Randgalt/zookeeper/tree/wip-grpc/zookeeper-grpc/src/main/java/org/apache/zookeeper/grpc <https://github.com/Randgalt/zookeeper/tree/wip-grpc/zookeeper-grpc/src/main/java/org/apache/zookeeper/grpc>) is meant to be part of ZooKeeper server itself (ignore ClientTester and ServerTester - those are just dummy test classes). Clients do not need to implement this. The mapping between protobuf data structures and ZooKeeper internal datastructures will all be inside of ZooKeeper itself.
Have a look at the quickie Go implementation I did. No mapping here. Just straight protobuf and gRPC.
https://github.com/Randgalt/zkgrpc/tree/master/zookeeper <https://github.com/Randgalt/zkgrpc/tree/master/zookeeper>
Or, am I missing something you're seeing?
-Jordan
> On Nov 24, 2019, at 4:40 PM, Michael Han <ha...@apache.org> wrote:
>
>>> That's 100% protobuf/gRPC
>
> Yes agree. Sorry, I should be probably more clear. What I meant
> "serialization format" in this case is jute's Record, which is still used
> in the POC code base. The wire serialization format is protobuf, and it's
> converted to / from Record through the rpc mapper utility. It's still
> difficult to write a client because there needs to be language specific
> implementations for the rpc mapper that maps jute Record to protobuf.
>
> On Thu, Nov 21, 2019 at 6:56 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
> wrote:
>
>>> The proof of concept is still using jute as serialization format, which
>>> makes write a client library harder. Use protobuf as serialization format
>>> might achieve this goal the marshal / unmarshal code can be generated
>>> automatically.
>>
>> No it doesn't. Have you seen "zookeeper.proto"? That's 100% protobuf/gRPC.
>>
>> -Jordan
>>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Michael Han <ha...@apache.org>.
>> That's 100% protobuf/gRPC
Yes agree. Sorry, I should be probably more clear. What I meant
"serialization format" in this case is jute's Record, which is still used
in the POC code base. The wire serialization format is protobuf, and it's
converted to / from Record through the rpc mapper utility. It's still
difficult to write a client because there needs to be language specific
implementations for the rpc mapper that maps jute Record to protobuf.
On Thu, Nov 21, 2019 at 6:56 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> > The proof of concept is still using jute as serialization format, which
> > makes write a client library harder. Use protobuf as serialization format
> > might achieve this goal the marshal / unmarshal code can be generated
> > automatically.
>
> No it doesn't. Have you seen "zookeeper.proto"? That's 100% protobuf/gRPC.
>
> -Jordan
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
> The proof of concept is still using jute as serialization format, which
> makes write a client library harder. Use protobuf as serialization format
> might achieve this goal the marshal / unmarshal code can be generated
> automatically.
No it doesn't. Have you seen "zookeeper.proto"? That's 100% protobuf/gRPC.
-Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Michael Han <ha...@apache.org>.
>> The goal is to make it possible to easily write ZooKeeper clients in
non-JVM languages.
The proof of concept is still using jute as serialization format, which
makes write a client library harder. Use protobuf as serialization format
might achieve this goal the marshal / unmarshal code can be generated
automatically.
>> I don't know enough about the gRPC client/server implementations to know
if it will meet the needs of ZooKeeper.
We have done a similar project at Twitter where we added Finagle (Twitter's
RPC library) network connection in addition to NIO and Netty server cnx
supported by ZooKeeper. The motivation was to utilize the advanced feature
such as load balancing and traffic control built in Finagle to improve
ZooKeeper's reliability and performance. Our implementation is similar to
this gRPC POC as both only use the RPC layer of the library, but still use
jute as serialization format (as opposed to protobuf for grpc and Thrift
for Finagle.). So technically, yes, this could be done. Though our solution
worked to certain extent but we decided to continue use NIO/Netty for
production and build load balancing / traffic control on top of it and take
patches from FB/upstream.
>> Then the question becomes what else do we fix at the same time?
It's a good point, if we are free of backward compatibility concerns, then
there are quite a few things to experiment. One idea is to get rid of jute,
the other is to have a new way of session management - for example decouple
session state with connection state.
On Mon, Nov 18, 2019 at 4:40 PM tison <wa...@gmail.com> wrote:
> Interesting, at a first glance I thought it is a client wrapper for
> polyglot communicating and
> has nothing to do with ZOOKEEPER-102. Will take a look at the JIRA since
> you attach
> a plan of the whole story.
>
> Best,
> tison.
>
>
> Jordan Zimmerman <jo...@jordanzimmerman.com> 于2019年11月19日周二 上午5:23写道：
>
> > We can resurrect https://issues.apache.org/jira/browse/ZOOKEEPER-102
> >
> > > On Nov 18, 2019, at 4:15 PM, Jonathan Wong <chi.jonathanwong@gmail.com
> >
> > wrote:
> > >
> > > I’d be willing to offer some of my spare time towards this effort.
> > >
> > > Jonathan Wong
> > >
> > >> On Nov 18, 2019, at 1:04 PM, Jordan Zimmerman <
> > jordan@jordanzimmerman.com> wrote:
> > >>
> > >> A fresh look at APIs is definitely in order. So, this would be a
> > potential ZK 4.x.
> > >>
> > >>> That said, we added things like rest in the past for similar reasons
> > and it
> > >>> never really took off... Would be a shame to see the same here.
> > >>
> > >> ZK has had a lot more activity with recent committers. So, maybe this
> > time will be different? If at least one other person will sign up to work
> > on it, I'll shepherd it.
> > >>
> > >> -JZ
> > >>
> > >>> On Nov 18, 2019, at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
> > >>>
> > >>> There are quite a few benefits to using grpc imo. It's come up a few
> > times
> > >>> where I've been part of the discussion - ala we make it b/w compat it
> > would
> > >>> be a good move imo. Then the question becomes what else do we fix at
> > the
> > >>> same time? e.g. make version fields 64 bit rather than 32? etc...
> > there are
> > >>> a bunch of zk4 such jiras that could be addressed at the same time
> (and
> > >>> likely in a b/w compat way - ie zk3)
> > >>>
> > >>> That said, we added things like rest in the past for similar reasons
> > and it
> > >>> never really took off... Would be a shame to see the same here.
> > >>>
> > >>> Patrick
> > >>>
> > >>> On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <
> > jordan@jordanzimmerman.com>
> > >>> wrote:
> > >>>
> > >>>>> That looks like great work. In order to address the issues, why not
> > >>>> build on top of curator (https://curator.apache.org)?
> > >>>>
> > >>>> (Note: I'm the main author of Curator). I'd definitely try to make
> > >>>> something like Curator for gRPC. I'm not sure exactly what that
> means
> > at
> > >>>> this point. But, my main goal is to enable non-JVM clients. We have
> C,
> > >>>> python and few others now but they always lag with changes and are
> > hard to
> > >>>> maintain.
> > >>>>
> > >>>> -JZ
> > >>>>
> > >>>>>> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com>
> > wrote:
> > >>>>>>
> > >>>>>> That looks like great work. In order to address the issues, why
> not
> > >>>> build on top of curator (https://curator.apache.org)?
> > >>>>>
> > >>>>> I could support in case question rise with SASL, but I am not sure
> > yet
> > >>>> if I find the time to actively develop for this unfortunately
> > >>>>>
> > >>>>>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
> > >>>> jordan@jordanzimmerman.com>:
> > >>>>>>
> > >>>>>> Hi Folks,
> > >>>>>>
> > >>>>>> I've written a proof of concept implementation of a
> > ServerCnxnFactory
> > >>>> that implements gRPC. The goal is to make it possible to easily
> write
> > >>>> ZooKeeper clients in non-JVM languages. Using the proof of concept I
> > was
> > >>>> able to write a Golang client easily. What's the interest level of
> > >>>> something like this? Let's discuss if it's worth pursuing. I'd be
> > willing
> > >>>> to move this from proof of concept to production but I'll need help
> > (1 or 2
> > >>>> co-developers).
> > >>>>>>
> > >>>>>> If you want to try it, I've pushed the Golang client and some
> > >>>> instructions here (let me know if you have any issues - I'm a go
> > neophyte).
> > >>>> Note: "zookeeper/test.go" is the interesting file:
> > >>>>>>
> > >>>>>> https://github.com/Randgalt/zkgrpc <
> > >>>> https://github.com/Randgalt/zkgrpc>
> > >>>>>>
> > >>>>>> Here's the proof of concept on the ZK server side (the interesting
> > >>>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> > >>>> RpcZooKeeperServer.java and zookeeper.proto):
> > >>>>>>
> > >>>>>>
> > >>>>
> > https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> > >>>>
> > https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
> > >>>>
> > >>>>>>
> > >>>>>> Issues:
> > >>>>>> Writing a client, even with gRPC, will require some work. Sessions
> > have
> > >>>> to be maintained, watchers have to be maintained, etc.
> > >>>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of
> > concept
> > >>>> has to emulate Jute byte buffers. Ideally, this will be abstracted
> so
> > that
> > >>>> only records could be used so that the gRPC connection doesn't have
> > to keep
> > >>>> marshalling/unmarshalling byte buffers
> > >>>>>> I don't know enough about the gRPC client/server implementations
> to
> > >>>> know if it will meet the needs of ZooKeeper. Anyone have experience
> > here?
> > >>>>>> I haven't completely thought through how much work it will take to
> > >>>> write useful clients. As I've shown with the proof of concept simple
> > ZK
> > >>>> CRUD db operations work well. I need to spend time writing a recipe
> > such as
> > >>>> Leader Election to see how much work is required.
> > >>>>>> I'm not sure how things like SASL and reconfig would work with
> gRPC
> > >>>>>>
> > >>>>>> -Jordan
> > >>>>
> > >>>>
> > >>
> >
> >
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by tison <wa...@gmail.com>.
Interesting, at a first glance I thought it is a client wrapper for
polyglot communicating and
has nothing to do with ZOOKEEPER-102. Will take a look at the JIRA since
you attach
a plan of the whole story.
Best,
tison.
Jordan Zimmerman <jo...@jordanzimmerman.com> 于2019年11月19日周二 上午5:23写道：
> We can resurrect https://issues.apache.org/jira/browse/ZOOKEEPER-102
>
> > On Nov 18, 2019, at 4:15 PM, Jonathan Wong <ch...@gmail.com>
> wrote:
> >
> > I’d be willing to offer some of my spare time towards this effort.
> >
> > Jonathan Wong
> >
> >> On Nov 18, 2019, at 1:04 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>
> >> A fresh look at APIs is definitely in order. So, this would be a
> potential ZK 4.x.
> >>
> >>> That said, we added things like rest in the past for similar reasons
> and it
> >>> never really took off... Would be a shame to see the same here.
> >>
> >> ZK has had a lot more activity with recent committers. So, maybe this
> time will be different? If at least one other person will sign up to work
> on it, I'll shepherd it.
> >>
> >> -JZ
> >>
> >>> On Nov 18, 2019, at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
> >>>
> >>> There are quite a few benefits to using grpc imo. It's come up a few
> times
> >>> where I've been part of the discussion - ala we make it b/w compat it
> would
> >>> be a good move imo. Then the question becomes what else do we fix at
> the
> >>> same time? e.g. make version fields 64 bit rather than 32? etc...
> there are
> >>> a bunch of zk4 such jiras that could be addressed at the same time (and
> >>> likely in a b/w compat way - ie zk3)
> >>>
> >>> That said, we added things like rest in the past for similar reasons
> and it
> >>> never really took off... Would be a shame to see the same here.
> >>>
> >>> Patrick
> >>>
> >>> On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <
> jordan@jordanzimmerman.com>
> >>> wrote:
> >>>
> >>>>> That looks like great work. In order to address the issues, why not
> >>>> build on top of curator (https://curator.apache.org)?
> >>>>
> >>>> (Note: I'm the main author of Curator). I'd definitely try to make
> >>>> something like Curator for gRPC. I'm not sure exactly what that means
> at
> >>>> this point. But, my main goal is to enable non-JVM clients. We have C,
> >>>> python and few others now but they always lag with changes and are
> hard to
> >>>> maintain.
> >>>>
> >>>> -JZ
> >>>>
> >>>>>> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> That looks like great work. In order to address the issues, why not
> >>>> build on top of curator (https://curator.apache.org)?
> >>>>>
> >>>>> I could support in case question rise with SASL, but I am not sure
> yet
> >>>> if I find the time to actively develop for this unfortunately
> >>>>>
> >>>>>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
> >>>> jordan@jordanzimmerman.com>:
> >>>>>>
> >>>>>> Hi Folks,
> >>>>>>
> >>>>>> I've written a proof of concept implementation of a
> ServerCnxnFactory
> >>>> that implements gRPC. The goal is to make it possible to easily write
> >>>> ZooKeeper clients in non-JVM languages. Using the proof of concept I
> was
> >>>> able to write a Golang client easily. What's the interest level of
> >>>> something like this? Let's discuss if it's worth pursuing. I'd be
> willing
> >>>> to move this from proof of concept to production but I'll need help
> (1 or 2
> >>>> co-developers).
> >>>>>>
> >>>>>> If you want to try it, I've pushed the Golang client and some
> >>>> instructions here (let me know if you have any issues - I'm a go
> neophyte).
> >>>> Note: "zookeeper/test.go" is the interesting file:
> >>>>>>
> >>>>>> https://github.com/Randgalt/zkgrpc <
> >>>> https://github.com/Randgalt/zkgrpc>
> >>>>>>
> >>>>>> Here's the proof of concept on the ZK server side (the interesting
> >>>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> >>>> RpcZooKeeperServer.java and zookeeper.proto):
> >>>>>>
> >>>>>>
> >>>>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> >>>>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
> >>>>
> >>>>>>
> >>>>>> Issues:
> >>>>>> Writing a client, even with gRPC, will require some work. Sessions
> have
> >>>> to be maintained, watchers have to be maintained, etc.
> >>>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of
> concept
> >>>> has to emulate Jute byte buffers. Ideally, this will be abstracted so
> that
> >>>> only records could be used so that the gRPC connection doesn't have
> to keep
> >>>> marshalling/unmarshalling byte buffers
> >>>>>> I don't know enough about the gRPC client/server implementations to
> >>>> know if it will meet the needs of ZooKeeper. Anyone have experience
> here?
> >>>>>> I haven't completely thought through how much work it will take to
> >>>> write useful clients. As I've shown with the proof of concept simple
> ZK
> >>>> CRUD db operations work well. I need to spend time writing a recipe
> such as
> >>>> Leader Election to see how much work is required.
> >>>>>> I'm not sure how things like SASL and reconfig would work with gRPC
> >>>>>>
> >>>>>> -Jordan
> >>>>
> >>>>
> >>
>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
We can resurrect https://issues.apache.org/jira/browse/ZOOKEEPER-102
> On Nov 18, 2019, at 4:15 PM, Jonathan Wong <ch...@gmail.com> wrote:
>
> I’d be willing to offer some of my spare time towards this effort.
>
> Jonathan Wong
>
>> On Nov 18, 2019, at 1:04 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>
>> A fresh look at APIs is definitely in order. So, this would be a potential ZK 4.x.
>>
>>> That said, we added things like rest in the past for similar reasons and it
>>> never really took off... Would be a shame to see the same here.
>>
>> ZK has had a lot more activity with recent committers. So, maybe this time will be different? If at least one other person will sign up to work on it, I'll shepherd it.
>>
>> -JZ
>>
>>> On Nov 18, 2019, at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
>>>
>>> There are quite a few benefits to using grpc imo. It's come up a few times
>>> where I've been part of the discussion - ala we make it b/w compat it would
>>> be a good move imo. Then the question becomes what else do we fix at the
>>> same time? e.g. make version fields 64 bit rather than 32? etc... there are
>>> a bunch of zk4 such jiras that could be addressed at the same time (and
>>> likely in a b/w compat way - ie zk3)
>>>
>>> That said, we added things like rest in the past for similar reasons and it
>>> never really took off... Would be a shame to see the same here.
>>>
>>> Patrick
>>>
>>> On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
>>> wrote:
>>>
>>>>> That looks like great work. In order to address the issues, why not
>>>> build on top of curator (https://curator.apache.org)?
>>>>
>>>> (Note: I'm the main author of Curator). I'd definitely try to make
>>>> something like Curator for gRPC. I'm not sure exactly what that means at
>>>> this point. But, my main goal is to enable non-JVM clients. We have C,
>>>> python and few others now but they always lag with changes and are hard to
>>>> maintain.
>>>>
>>>> -JZ
>>>>
>>>>>> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com> wrote:
>>>>>>
>>>>>> That looks like great work. In order to address the issues, why not
>>>> build on top of curator (https://curator.apache.org)?
>>>>>
>>>>> I could support in case question rise with SASL, but I am not sure yet
>>>> if I find the time to actively develop for this unfortunately
>>>>>
>>>>>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
>>>> jordan@jordanzimmerman.com>:
>>>>>>
>>>>>> Hi Folks,
>>>>>>
>>>>>> I've written a proof of concept implementation of a ServerCnxnFactory
>>>> that implements gRPC. The goal is to make it possible to easily write
>>>> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
>>>> able to write a Golang client easily. What's the interest level of
>>>> something like this? Let's discuss if it's worth pursuing. I'd be willing
>>>> to move this from proof of concept to production but I'll need help (1 or 2
>>>> co-developers).
>>>>>>
>>>>>> If you want to try it, I've pushed the Golang client and some
>>>> instructions here (let me know if you have any issues - I'm a go neophyte).
>>>> Note: "zookeeper/test.go" is the interesting file:
>>>>>>
>>>>>> https://github.com/Randgalt/zkgrpc <
>>>> https://github.com/Randgalt/zkgrpc>
>>>>>>
>>>>>> Here's the proof of concept on the ZK server side (the interesting
>>>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
>>>> RpcZooKeeperServer.java and zookeeper.proto):
>>>>>>
>>>>>>
>>>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
>>>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>>>
>>>>>>
>>>>>> Issues:
>>>>>> Writing a client, even with gRPC, will require some work. Sessions have
>>>> to be maintained, watchers have to be maintained, etc.
>>>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
>>>> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
>>>> only records could be used so that the gRPC connection doesn't have to keep
>>>> marshalling/unmarshalling byte buffers
>>>>>> I don't know enough about the gRPC client/server implementations to
>>>> know if it will meet the needs of ZooKeeper. Anyone have experience here?
>>>>>> I haven't completely thought through how much work it will take to
>>>> write useful clients. As I've shown with the proof of concept simple ZK
>>>> CRUD db operations work well. I need to spend time writing a recipe such as
>>>> Leader Election to see how much work is required.
>>>>>> I'm not sure how things like SASL and reconfig would work with gRPC
>>>>>>
>>>>>> -Jordan
>>>>
>>>>
>>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jonathan Wong <ch...@gmail.com>.
I’d be willing to offer some of my spare time towards this effort.
Jonathan Wong
> On Nov 18, 2019, at 1:04 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> A fresh look at APIs is definitely in order. So, this would be a potential ZK 4.x.
>
>> That said, we added things like rest in the past for similar reasons and it
>> never really took off... Would be a shame to see the same here.
>
> ZK has had a lot more activity with recent committers. So, maybe this time will be different? If at least one other person will sign up to work on it, I'll shepherd it.
>
> -JZ
>
>> On Nov 18, 2019, at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> There are quite a few benefits to using grpc imo. It's come up a few times
>> where I've been part of the discussion - ala we make it b/w compat it would
>> be a good move imo. Then the question becomes what else do we fix at the
>> same time? e.g. make version fields 64 bit rather than 32? etc... there are
>> a bunch of zk4 such jiras that could be addressed at the same time (and
>> likely in a b/w compat way - ie zk3)
>>
>> That said, we added things like rest in the past for similar reasons and it
>> never really took off... Would be a shame to see the same here.
>>
>> Patrick
>>
>> On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
>> wrote:
>>
>>>> That looks like great work. In order to address the issues, why not
>>> build on top of curator (https://curator.apache.org)?
>>>
>>> (Note: I'm the main author of Curator). I'd definitely try to make
>>> something like Curator for gRPC. I'm not sure exactly what that means at
>>> this point. But, my main goal is to enable non-JVM clients. We have C,
>>> python and few others now but they always lag with changes and are hard to
>>> maintain.
>>>
>>> -JZ
>>>
>>>>> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com> wrote:
>>>>>
>>>>> That looks like great work. In order to address the issues, why not
>>> build on top of curator (https://curator.apache.org)?
>>>>
>>>> I could support in case question rise with SASL, but I am not sure yet
>>> if I find the time to actively develop for this unfortunately
>>>>
>>>>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
>>> jordan@jordanzimmerman.com>:
>>>>>
>>>>> Hi Folks,
>>>>>
>>>>> I've written a proof of concept implementation of a ServerCnxnFactory
>>> that implements gRPC. The goal is to make it possible to easily write
>>> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
>>> able to write a Golang client easily. What's the interest level of
>>> something like this? Let's discuss if it's worth pursuing. I'd be willing
>>> to move this from proof of concept to production but I'll need help (1 or 2
>>> co-developers).
>>>>>
>>>>> If you want to try it, I've pushed the Golang client and some
>>> instructions here (let me know if you have any issues - I'm a go neophyte).
>>> Note: "zookeeper/test.go" is the interesting file:
>>>>>
>>>>> https://github.com/Randgalt/zkgrpc <
>>> https://github.com/Randgalt/zkgrpc>
>>>>>
>>>>> Here's the proof of concept on the ZK server side (the interesting
>>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
>>> RpcZooKeeperServer.java and zookeeper.proto):
>>>>>
>>>>>
>>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
>>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>>
>>>>>
>>>>> Issues:
>>>>> Writing a client, even with gRPC, will require some work. Sessions have
>>> to be maintained, watchers have to be maintained, etc.
>>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
>>> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
>>> only records could be used so that the gRPC connection doesn't have to keep
>>> marshalling/unmarshalling byte buffers
>>>>> I don't know enough about the gRPC client/server implementations to
>>> know if it will meet the needs of ZooKeeper. Anyone have experience here?
>>>>> I haven't completely thought through how much work it will take to
>>> write useful clients. As I've shown with the proof of concept simple ZK
>>> CRUD db operations work well. I need to spend time writing a recipe such as
>>> Leader Election to see how much work is required.
>>>>> I'm not sure how things like SASL and reconfig would work with gRPC
>>>>>
>>>>> -Jordan
>>>
>>>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
A fresh look at APIs is definitely in order. So, this would be a potential ZK 4.x.
> That said, we added things like rest in the past for similar reasons and it
> never really took off... Would be a shame to see the same here.
ZK has had a lot more activity with recent committers. So, maybe this time will be different? If at least one other person will sign up to work on it, I'll shepherd it.
-JZ
> On Nov 18, 2019, at 12:22 PM, Patrick Hunt <ph...@apache.org> wrote:
>
> There are quite a few benefits to using grpc imo. It's come up a few times
> where I've been part of the discussion - ala we make it b/w compat it would
> be a good move imo. Then the question becomes what else do we fix at the
> same time? e.g. make version fields 64 bit rather than 32? etc... there are
> a bunch of zk4 such jiras that could be addressed at the same time (and
> likely in a b/w compat way - ie zk3)
>
> That said, we added things like rest in the past for similar reasons and it
> never really took off... Would be a shame to see the same here.
>
> Patrick
>
> On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
> wrote:
>
>>> That looks like great work. In order to address the issues, why not
>> build on top of curator (https://curator.apache.org)?
>>
>> (Note: I'm the main author of Curator). I'd definitely try to make
>> something like Curator for gRPC. I'm not sure exactly what that means at
>> this point. But, my main goal is to enable non-JVM clients. We have C,
>> python and few others now but they always lag with changes and are hard to
>> maintain.
>>
>> -JZ
>>
>>> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com> wrote:
>>>
>>> That looks like great work. In order to address the issues, why not
>> build on top of curator (https://curator.apache.org)?
>>>
>>> I could support in case question rise with SASL, but I am not sure yet
>> if I find the time to actively develop for this unfortunately
>>>
>>>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
>> jordan@jordanzimmerman.com>:
>>>>
>>>> Hi Folks,
>>>>
>>>> I've written a proof of concept implementation of a ServerCnxnFactory
>> that implements gRPC. The goal is to make it possible to easily write
>> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
>> able to write a Golang client easily. What's the interest level of
>> something like this? Let's discuss if it's worth pursuing. I'd be willing
>> to move this from proof of concept to production but I'll need help (1 or 2
>> co-developers).
>>>>
>>>> If you want to try it, I've pushed the Golang client and some
>> instructions here (let me know if you have any issues - I'm a go neophyte).
>> Note: "zookeeper/test.go" is the interesting file:
>>>>
>>>> https://github.com/Randgalt/zkgrpc <
>> https://github.com/Randgalt/zkgrpc>
>>>>
>>>> Here's the proof of concept on the ZK server side (the interesting
>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
>> RpcZooKeeperServer.java and zookeeper.proto):
>>>>
>>>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>
>>>>
>>>> Issues:
>>>> Writing a client, even with gRPC, will require some work. Sessions have
>> to be maintained, watchers have to be maintained, etc.
>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
>> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
>> only records could be used so that the gRPC connection doesn't have to keep
>> marshalling/unmarshalling byte buffers
>>>> I don't know enough about the gRPC client/server implementations to
>> know if it will meet the needs of ZooKeeper. Anyone have experience here?
>>>> I haven't completely thought through how much work it will take to
>> write useful clients. As I've shown with the proof of concept simple ZK
>> CRUD db operations work well. I need to spend time writing a recipe such as
>> Leader Election to see how much work is required.
>>>> I'm not sure how things like SASL and reconfig would work with gRPC
>>>>
>>>> -Jordan
>>
>>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Patrick Hunt <ph...@apache.org>.
There are quite a few benefits to using grpc imo. It's come up a few times
where I've been part of the discussion - ala we make it b/w compat it would
be a good move imo. Then the question becomes what else do we fix at the
same time? e.g. make version fields 64 bit rather than 32? etc... there are
a bunch of zk4 such jiras that could be addressed at the same time (and
likely in a b/w compat way - ie zk3)
That said, we added things like rest in the past for similar reasons and it
never really took off... Would be a shame to see the same here.
Patrick
On Mon, Nov 18, 2019 at 6:48 AM Jordan Zimmerman <jo...@jordanzimmerman.com>
wrote:
> > That looks like great work. In order to address the issues, why not
> build on top of curator (https://curator.apache.org)?
>
> (Note: I'm the main author of Curator). I'd definitely try to make
> something like Curator for gRPC. I'm not sure exactly what that means at
> this point. But, my main goal is to enable non-JVM clients. We have C,
> python and few others now but they always lag with changes and are hard to
> maintain.
>
> -JZ
>
> > On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com> wrote:
> >
> > That looks like great work. In order to address the issues, why not
> build on top of curator (https://curator.apache.org)?
> >
> > I could support in case question rise with SASL, but I am not sure yet
> if I find the time to actively develop for this unfortunately
> >
> >> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <
> jordan@jordanzimmerman.com>:
> >>
> >> Hi Folks,
> >>
> >> I've written a proof of concept implementation of a ServerCnxnFactory
> that implements gRPC. The goal is to make it possible to easily write
> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
> able to write a Golang client easily. What's the interest level of
> something like this? Let's discuss if it's worth pursuing. I'd be willing
> to move this from proof of concept to production but I'll need help (1 or 2
> co-developers).
> >>
> >> If you want to try it, I've pushed the Golang client and some
> instructions here (let me know if you have any issues - I'm a go neophyte).
> Note: "zookeeper/test.go" is the interesting file:
> >>
> >> https://github.com/Randgalt/zkgrpc <
> https://github.com/Randgalt/zkgrpc>
> >>
> >> Here's the proof of concept on the ZK server side (the interesting
> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> RpcZooKeeperServer.java and zookeeper.proto):
> >>
> >>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> >>
> >> Issues:
> >> Writing a client, even with gRPC, will require some work. Sessions have
> to be maintained, watchers have to be maintained, etc.
> >> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
> only records could be used so that the gRPC connection doesn't have to keep
> marshalling/unmarshalling byte buffers
> >> I don't know enough about the gRPC client/server implementations to
> know if it will meet the needs of ZooKeeper. Anyone have experience here?
> >> I haven't completely thought through how much work it will take to
> write useful clients. As I've shown with the proof of concept simple ZK
> CRUD db operations work well. I need to spend time writing a recipe such as
> Leader Election to see how much work is required.
> >> I'm not sure how things like SASL and reconfig would work with gRPC
> >>
> >> -Jordan
>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
> That looks like great work. In order to address the issues, why not build on top of curator (https://curator.apache.org)?
(Note: I'm the main author of Curator). I'd definitely try to make something like Curator for gRPC. I'm not sure exactly what that means at this point. But, my main goal is to enable non-JVM clients. We have C, python and few others now but they always lag with changes and are hard to maintain.
-JZ
> On Nov 18, 2019, at 9:45 AM, Jörn Franke <jo...@gmail.com> wrote:
>
> That looks like great work. In order to address the issues, why not build on top of curator (https://curator.apache.org)?
>
> I could support in case question rise with SASL, but I am not sure yet if I find the time to actively develop for this unfortunately
>
>> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <jo...@jordanzimmerman.com>:
>>
>> Hi Folks,
>>
>> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>>
>> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>>
>> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>>
>> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>
>> Issues:
>> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
>> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
>> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
>> I'm not sure how things like SASL and reconfig would work with gRPC
>>
>> -Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jörn Franke <jo...@gmail.com>.
That looks like great work. In order to address the issues, why not build on top of curator (https://curator.apache.org)?
I could support in case question rise with SASL, but I am not sure yet if I find the time to actively develop for this unfortunately
> Am 18.11.2019 um 15:25 schrieb Jordan Zimmerman <jo...@jordanzimmerman.com>:
>
> Hi Folks,
>
> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>
> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>
> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>
> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> Issues:
> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
> I'm not sure how things like SASL and reconfig would work with gRPC
>
> -Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
FYI
We have an open discussion regarding replacing Jute, using gRPC and related things in this sub channel on the ASF Slack board. All are welcome to join in:
https://the-asf.slack.com/archives/CQKS7A3FT <https://the-asf.slack.com/archives/CQKS7A3FT>
-Jordan
> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> Hi Folks,
>
> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>
> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>
> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>
> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> Issues:
> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
> I'm not sure how things like SASL and reconfig would work with gRPC
>
> -Jordan