You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Jordan Zimmerman <jo...@jordanzimmerman.com> on 2019/11/27 18:18:24 UTC
Re: Any interest in a gRPC version of ZooKeeper
FYI
We have an open discussion regarding replacing Jute, using gRPC and related things in this sub channel on the ASF Slack board. All are welcome to join in:
https://the-asf.slack.com/archives/CQKS7A3FT <https://the-asf.slack.com/archives/CQKS7A3FT>
-Jordan
> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> Hi Folks,
>
> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>
> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>
> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>
> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>
> Issues:
> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
> I'm not sure how things like SASL and reconfig would work with gRPC
>
> -Jordan
Re: Any interest in a gRPC version of ZooKeeper
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I’ll ping Infra and see what they say
> On Nov 28, 2019, at 8:04 AM, Enrico Olivelli <eo...@gmail.com> wrote:
>
> Il gio 28 nov 2019, 14:02 Andor Molnar <an...@apache.org> ha scritto:
>
>> Unfortunately non-committers are having some difficulties joining that
>> channel.
>>
>> For example:
>> "@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can
>> invite them to a public channel.”
>>
>> Does anyone have an idea what does that mean?
>>
>
> No.
> But I am sure that we can invite people.
> Some ASF projects have a link on the website (Beam?)
>
> Enrico
>
>
>
>> Andor
>>
>>
>>
>>> On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com>
>> wrote:
>>>
>>> FYI
>>>
>>> We have an open discussion regarding replacing Jute, using gRPC and
>> related things in this sub channel on the ASF Slack board. All are welcome
>> to join in:
>>>
>>> https://the-asf.slack.com/archives/CQKS7A3FT <
>> https://the-asf.slack.com/archives/CQKS7A3FT>
>>>
>>> -Jordan
>>>
>>>> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>>>
>>>> Hi Folks,
>>>>
>>>> I've written a proof of concept implementation of a ServerCnxnFactory
>> that implements gRPC. The goal is to make it possible to easily write
>> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
>> able to write a Golang client easily. What's the interest level of
>> something like this? Let's discuss if it's worth pursuing. I'd be willing
>> to move this from proof of concept to production but I'll need help (1 or 2
>> co-developers).
>>>>
>>>> If you want to try it, I've pushed the Golang client and some
>> instructions here (let me know if you have any issues - I'm a go neophyte).
>> Note: "zookeeper/test.go" is the interesting file:
>>>>
>>>> https://github.com/Randgalt/zkgrpc <
>> https://github.com/Randgalt/zkgrpc>
>>>>
>>>> Here's the proof of concept on the ZK server side (the interesting
>> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
>> RpcZooKeeperServer.java and zookeeper.proto):
>>>>
>>>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>>>
>>>> Issues:
>>>> Writing a client, even with gRPC, will require some work. Sessions have
>> to be maintained, watchers have to be maintained, etc.
>>>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
>> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
>> only records could be used so that the gRPC connection doesn't have to keep
>> marshalling/unmarshalling byte buffers
>>>> I don't know enough about the gRPC client/server implementations to
>> know if it will meet the needs of ZooKeeper. Anyone have experience here?
>>>> I haven't completely thought through how much work it will take to
>> write useful clients. As I've shown with the proof of concept simple ZK
>> CRUD db operations work well. I need to spend time writing a recipe such as
>> Leader Election to see how much work is required.
>>>> I'm not sure how things like SASL and reconfig would work with gRPC
>>>>
>>>> -Jordan
>>>
>>
>>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Enrico Olivelli <eo...@gmail.com>.
Il gio 28 nov 2019, 14:02 Andor Molnar <an...@apache.org> ha scritto:
> Unfortunately non-committers are having some difficulties joining that
> channel.
>
> For example:
> "@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can
> invite them to a public channel.”
>
> Does anyone have an idea what does that mean?
>
No.
But I am sure that we can invite people.
Some ASF projects have a link on the website (Beam?)
Enrico
> Andor
>
>
>
> > On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com>
> wrote:
> >
> > FYI
> >
> > We have an open discussion regarding replacing Jute, using gRPC and
> related things in this sub channel on the ASF Slack board. All are welcome
> to join in:
> >
> > https://the-asf.slack.com/archives/CQKS7A3FT <
> https://the-asf.slack.com/archives/CQKS7A3FT>
> >
> > -Jordan
> >
> >> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>
> >> Hi Folks,
> >>
> >> I've written a proof of concept implementation of a ServerCnxnFactory
> that implements gRPC. The goal is to make it possible to easily write
> ZooKeeper clients in non-JVM languages. Using the proof of concept I was
> able to write a Golang client easily. What's the interest level of
> something like this? Let's discuss if it's worth pursuing. I'd be willing
> to move this from proof of concept to production but I'll need help (1 or 2
> co-developers).
> >>
> >> If you want to try it, I've pushed the Golang client and some
> instructions here (let me know if you have any issues - I'm a go neophyte).
> Note: "zookeeper/test.go" is the interesting file:
> >>
> >> https://github.com/Randgalt/zkgrpc <
> https://github.com/Randgalt/zkgrpc>
> >>
> >> Here's the proof of concept on the ZK server side (the interesting
> files are RpcServerCnxn.java, RpcServerCnxnFactory.java,
> RpcZooKeeperServer.java and zookeeper.proto):
> >>
> >>
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <
> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
> >>
> >> Issues:
> >> Writing a client, even with gRPC, will require some work. Sessions have
> to be maintained, watchers have to be maintained, etc.
> >> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept
> has to emulate Jute byte buffers. Ideally, this will be abstracted so that
> only records could be used so that the gRPC connection doesn't have to keep
> marshalling/unmarshalling byte buffers
> >> I don't know enough about the gRPC client/server implementations to
> know if it will meet the needs of ZooKeeper. Anyone have experience here?
> >> I haven't completely thought through how much work it will take to
> write useful clients. As I've shown with the proof of concept simple ZK
> CRUD db operations work well. I need to spend time writing a recipe such as
> Leader Election to see how much work is required.
> >> I'm not sure how things like SASL and reconfig would work with gRPC
> >>
> >> -Jordan
> >
>
>
Re: Any interest in a gRPC version of ZooKeeper
Posted by Andor Molnar <an...@apache.org>.
Unfortunately non-committers are having some difficulties joining that channel.
For example:
"@Mate Szalay-Beko is a Multi-Channel Guest. Only your Workspace Admin can invite them to a public channel.”
Does anyone have an idea what does that mean?
Andor
> On 2019. Nov 27., at 19:18, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> FYI
>
> We have an open discussion regarding replacing Jute, using gRPC and related things in this sub channel on the ASF Slack board. All are welcome to join in:
>
> https://the-asf.slack.com/archives/CQKS7A3FT <https://the-asf.slack.com/archives/CQKS7A3FT>
>
> -Jordan
>
>> On Nov 18, 2019, at 9:25 AM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>
>> Hi Folks,
>>
>> I've written a proof of concept implementation of a ServerCnxnFactory that implements gRPC. The goal is to make it possible to easily write ZooKeeper clients in non-JVM languages. Using the proof of concept I was able to write a Golang client easily. What's the interest level of something like this? Let's discuss if it's worth pursuing. I'd be willing to move this from proof of concept to production but I'll need help (1 or 2 co-developers).
>>
>> If you want to try it, I've pushed the Golang client and some instructions here (let me know if you have any issues - I'm a go neophyte). Note: "zookeeper/test.go" is the interesting file:
>>
>> https://github.com/Randgalt/zkgrpc <https://github.com/Randgalt/zkgrpc>
>>
>> Here's the proof of concept on the ZK server side (the interesting files are RpcServerCnxn.java, RpcServerCnxnFactory.java, RpcZooKeeperServer.java and zookeeper.proto):
>>
>> https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc <https://github.com/apache/zookeeper/compare/master...Randgalt:wip-grpc>
>>
>> Issues:
>> Writing a client, even with gRPC, will require some work. Sessions have to be maintained, watchers have to be maintained, etc.
>> Currently, Jute is deeply embedded in ZooKeeper. The proof of concept has to emulate Jute byte buffers. Ideally, this will be abstracted so that only records could be used so that the gRPC connection doesn't have to keep marshalling/unmarshalling byte buffers
>> I don't know enough about the gRPC client/server implementations to know if it will meet the needs of ZooKeeper. Anyone have experience here?
>> I haven't completely thought through how much work it will take to write useful clients. As I've shown with the proof of concept simple ZK CRUD db operations work well. I need to spend time writing a recipe such as Leader Election to see how much work is required.
>> I'm not sure how things like SASL and reconfig would work with gRPC
>>
>> -Jordan
>