You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Matthias Amberg <ma...@unblu.com> on 2016/09/27 09:07:27 UTC
Feature Proposal: Embedded (non network stack) connection option for
embedded Zookeeper
Dear Zookeeper developers,
I have a feature request for allowing an embedded Zookeeper be run and
connected to without using any system networking stack. All
client-server communications would be handled internally of a single
runtime.
We at Unblu offer our product as a cloud service (orchestrated by
Zookeeper) as well as an on-premises installable product.
In the latter scenario we use a single embedded version of Zookeeper to
orchestrate all the different embedded Version.
This setup allows us to use the same mechanisms for service discovery and
synchronization, whether we run in a distributed cloud setup or
a single on-premises node.
For this use case we have implemented versions of ServerCnxn,
ServerCnxnFactory and ClientCnxnSocket that handle client-server
connections internally without
using any server networking stack.
I would very much love to commit a patch that implements this feature or
open a jira issue to discuss this further, if there is interest from the
Zookeeper community for such a feature.
We think this feature offers several advantages over running Zookeeper
locally or embedded via a networking stack:
- less system resource usage (no need for additional networking overhead in
what could be systems that already have a high load of networking IO)
- simpler configuration and potentially better security: no need to
configure any ports and network addresses, firewalls, loopback interfaces
etc. (Our software runs in banking environments where security constraints
often makes it
difficult to (even) have loopback connections)
- simpler Continuous Integration: Different Test Runs of our builds
constantly interfere with each other due to ports being already in use and
so forth.
Additionally as this feature simply would be an extension to the current
code base that wouldn't even be instantiated in the traditional server
setup there would be almost no interference with the current code base and
hence no threat
to Zookeeper stability in traditional setups.
It would be great if you could provide your thoughts and feedback on this
and whether this is a welcome feature, in which case I would create a Jira
Issue for further discussion and patch submission.
Best regards,
Matthias Amberg
Developer
Office: ++41 511 11 64
unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
unblu makes websites personal http://www.unblu.com
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Giorgio Zoppi <gi...@gmail.com>.
I would like to join this challange as well rewriting just the protocol
layer in C++. A simple implementation is Jzab.
On Fri, Sep 30, 2016 at 2:13 PM, Matthias Amberg <ma...@unblu.com>
wrote:
> Hi Enrico,
>
> thanks for your response, sorry for the late reply.
> Would you mind sharing some details of your upcoming implementation, do you
> think it will cover our use case of using it for an embedded Zookeeper
> instance for a production system?
> If yes, we'll continue to use our own implementation until this becomes
> available in Zookeeper, do you have an ETA?
>
> Best regards,
>
> Matthias
>
>
> Matthias Amberg
> Developer
> Office: ++41 511 11 64
>
> unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>
> unblu makes websites personal http://www.unblu.com
>
> Please consider the environment before printing this email.
>
> This e-mail contains including its attachments confidential protected
> information by law. If you are not the intended recipient, please contact
> the sender and delete this message. Any unauthorised copying of this
> message or unauthorised distribution of the information contained herein is
> prohibited.
>
> Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
> geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind,
> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail. Das
> unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
> Informationen ist nicht gestattet.
>
> On 27 September 2016 at 12:40, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi Matthias,
> > I am working on a proposal on this topic. My idea is to use local netty
> > features.
> > I will share some code.
> > Beware that this usage of zookeeper is to be adopted only if needed. My
> > case is for instance to run unit tests without a real network.
> > Maybe you should consider to abstract your zookkeper access layer
> >
> > Enrico
> >
> > Il mar 27 set 2016, 11:07 Matthias Amberg <ma...@unblu.com> ha
> > scritto:
> >
> > > Dear Zookeeper developers,
> > >
> > > I have a feature request for allowing an embedded Zookeeper be run and
> > > connected to without using any system networking stack. All
> > > client-server communications would be handled internally of a single
> > > runtime.
> > > We at Unblu offer our product as a cloud service (orchestrated by
> > > Zookeeper) as well as an on-premises installable product.
> > > In the latter scenario we use a single embedded version of Zookeeper to
> > > orchestrate all the different embedded Version.
> > > This setup allows us to use the same mechanisms for service discovery
> and
> > > synchronization, whether we run in a distributed cloud setup or
> > > a single on-premises node.
> > >
> > > For this use case we have implemented versions of ServerCnxn,
> > > ServerCnxnFactory and ClientCnxnSocket that handle client-server
> > > connections internally without
> > > using any server networking stack.
> > >
> > > I would very much love to commit a patch that implements this feature
> or
> > > open a jira issue to discuss this further, if there is interest from
> the
> > > Zookeeper community for such a feature.
> > >
> > > We think this feature offers several advantages over running Zookeeper
> > > locally or embedded via a networking stack:
> > > - less system resource usage (no need for additional networking
> overhead
> > in
> > > what could be systems that already have a high load of networking IO)
> > > - simpler configuration and potentially better security: no need to
> > > configure any ports and network addresses, firewalls, loopback
> interfaces
> > > etc. (Our software runs in banking environments where security
> > constraints
> > > often makes it
> > > difficult to (even) have loopback connections)
> > > - simpler Continuous Integration: Different Test Runs of our builds
> > > constantly interfere with each other due to ports being already in use
> > and
> > > so forth.
> > >
> > > Additionally as this feature simply would be an extension to the
> current
> > > code base that wouldn't even be instantiated in the traditional server
> > > setup there would be almost no interference with the current code base
> > and
> > > hence no threat
> > > to Zookeeper stability in traditional setups.
> > >
> > > It would be great if you could provide your thoughts and feedback on
> this
> > > and whether this is a welcome feature, in which case I would create a
> > Jira
> > > Issue for further discussion and patch submission.
> > >
> > > Best regards,
> > >
> > > Matthias Amberg
> > > Developer
> > > Office: ++41 511 11 64
> > >
> > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
> > >
> > > unblu makes websites personal http://www.unblu.com
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Enrico Olivelli <eo...@gmail.com>.
Here I'm back on the topic
I have created this issue
https://issues.apache.org/jira/browse/ZOOKEEPER-2755 Allow to subclass
ClientCnxnSocketNetty and NettyServerCnxn in order to use Netty Local
transport
and submitted a patch
https://github.com/apache/zookeeper/pull/227
My idea is not to provide an official ZooKeeper network less version at
this moment but to allow developers to easly extend/subclass ZooKeeper
networking in a more future-proof way.
As ZooKeeper 3.5 onwards uses Netty it is possibile to use it to have a
network-less zookeeper setup
The implementation will be very "easy" using Netty,
look at this test case:
https://github.com/eolivelli/zookeeper/blob/ZOOKEEPER-2755-localaddress/src/java/test/org/apache/zookeeper/test/NetworklessTest.java
I hope that helps
-- Enrico
2017-01-09 15:33 GMT+01:00 Enrico Olivelli <eo...@gmail.com>:
> Hi Matthias,
> sorry for late reply. I've taken a look to your code.
> Personally I would prefer to use Netty directly, but I think that if your
> implementation is working you should try to file a PR in GitHub and create
> a JIRA
>
> did you have any other answer from the ZK community ?
>
> -- Enrico
>
>
>
>
>
>
>
> 2016-10-04 16:22 GMT+02:00 Matthias Amberg <ma...@unblu.com>:
>
>> Hi Enrico,
>>
>> I set up a github repo with just the 'embedded' connection classes.
>> They unfortunately are from an older (3.4.6) release and would some clean
>> up to fit into the current dev branch of Zookeeper.
>>
>> However feel free to have a look. Unfortunately there is a lot of code in
>> there (just like in the other implementations of that version, that aren't
>> directly related to
>> the connection (which later got cleaned up in ZK and would need clean up
>> on
>> my side too.)
>>
>> The biggest changes are in the ServerCnxn class starting here:
>> https://github.com/matthiasamberg/ZookeeperEmbeddedConnectio
>> n/blob/master/org/apache/zookeeper/server/EmbeddedServerCnxn.java#L416
>>
>> The gist of it is:
>> EmbeddedClientCnxnSocket and EmbeddedServerCnxn get instatiated together
>> when an (internal) client tries to connect. It finds the
>> EmbeddedServerCnxnFactory via a singleton instance of that class.
>> The ByteBuffers that normally get sent over the network simply end up in a
>> pair of Piped Input/Output streams that are shared between
>> ClientCnxnSocket
>> and ServerCnxn, that's already it. Everything around it just got adapted
>> to fit into the ZK architecture.
>>
>> I happily port this to a current version of ZK if you (and/or others)
>> think
>> this implementation approach would actually get patched into ZK, but it
>> would be great to hear from you guys whether it is worthwile for me to
>> port
>> this to the newest version first.
>>
>> Best regards,
>>
>> Matthias Amberg
>> Developer
>> Office: ++41 511 11 64
>>
>> unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>>
>> unblu makes websites personal http://www.unblu.com
>>
>> Please consider the environment before printing this email.
>>
>> This e-mail contains including its attachments confidential protected
>> information by law. If you are not the intended recipient, please contact
>> the sender and delete this message. Any unauthorised copying of this
>> message or unauthorised distribution of the information contained herein
>> is
>> prohibited.
>>
>> Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
>> geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind,
>> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
>> Das
>> unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
>> Informationen ist nicht gestattet.
>>
>> On 30 September 2016 at 14:30, Enrico Olivelli <eo...@gmail.com>
>> wrote:
>>
>> > Matthias,
>> > I am sorry I so not have time to pack a good patch these days.
>> > If you already have some working code maybe you can create a PR on
>> GitHub
>> > so that the community can review and comment about it.
>> >
>> > My actual idea is to make all Netty-aware classes extensible in order to
>> > support "local" transport.
>> > In 3.5 branch Netty is fully supported for both client and server sides
>> of
>> > the connection
>> >
>> > On BookKeeper I have already implemented such a feature
>> > https://issues.apache.org/jira/browse/BOOKKEEPER-896
>> > https://github.com/apache/bookkeeper/pull/20
>> >
>> > Regards
>> > Enrico
>> >
>> >
>> >
>> > Il ven 30 set 2016, 14:14 Matthias Amberg <ma...@unblu.com>
>> ha
>> > scritto:
>> >
>> > > Hi Enrico,
>> > >
>> > > thanks for your response, sorry for the late reply.
>> > > Would you mind sharing some details of your upcoming implementation,
>> do
>> > you
>> > > think it will cover our use case of using it for an embedded Zookeeper
>> > > instance for a production system?
>> > > If yes, we'll continue to use our own implementation until this
>> becomes
>> > > available in Zookeeper, do you have an ETA?
>> > >
>> > > Best regards,
>> > >
>> > > Matthias
>> > >
>> > >
>> > > Matthias Amberg
>> > > Developer
>> > > Office: ++41 511 11 64
>> > >
>> > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>> > >
>> > > unblu makes websites personal http://www.unblu.com
>> > >
>> > > Please consider the environment before printing this email.
>> > >
>> > > This e-mail contains including its attachments confidential protected
>> > > information by law. If you are not the intended recipient, please
>> contact
>> > > the sender and delete this message. Any unauthorised copying of this
>> > > message or unauthorised distribution of the information contained
>> herein
>> > is
>> > > prohibited.
>> > >
>> > > Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche
>> rechtlich
>> > > geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger
>> > sind,
>> > > informieren Sie bitte sofort den Absender und löschen Sie diese
>> E-Mail.
>> > Das
>> > > unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
>> > > Informationen ist nicht gestattet.
>> > >
>> > > On 27 September 2016 at 12:40, Enrico Olivelli <eo...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Matthias,
>> > > > I am working on a proposal on this topic. My idea is to use local
>> netty
>> > > > features.
>> > > > I will share some code.
>> > > > Beware that this usage of zookeeper is to be adopted only if
>> needed. My
>> > > > case is for instance to run unit tests without a real network.
>> > > > Maybe you should consider to abstract your zookkeper access layer
>> > > >
>> > > > Enrico
>> > > >
>> > > > Il mar 27 set 2016, 11:07 Matthias Amberg <
>> matthias.amberg@unblu.com>
>> > ha
>> > > > scritto:
>> > > >
>> > > > > Dear Zookeeper developers,
>> > > > >
>> > > > > I have a feature request for allowing an embedded Zookeeper be run
>> > and
>> > > > > connected to without using any system networking stack. All
>> > > > > client-server communications would be handled internally of a
>> single
>> > > > > runtime.
>> > > > > We at Unblu offer our product as a cloud service (orchestrated by
>> > > > > Zookeeper) as well as an on-premises installable product.
>> > > > > In the latter scenario we use a single embedded version of
>> Zookeeper
>> > to
>> > > > > orchestrate all the different embedded Version.
>> > > > > This setup allows us to use the same mechanisms for service
>> discovery
>> > > and
>> > > > > synchronization, whether we run in a distributed cloud setup or
>> > > > > a single on-premises node.
>> > > > >
>> > > > > For this use case we have implemented versions of ServerCnxn,
>> > > > > ServerCnxnFactory and ClientCnxnSocket that handle client-server
>> > > > > connections internally without
>> > > > > using any server networking stack.
>> > > > >
>> > > > > I would very much love to commit a patch that implements this
>> feature
>> > > or
>> > > > > open a jira issue to discuss this further, if there is interest
>> from
>> > > the
>> > > > > Zookeeper community for such a feature.
>> > > > >
>> > > > > We think this feature offers several advantages over running
>> > Zookeeper
>> > > > > locally or embedded via a networking stack:
>> > > > > - less system resource usage (no need for additional networking
>> > > overhead
>> > > > in
>> > > > > what could be systems that already have a high load of networking
>> IO)
>> > > > > - simpler configuration and potentially better security: no need
>> to
>> > > > > configure any ports and network addresses, firewalls, loopback
>> > > interfaces
>> > > > > etc. (Our software runs in banking environments where security
>> > > > constraints
>> > > > > often makes it
>> > > > > difficult to (even) have loopback connections)
>> > > > > - simpler Continuous Integration: Different Test Runs of our
>> builds
>> > > > > constantly interfere with each other due to ports being already in
>> > use
>> > > > and
>> > > > > so forth.
>> > > > >
>> > > > > Additionally as this feature simply would be an extension to the
>> > > current
>> > > > > code base that wouldn't even be instantiated in the traditional
>> > server
>> > > > > setup there would be almost no interference with the current code
>> > base
>> > > > and
>> > > > > hence no threat
>> > > > > to Zookeeper stability in traditional setups.
>> > > > >
>> > > > > It would be great if you could provide your thoughts and feedback
>> on
>> > > this
>> > > > > and whether this is a welcome feature, in which case I would
>> create a
>> > > > Jira
>> > > > > Issue for further discussion and patch submission.
>> > > > >
>> > > > > Best regards,
>> > > > >
>> > > > > Matthias Amberg
>> > > > > Developer
>> > > > > Office: ++41 511 11 64
>> > > > >
>> > > > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>> > > > >
>> > > > > unblu makes websites personal http://www.unblu.com
>> > > > >
>> > > > --
>> > > >
>> > > >
>> > > > -- Enrico Olivelli
>> > > >
>> > >
>> >
>>
>
>
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Matthias Amberg <ma...@unblu.com>.
Hi Enrico,
I set up a github repo with just the 'embedded' connection classes.
They unfortunately are from an older (3.4.6) release and would some clean
up to fit into the current dev branch of Zookeeper.
However feel free to have a look. Unfortunately there is a lot of code in
there (just like in the other implementations of that version, that aren't
directly related to
the connection (which later got cleaned up in ZK and would need clean up on
my side too.)
The biggest changes are in the ServerCnxn class starting here:
https://github.com/matthiasamberg/ZookeeperEmbeddedConnection/blob/master/org/apache/zookeeper/server/EmbeddedServerCnxn.java#L416
The gist of it is:
EmbeddedClientCnxnSocket and EmbeddedServerCnxn get instatiated together
when an (internal) client tries to connect. It finds the
EmbeddedServerCnxnFactory via a singleton instance of that class.
The ByteBuffers that normally get sent over the network simply end up in a
pair of Piped Input/Output streams that are shared between ClientCnxnSocket
and ServerCnxn, that's already it. Everything around it just got adapted
to fit into the ZK architecture.
I happily port this to a current version of ZK if you (and/or others) think
this implementation approach would actually get patched into ZK, but it
would be great to hear from you guys whether it is worthwile for me to port
this to the newest version first.
Best regards,
Matthias Amberg
Developer
Office: ++41 511 11 64
unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
unblu makes websites personal http://www.unblu.com
Please consider the environment before printing this email.
This e-mail contains including its attachments confidential protected
information by law. If you are not the intended recipient, please contact
the sender and delete this message. Any unauthorised copying of this
message or unauthorised distribution of the information contained herein is
prohibited.
Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind,
informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail. Das
unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
Informationen ist nicht gestattet.
On 30 September 2016 at 14:30, Enrico Olivelli <eo...@gmail.com> wrote:
> Matthias,
> I am sorry I so not have time to pack a good patch these days.
> If you already have some working code maybe you can create a PR on GitHub
> so that the community can review and comment about it.
>
> My actual idea is to make all Netty-aware classes extensible in order to
> support "local" transport.
> In 3.5 branch Netty is fully supported for both client and server sides of
> the connection
>
> On BookKeeper I have already implemented such a feature
> https://issues.apache.org/jira/browse/BOOKKEEPER-896
> https://github.com/apache/bookkeeper/pull/20
>
> Regards
> Enrico
>
>
>
> Il ven 30 set 2016, 14:14 Matthias Amberg <ma...@unblu.com> ha
> scritto:
>
> > Hi Enrico,
> >
> > thanks for your response, sorry for the late reply.
> > Would you mind sharing some details of your upcoming implementation, do
> you
> > think it will cover our use case of using it for an embedded Zookeeper
> > instance for a production system?
> > If yes, we'll continue to use our own implementation until this becomes
> > available in Zookeeper, do you have an ETA?
> >
> > Best regards,
> >
> > Matthias
> >
> >
> > Matthias Amberg
> > Developer
> > Office: ++41 511 11 64
> >
> > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
> >
> > unblu makes websites personal http://www.unblu.com
> >
> > Please consider the environment before printing this email.
> >
> > This e-mail contains including its attachments confidential protected
> > information by law. If you are not the intended recipient, please contact
> > the sender and delete this message. Any unauthorised copying of this
> > message or unauthorised distribution of the information contained herein
> is
> > prohibited.
> >
> > Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
> > geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger
> sind,
> > informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
> Das
> > unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
> > Informationen ist nicht gestattet.
> >
> > On 27 September 2016 at 12:40, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> >
> > > Hi Matthias,
> > > I am working on a proposal on this topic. My idea is to use local netty
> > > features.
> > > I will share some code.
> > > Beware that this usage of zookeeper is to be adopted only if needed. My
> > > case is for instance to run unit tests without a real network.
> > > Maybe you should consider to abstract your zookkeper access layer
> > >
> > > Enrico
> > >
> > > Il mar 27 set 2016, 11:07 Matthias Amberg <ma...@unblu.com>
> ha
> > > scritto:
> > >
> > > > Dear Zookeeper developers,
> > > >
> > > > I have a feature request for allowing an embedded Zookeeper be run
> and
> > > > connected to without using any system networking stack. All
> > > > client-server communications would be handled internally of a single
> > > > runtime.
> > > > We at Unblu offer our product as a cloud service (orchestrated by
> > > > Zookeeper) as well as an on-premises installable product.
> > > > In the latter scenario we use a single embedded version of Zookeeper
> to
> > > > orchestrate all the different embedded Version.
> > > > This setup allows us to use the same mechanisms for service discovery
> > and
> > > > synchronization, whether we run in a distributed cloud setup or
> > > > a single on-premises node.
> > > >
> > > > For this use case we have implemented versions of ServerCnxn,
> > > > ServerCnxnFactory and ClientCnxnSocket that handle client-server
> > > > connections internally without
> > > > using any server networking stack.
> > > >
> > > > I would very much love to commit a patch that implements this feature
> > or
> > > > open a jira issue to discuss this further, if there is interest from
> > the
> > > > Zookeeper community for such a feature.
> > > >
> > > > We think this feature offers several advantages over running
> Zookeeper
> > > > locally or embedded via a networking stack:
> > > > - less system resource usage (no need for additional networking
> > overhead
> > > in
> > > > what could be systems that already have a high load of networking IO)
> > > > - simpler configuration and potentially better security: no need to
> > > > configure any ports and network addresses, firewalls, loopback
> > interfaces
> > > > etc. (Our software runs in banking environments where security
> > > constraints
> > > > often makes it
> > > > difficult to (even) have loopback connections)
> > > > - simpler Continuous Integration: Different Test Runs of our builds
> > > > constantly interfere with each other due to ports being already in
> use
> > > and
> > > > so forth.
> > > >
> > > > Additionally as this feature simply would be an extension to the
> > current
> > > > code base that wouldn't even be instantiated in the traditional
> server
> > > > setup there would be almost no interference with the current code
> base
> > > and
> > > > hence no threat
> > > > to Zookeeper stability in traditional setups.
> > > >
> > > > It would be great if you could provide your thoughts and feedback on
> > this
> > > > and whether this is a welcome feature, in which case I would create a
> > > Jira
> > > > Issue for further discussion and patch submission.
> > > >
> > > > Best regards,
> > > >
> > > > Matthias Amberg
> > > > Developer
> > > > Office: ++41 511 11 64
> > > >
> > > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
> > > >
> > > > unblu makes websites personal http://www.unblu.com
> > > >
> > > --
> > >
> > >
> > > -- Enrico Olivelli
> > >
> >
>
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Enrico Olivelli <eo...@gmail.com>.
Matthias,
I am sorry I so not have time to pack a good patch these days.
If you already have some working code maybe you can create a PR on GitHub
so that the community can review and comment about it.
My actual idea is to make all Netty-aware classes extensible in order to
support "local" transport.
In 3.5 branch Netty is fully supported for both client and server sides of
the connection
On BookKeeper I have already implemented such a feature
https://issues.apache.org/jira/browse/BOOKKEEPER-896
https://github.com/apache/bookkeeper/pull/20
Regards
Enrico
Il ven 30 set 2016, 14:14 Matthias Amberg <ma...@unblu.com> ha
scritto:
> Hi Enrico,
>
> thanks for your response, sorry for the late reply.
> Would you mind sharing some details of your upcoming implementation, do you
> think it will cover our use case of using it for an embedded Zookeeper
> instance for a production system?
> If yes, we'll continue to use our own implementation until this becomes
> available in Zookeeper, do you have an ETA?
>
> Best regards,
>
> Matthias
>
>
> Matthias Amberg
> Developer
> Office: ++41 511 11 64
>
> unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>
> unblu makes websites personal http://www.unblu.com
>
> Please consider the environment before printing this email.
>
> This e-mail contains including its attachments confidential protected
> information by law. If you are not the intended recipient, please contact
> the sender and delete this message. Any unauthorised copying of this
> message or unauthorised distribution of the information contained herein is
> prohibited.
>
> Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
> geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind,
> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail. Das
> unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
> Informationen ist nicht gestattet.
>
> On 27 September 2016 at 12:40, Enrico Olivelli <eo...@gmail.com>
> wrote:
>
> > Hi Matthias,
> > I am working on a proposal on this topic. My idea is to use local netty
> > features.
> > I will share some code.
> > Beware that this usage of zookeeper is to be adopted only if needed. My
> > case is for instance to run unit tests without a real network.
> > Maybe you should consider to abstract your zookkeper access layer
> >
> > Enrico
> >
> > Il mar 27 set 2016, 11:07 Matthias Amberg <ma...@unblu.com> ha
> > scritto:
> >
> > > Dear Zookeeper developers,
> > >
> > > I have a feature request for allowing an embedded Zookeeper be run and
> > > connected to without using any system networking stack. All
> > > client-server communications would be handled internally of a single
> > > runtime.
> > > We at Unblu offer our product as a cloud service (orchestrated by
> > > Zookeeper) as well as an on-premises installable product.
> > > In the latter scenario we use a single embedded version of Zookeeper to
> > > orchestrate all the different embedded Version.
> > > This setup allows us to use the same mechanisms for service discovery
> and
> > > synchronization, whether we run in a distributed cloud setup or
> > > a single on-premises node.
> > >
> > > For this use case we have implemented versions of ServerCnxn,
> > > ServerCnxnFactory and ClientCnxnSocket that handle client-server
> > > connections internally without
> > > using any server networking stack.
> > >
> > > I would very much love to commit a patch that implements this feature
> or
> > > open a jira issue to discuss this further, if there is interest from
> the
> > > Zookeeper community for such a feature.
> > >
> > > We think this feature offers several advantages over running Zookeeper
> > > locally or embedded via a networking stack:
> > > - less system resource usage (no need for additional networking
> overhead
> > in
> > > what could be systems that already have a high load of networking IO)
> > > - simpler configuration and potentially better security: no need to
> > > configure any ports and network addresses, firewalls, loopback
> interfaces
> > > etc. (Our software runs in banking environments where security
> > constraints
> > > often makes it
> > > difficult to (even) have loopback connections)
> > > - simpler Continuous Integration: Different Test Runs of our builds
> > > constantly interfere with each other due to ports being already in use
> > and
> > > so forth.
> > >
> > > Additionally as this feature simply would be an extension to the
> current
> > > code base that wouldn't even be instantiated in the traditional server
> > > setup there would be almost no interference with the current code base
> > and
> > > hence no threat
> > > to Zookeeper stability in traditional setups.
> > >
> > > It would be great if you could provide your thoughts and feedback on
> this
> > > and whether this is a welcome feature, in which case I would create a
> > Jira
> > > Issue for further discussion and patch submission.
> > >
> > > Best regards,
> > >
> > > Matthias Amberg
> > > Developer
> > > Office: ++41 511 11 64
> > >
> > > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
> > >
> > > unblu makes websites personal http://www.unblu.com
> > >
> > --
> >
> >
> > -- Enrico Olivelli
> >
>
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Matthias Amberg <ma...@unblu.com>.
Hi Enrico,
thanks for your response, sorry for the late reply.
Would you mind sharing some details of your upcoming implementation, do you
think it will cover our use case of using it for an embedded Zookeeper
instance for a production system?
If yes, we'll continue to use our own implementation until this becomes
available in Zookeeper, do you have an ETA?
Best regards,
Matthias
Matthias Amberg
Developer
Office: ++41 511 11 64
unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
unblu makes websites personal http://www.unblu.com
Please consider the environment before printing this email.
This e-mail contains including its attachments confidential protected
information by law. If you are not the intended recipient, please contact
the sender and delete this message. Any unauthorised copying of this
message or unauthorised distribution of the information contained herein is
prohibited.
Diese E-Mail einschließlich Ihrer Anhänge enthält vertrauliche rechtlich
geschützte Informationen. Wenn Sie nicht der beabsichtigte Empfänger sind,
informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail. Das
unbefugte Kopieren oder die unbefugte Weitergabe der enthaltenen
Informationen ist nicht gestattet.
On 27 September 2016 at 12:40, Enrico Olivelli <eo...@gmail.com> wrote:
> Hi Matthias,
> I am working on a proposal on this topic. My idea is to use local netty
> features.
> I will share some code.
> Beware that this usage of zookeeper is to be adopted only if needed. My
> case is for instance to run unit tests without a real network.
> Maybe you should consider to abstract your zookkeper access layer
>
> Enrico
>
> Il mar 27 set 2016, 11:07 Matthias Amberg <ma...@unblu.com> ha
> scritto:
>
> > Dear Zookeeper developers,
> >
> > I have a feature request for allowing an embedded Zookeeper be run and
> > connected to without using any system networking stack. All
> > client-server communications would be handled internally of a single
> > runtime.
> > We at Unblu offer our product as a cloud service (orchestrated by
> > Zookeeper) as well as an on-premises installable product.
> > In the latter scenario we use a single embedded version of Zookeeper to
> > orchestrate all the different embedded Version.
> > This setup allows us to use the same mechanisms for service discovery and
> > synchronization, whether we run in a distributed cloud setup or
> > a single on-premises node.
> >
> > For this use case we have implemented versions of ServerCnxn,
> > ServerCnxnFactory and ClientCnxnSocket that handle client-server
> > connections internally without
> > using any server networking stack.
> >
> > I would very much love to commit a patch that implements this feature or
> > open a jira issue to discuss this further, if there is interest from the
> > Zookeeper community for such a feature.
> >
> > We think this feature offers several advantages over running Zookeeper
> > locally or embedded via a networking stack:
> > - less system resource usage (no need for additional networking overhead
> in
> > what could be systems that already have a high load of networking IO)
> > - simpler configuration and potentially better security: no need to
> > configure any ports and network addresses, firewalls, loopback interfaces
> > etc. (Our software runs in banking environments where security
> constraints
> > often makes it
> > difficult to (even) have loopback connections)
> > - simpler Continuous Integration: Different Test Runs of our builds
> > constantly interfere with each other due to ports being already in use
> and
> > so forth.
> >
> > Additionally as this feature simply would be an extension to the current
> > code base that wouldn't even be instantiated in the traditional server
> > setup there would be almost no interference with the current code base
> and
> > hence no threat
> > to Zookeeper stability in traditional setups.
> >
> > It would be great if you could provide your thoughts and feedback on this
> > and whether this is a welcome feature, in which case I would create a
> Jira
> > Issue for further discussion and patch submission.
> >
> > Best regards,
> >
> > Matthias Amberg
> > Developer
> > Office: ++41 511 11 64
> >
> > unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
> >
> > unblu makes websites personal http://www.unblu.com
> >
> --
>
>
> -- Enrico Olivelli
>
Re: Feature Proposal: Embedded (non network stack) connection option
for embedded Zookeeper
Posted by Enrico Olivelli <eo...@gmail.com>.
Hi Matthias,
I am working on a proposal on this topic. My idea is to use local netty
features.
I will share some code.
Beware that this usage of zookeeper is to be adopted only if needed. My
case is for instance to run unit tests without a real network.
Maybe you should consider to abstract your zookkeper access layer
Enrico
Il mar 27 set 2016, 11:07 Matthias Amberg <ma...@unblu.com> ha
scritto:
> Dear Zookeeper developers,
>
> I have a feature request for allowing an embedded Zookeeper be run and
> connected to without using any system networking stack. All
> client-server communications would be handled internally of a single
> runtime.
> We at Unblu offer our product as a cloud service (orchestrated by
> Zookeeper) as well as an on-premises installable product.
> In the latter scenario we use a single embedded version of Zookeeper to
> orchestrate all the different embedded Version.
> This setup allows us to use the same mechanisms for service discovery and
> synchronization, whether we run in a distributed cloud setup or
> a single on-premises node.
>
> For this use case we have implemented versions of ServerCnxn,
> ServerCnxnFactory and ClientCnxnSocket that handle client-server
> connections internally without
> using any server networking stack.
>
> I would very much love to commit a patch that implements this feature or
> open a jira issue to discuss this further, if there is interest from the
> Zookeeper community for such a feature.
>
> We think this feature offers several advantages over running Zookeeper
> locally or embedded via a networking stack:
> - less system resource usage (no need for additional networking overhead in
> what could be systems that already have a high load of networking IO)
> - simpler configuration and potentially better security: no need to
> configure any ports and network addresses, firewalls, loopback interfaces
> etc. (Our software runs in banking environments where security constraints
> often makes it
> difficult to (even) have loopback connections)
> - simpler Continuous Integration: Different Test Runs of our builds
> constantly interfere with each other due to ports being already in use and
> so forth.
>
> Additionally as this feature simply would be an extension to the current
> code base that wouldn't even be instantiated in the traditional server
> setup there would be almost no interference with the current code base and
> hence no threat
> to Zookeeper stability in traditional setups.
>
> It would be great if you could provide your thoughts and feedback on this
> and whether this is a welcome feature, in which case I would create a Jira
> Issue for further discussion and patch submission.
>
> Best regards,
>
> Matthias Amberg
> Developer
> Office: ++41 511 11 64
>
> unblu inc. - Kernserstrasse 17 - 6060 Sarnen - Switzerland
>
> unblu makes websites personal http://www.unblu.com
>
--
-- Enrico Olivelli