You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Chris Miles <ch...@chrismiles.org> on 2018/03/03 14:25:17 UTC
Connection Factories for Curator / Zookeeper / HTTP Tunneling
Firstly, I apologise for the cross post, but I think this is a question
which may need to be seen by both users, and devs who understand the
underlying code.
I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
where the only communication can happen between nodes is through HTTP,
so I am looking at ways of getting ZooKeeper communicating through HTTP
tunnelling.
As far as I can determine, ZooKeeper only allows the configuring of the
main client connection via server and client connection factories, but
not for the 2888 and 3888 connectivity, which is I think ((correct me if
wrong)) node to node communication on the first one, and leader election
on the second?
Does Curator's connection handling give me any ability to intercept and
wrap the connections used for the rest of these ports? (Netty Http Tunnel).
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Abraham Fine <af...@apache.org>.
Hi Chris-
Wouldn't it also be posssible to make use of an external tunneling tool without modifying ZooKeeper at all?
Abe
On Sun, Mar 4, 2018, at 11:58, Chris Miles wrote:
> Thanks Mark.
>
> I've only had a glance at the code around the server connection factory,
> and the fact there is a netty one there seems like a good sign as there
> are some generic netty http tunnel examples out there.
>
> If there's anything you can suggest as a starter for ten, I'd be appreciated.
>
> Thanks
>
> Chris
>
> Sent from my iPhone
>
> > On 4 Mar 2018, at 11:40, Mark Fenes <mf...@cloudera.com> wrote:
> >
> >
> > Hi Chris,
> >
> > yes, ports 2888 and 3888 are the default ports for quorum communication and leader election.
> > By default, ZK uses NIOServerCnxnFactory, unless the zookeeper.serverCnxnFactory system property is set to a different connection factory (e.g. Netty).
> >
> > So, you would like to configure and run the ZooKeeper server instances so that the quorum communication and leader election would also take place on the HTTP port via tunnelling?
> > Let me check this as I need to do further research to answer this question.
> >
> > And yes, should ZK not have this functionality, we would be very thankful for your willingness to contribute to the source code.
> >
> > Regards,
> > Mark
> >
> >
> >> On Sat, Mar 3, 2018 at 3:34 PM, Chris Miles <ch...@chrismiles.org> wrote:
> >>
> >> Firstly, I apologise for the cross post, but I think this is a question
> >> which may need to be seen by both users, and devs who understand the
> >> underlying code.
> >>
> >> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
> >> where the only communication can happen between nodes is through HTTP,
> >> so I am looking at ways of getting ZooKeeper communicating through HTTP
> >> tunnelling.
> >>
> >> As far as I can determine, ZooKeeper only allows the configuring of the
> >> main client connection via server and client connection factories, but
> >> not for the 2888 and 3888 connectivity, which is I think ((correct me if
> >> wrong)) node to node communication on the first one, and leader election
> >> on the second?
> >>
> >> Does Zookeeper's connection handling give me any ability to intercept and
> >> wrap the connections used for the rest of these ports? (Netty Http Tunnel).
> >>
> >> I am willing to contribute to the source to get this functionality if required
> >> as this is currently our only way of getting Zookeeper on our cloud.
> >>
> >> thanks
> >>
> >> Chris
> >>
> >>
> >>
> >
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Chris Miles <ch...@chrismiles.org>.
Thanks Mark.
I've only had a glance at the code around the server connection factory, and the fact there is a netty one there seems like a good sign as there are some generic netty http tunnel examples out there.
If there's anything you can suggest as a starter for ten, I'd be appreciated.
Thanks
Chris
Sent from my iPhone
> On 4 Mar 2018, at 11:40, Mark Fenes <mf...@cloudera.com> wrote:
>
>
> Hi Chris,
>
> yes, ports 2888 and 3888 are the default ports for quorum communication and leader election.
> By default, ZK uses NIOServerCnxnFactory, unless the zookeeper.serverCnxnFactory system property is set to a different connection factory (e.g. Netty).
>
> So, you would like to configure and run the ZooKeeper server instances so that the quorum communication and leader election would also take place on the HTTP port via tunnelling?
> Let me check this as I need to do further research to answer this question.
>
> And yes, should ZK not have this functionality, we would be very thankful for your willingness to contribute to the source code.
>
> Regards,
> Mark
>
>
>> On Sat, Mar 3, 2018 at 3:34 PM, Chris Miles <ch...@chrismiles.org> wrote:
>>
>> Firstly, I apologise for the cross post, but I think this is a question
>> which may need to be seen by both users, and devs who understand the
>> underlying code.
>>
>> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
>> where the only communication can happen between nodes is through HTTP,
>> so I am looking at ways of getting ZooKeeper communicating through HTTP
>> tunnelling.
>>
>> As far as I can determine, ZooKeeper only allows the configuring of the
>> main client connection via server and client connection factories, but
>> not for the 2888 and 3888 connectivity, which is I think ((correct me if
>> wrong)) node to node communication on the first one, and leader election
>> on the second?
>>
>> Does Zookeeper's connection handling give me any ability to intercept and
>> wrap the connections used for the rest of these ports? (Netty Http Tunnel).
>>
>> I am willing to contribute to the source to get this functionality if required
>> as this is currently our only way of getting Zookeeper on our cloud.
>>
>> thanks
>>
>> Chris
>>
>>
>>
>
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Mark Fenes <mf...@cloudera.com>.
Hi Chris,
yes, ports 2888 and 3888 are the default ports for quorum communication and
leader election.
By default, ZK uses NIOServerCnxnFactory, unless the
zookeeper.serverCnxnFactory system property is set to a different
connection factory (e.g. Netty).
So, you would like to configure and run the ZooKeeper server instances so
that the quorum communication and leader election would also take place on
the HTTP port via tunnelling?
Let me check this as I need to do further research to answer this question.
And yes, should ZK not have this functionality, we would be very thankful
for your willingness to contribute to the source code.
Regards,
Mark
On Sat, Mar 3, 2018 at 3:34 PM, Chris Miles <ch...@chrismiles.org> wrote:
>
> Firstly, I apologise for the cross post, but I think this is a question
> which may need to be seen by both users, and devs who understand the
> underlying code.
>
> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
> where the only communication can happen between nodes is through HTTP,
> so I am looking at ways of getting ZooKeeper communicating through HTTP
> tunnelling.
>
> As far as I can determine, ZooKeeper only allows the configuring of the
> main client connection via server and client connection factories, but
> not for the 2888 and 3888 connectivity, which is I think ((correct me if
> wrong)) node to node communication on the first one, and leader election
> on the second?
>
> Does Zookeeper's connection handling give me any ability to intercept and
> wrap the connections used for the rest of these ports? (Netty Http Tunnel).
>
> I am willing to contribute to the source to get this functionality if
> required
> as this is currently our only way of getting Zookeeper on our cloud.
>
> thanks
>
> Chris
>
>
>
>
Fwd: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Chris Miles <ch...@chrismiles.org>.
Firstly, I apologise for the cross post, but I think this is a question
which may need to be seen by both users, and devs who understand the
underlying code.
I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
where the only communication can happen between nodes is through HTTP,
so I am looking at ways of getting ZooKeeper communicating through HTTP
tunnelling.
As far as I can determine, ZooKeeper only allows the configuring of the
main client connection via server and client connection factories, but
not for the 2888 and 3888 connectivity, which is I think ((correct me if
wrong)) node to node communication on the first one, and leader election
on the second?
Does Zookeeper's connection handling give me any ability to intercept and
wrap the connections used for the rest of these ports? (Netty Http Tunnel).
I am willing to contribute to the source to get this functionality if required
as this is currently our only way of getting Zookeeper on our cloud.
thanks
Chris
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Chris Miles <ch...@chrismiles.org>.
Unless they support HTTP communication, they are not supported on our cloud.
CF supports TCP communication however the feature has been disabled in
our cloud on a firmwide basis for security reasons (global firm which I
can't name as I am under my personal address doing some weekend
researching). So the jist is I need to find a way to get it
communicating over HTTP.
I can create a connection factory for the main client connection which
will wrap it into a http request and response between nodes (I am using
it embedded), but I need to resolve the issue with the other ports and
find a way to intercept that traffic within the code.
I have also posted to Zookeeper, which is where this discussion should
likely move now as you have explained Curator does not give me anything
additional.
Thanks for the help. Hopefully someone else might chime in with some
more info. This has turned into a real blocker, as Zookeper is the only
implementation I can use for distributed state machine with spring state
machine.
Chris
> I don't know much about Cloud Foundry but how does it handle things like replicated databases, etc.? There are copious systems that don't do HTTP.
>
>> On Mar 3, 2018, at 10:13 AM, Chris Miles <ch...@chrismiles.org> wrote:
>>
>> Thanks Jordan,
>>
>> It is a cloud-foundry cloud, the network restrictions are between nodes of deployed applications. So server instances cant communicate unless I can tunnel them through HTTP.
>>
>> HTTP is the only protocol of communication I am able to use. This is not currently changeable.
>>
>> Chris
>>
>>> Curator wraps the built-in ZooKeeper client, so Curator doesn't give you any benefit that isn't already present in ZooKeeper itself. You can easily use port 80 or 443 as the ZK/Curator client port. But, the ZooKeeper protocol (jute) is of course not HTTP. If the Firewall is expecting HTTP it won't work. Ports 2888/3888 are only used internally between ZooKeeper server instances. Those should all be behind a firewall so should be OK.
>>>
>>> -JZ
>>>
>>>> On Mar 3, 2018, at 9:25 AM, Chris Miles <ch...@chrismiles.org> wrote:
>>>>
>>>> Firstly, I apologise for the cross post, but I think this is a question which may need to be seen by both users, and devs who understand the underlying code.
>>>>
>>>> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud, where the only communication can happen between nodes is through HTTP, so I am looking at ways of getting ZooKeeper communicating through HTTP tunnelling.
>>>>
>>>> As far as I can determine, ZooKeeper only allows the configuring of the main client connection via server and client connection factories, but not for the 2888 and 3888 connectivity, which is I think ((correct me if wrong)) node to node communication on the first one, and leader election on the second?
>>>>
>>>> Does Curator's connection handling give me any ability to intercept and wrap the connections used for the rest of these ports? (Netty Http Tunnel).
>
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I don't know much about Cloud Foundry but how does it handle things like replicated databases, etc.? There are copious systems that don't do HTTP.
> On Mar 3, 2018, at 10:13 AM, Chris Miles <ch...@chrismiles.org> wrote:
>
> Thanks Jordan,
>
> It is a cloud-foundry cloud, the network restrictions are between nodes of deployed applications. So server instances cant communicate unless I can tunnel them through HTTP.
>
> HTTP is the only protocol of communication I am able to use. This is not currently changeable.
>
> Chris
>
>> Curator wraps the built-in ZooKeeper client, so Curator doesn't give you any benefit that isn't already present in ZooKeeper itself. You can easily use port 80 or 443 as the ZK/Curator client port. But, the ZooKeeper protocol (jute) is of course not HTTP. If the Firewall is expecting HTTP it won't work. Ports 2888/3888 are only used internally between ZooKeeper server instances. Those should all be behind a firewall so should be OK.
>>
>> -JZ
>>
>>> On Mar 3, 2018, at 9:25 AM, Chris Miles <ch...@chrismiles.org> wrote:
>>>
>>> Firstly, I apologise for the cross post, but I think this is a question which may need to be seen by both users, and devs who understand the underlying code.
>>>
>>> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud, where the only communication can happen between nodes is through HTTP, so I am looking at ways of getting ZooKeeper communicating through HTTP tunnelling.
>>>
>>> As far as I can determine, ZooKeeper only allows the configuring of the main client connection via server and client connection factories, but not for the 2888 and 3888 connectivity, which is I think ((correct me if wrong)) node to node communication on the first one, and leader election on the second?
>>>
>>> Does Curator's connection handling give me any ability to intercept and wrap the connections used for the rest of these ports? (Netty Http Tunnel).
>>
>
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Chris Miles <ch...@chrismiles.org>.
Thanks Jordan,
It is a cloud-foundry cloud, the network restrictions are between nodes
of deployed applications. So server instances cant communicate unless I
can tunnel them through HTTP.
HTTP is the only protocol of communication I am able to use. This is not
currently changeable.
Chris
> Curator wraps the built-in ZooKeeper client, so Curator doesn't give you any benefit that isn't already present in ZooKeeper itself. You can easily use port 80 or 443 as the ZK/Curator client port. But, the ZooKeeper protocol (jute) is of course not HTTP. If the Firewall is expecting HTTP it won't work. Ports 2888/3888 are only used internally between ZooKeeper server instances. Those should all be behind a firewall so should be OK.
>
> -JZ
>
>> On Mar 3, 2018, at 9:25 AM, Chris Miles <ch...@chrismiles.org> wrote:
>>
>> Firstly, I apologise for the cross post, but I think this is a question which may need to be seen by both users, and devs who understand the underlying code.
>>
>> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud, where the only communication can happen between nodes is through HTTP, so I am looking at ways of getting ZooKeeper communicating through HTTP tunnelling.
>>
>> As far as I can determine, ZooKeeper only allows the configuring of the main client connection via server and client connection factories, but not for the 2888 and 3888 connectivity, which is I think ((correct me if wrong)) node to node communication on the first one, and leader election on the second?
>>
>> Does Curator's connection handling give me any ability to intercept and wrap the connections used for the rest of these ports? (Netty Http Tunnel).
>
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Curator wraps the built-in ZooKeeper client, so Curator doesn't give you any benefit that isn't already present in ZooKeeper itself. You can easily use port 80 or 443 as the ZK/Curator client port. But, the ZooKeeper protocol (jute) is of course not HTTP. If the Firewall is expecting HTTP it won't work. Ports 2888/3888 are only used internally between ZooKeeper server instances. Those should all be behind a firewall so should be OK.
-JZ
> On Mar 3, 2018, at 9:25 AM, Chris Miles <ch...@chrismiles.org> wrote:
>
> Firstly, I apologise for the cross post, but I think this is a question which may need to be seen by both users, and devs who understand the underlying code.
>
> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud, where the only communication can happen between nodes is through HTTP, so I am looking at ways of getting ZooKeeper communicating through HTTP tunnelling.
>
> As far as I can determine, ZooKeeper only allows the configuring of the main client connection via server and client connection factories, but not for the 2888 and 3888 connectivity, which is I think ((correct me if wrong)) node to node communication on the first one, and leader election on the second?
>
> Does Curator's connection handling give me any ability to intercept and wrap the connections used for the rest of these ports? (Netty Http Tunnel).
Fwd: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Chris Miles <ch...@chrismiles.org>.
Firstly, I apologise for the cross post, but I think this is a question
which may need to be seen by both users, and devs who understand the
underlying code.
I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud,
where the only communication can happen between nodes is through HTTP,
so I am looking at ways of getting ZooKeeper communicating through HTTP
tunnelling.
As far as I can determine, ZooKeeper only allows the configuring of the
main client connection via server and client connection factories, but
not for the 2888 and 3888 connectivity, which is I think ((correct me if
wrong)) node to node communication on the first one, and leader election
on the second?
Does Zookeeper's connection handling give me any ability to intercept and
wrap the connections used for the rest of these ports? (Netty Http Tunnel).
I am willing to contribute to the source to get this functionality if required
as this is currently our only way of getting Zookeeper on our cloud.
thanks
Chris
Re: Connection Factories for Curator / Zookeeper / HTTP Tunneling
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
Curator wraps the built-in ZooKeeper client, so Curator doesn't give you any benefit that isn't already present in ZooKeeper itself. You can easily use port 80 or 443 as the ZK/Curator client port. But, the ZooKeeper protocol (jute) is of course not HTTP. If the Firewall is expecting HTTP it won't work. Ports 2888/3888 are only used internally between ZooKeeper server instances. Those should all be behind a firewall so should be OK.
-JZ
> On Mar 3, 2018, at 9:25 AM, Chris Miles <ch...@chrismiles.org> wrote:
>
> Firstly, I apologise for the cross post, but I think this is a question which may need to be seen by both users, and devs who understand the underlying code.
>
> I need to deploy Zookeeper to a firewall restricted cloud-foundry cloud, where the only communication can happen between nodes is through HTTP, so I am looking at ways of getting ZooKeeper communicating through HTTP tunnelling.
>
> As far as I can determine, ZooKeeper only allows the configuring of the main client connection via server and client connection factories, but not for the 2888 and 3888 connectivity, which is I think ((correct me if wrong)) node to node communication on the first one, and leader election on the second?
>
> Does Curator's connection handling give me any ability to intercept and wrap the connections used for the rest of these ports? (Netty Http Tunnel).