You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Alberto Bustamante Reyes <al...@est.tech> on 2021/01/19 00:45:25 UTC
Different binding addresses for traffic & membership
Hi geode-devs,
I have a question related with Geode & Kubernetes:
We would like to use Istio with Geode. For that, a sidecar container (Envoy) has to be added in each Geode pod. That sidecar container intercepts and handles all incoming and outgoing traffic for that pod. One of the requirements set by Istio towards applications trying to integrate with it is that the application listening ports need to be bound to either localhost or 0.0.0.0 address (which listens on all interfaces).
Geode binds the locator and server traffic port by default to 0.0.0.0, but the membership ports are bound to the pod IP.
And with Envoy listening on the pod IP for incoming traffic and proxying everything towards localhost, applications binding to pod IPs won't receive any traffic.
We have tried using the "bind-address" parameter, but that doesn't work for our case. Geode binds the listening ports to the configured address, but it also shares that same address to other members in the system as the address to be used to reach it. If we configure that address to localhost, it just won't work.
Is there any way to configure a bind address to be used only for membership? I have not seen any configuration parameter or property that could be useful to solve this problem, maybe I missed it.
Thanks in advance,
BR/
Alberto Bustamante
RE: Different binding addresses for traffic & membership
Posted by Alberto Bustamante Reyes <al...@est.tech>.
Thanks for your answer Dan! We have checked that setting "0.0.0.0" in DirectChannel and in GMSHealthMonitor seems to solve our problem. I created PR to check if using "0.0.0.0" could have an impact and all public test cases seems to work fine in that case (https://github.com/apache/geode/pull/5946).
Next step will be the implementation of an option to allow a user to set "0.0.0.0" address in these classes.
BR/
Alberto Bustamante
________________________________
De: Dan Smith <da...@vmware.com>
Enviado: jueves, 21 de enero de 2021 0:48
Para: dev@geode.apache.org <de...@geode.apache.org>
Asunto: Re: Different binding addresses for traffic & membership
I've been looking into the code a little bit to see if this is possible. I'm not sure it is right now.
Here's some pointers at where to look at. Most of the magic is happening in JGroupsMessenger. JGroupsMessenger wraps jgroups, which we are using to UDP messaging related to membership.
The first thing that happens is that JGroupsMessenger.init creates a jgroups configuration. It does some string replacement on the jgroups-config.xml file that is checked. It puts the configured bind-address into that configuration.
When JGroupsMessenger.start() is called jgroups will bind to that address. Right after that, JGroupsMessenger calls establishLocalAddress, which takes the IP address that jgroups just bound to and creates our local MemberIdentifier.
Later in GMSJoinLeave.attemptToJoin, it sends that local address to the coordinator. Assuming the join is successful, the coordinator will send out a view that includes that MemberIdentifier.
I was really hoping that just setting a bind address of "0.0.0.0" would do the right thing in this case. But it looks like jgroups won't let you bind to that address. I don't currently see a way to get a different address in the MemberIdentifier than the one that jgroups is listening on right now.
Besides the UDP port that jgroups is listening on, there are a couple of other TCP ports used for peer-to-peer messaging. GMSHealthMonitor also starts listening on the same local address returned from jgroups. And the DirectChannel class I think also eventually ends up creating a server socket that listens on the same bind-address. That one might be ok with "0.0.0.0".
-Dan
PS - there is a lot more information on membership on the wiki if it is helpful, but I don't think it gets into this level of detail about what address gets used - https://cwiki.apache.org/confluence/display/GEODE/Membership+Manager+and+Messaging.
________________________________
From: Aaron Lindsey <al...@vmware.com>
Sent: Wednesday, January 20, 2021 2:51 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Different binding addresses for traffic & membership
> Is there any way to configure a bind address to be used only for membership?
To your first question, I asked around but I’m not aware of anything like what you are looking for. What you are describing does seem like it could become a common setup on Kubernetes, but I personally haven’t tried using Geode with Istio and Envoy. Please share what you learn!
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
Maybe someone with more experience in the membership code could comment on this?
Aaron
> On Jan 20, 2021, at 9:07 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
>
> It seems this is not a trendic topic... 🙂 Let me share my approach by the moment, maybe this will receive more comments:
>
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
>
> Thanks!
>
> BR/
>
> Alberto Bustamante
>
>
>
>
>
> ________________________________
> De: Alberto Bustamante Reyes <al...@est.tech>
> Enviado: martes, 19 de enero de 2021 1:45
> Para: dev@geode.apache.org <de...@geode.apache.org>
> Asunto: Different binding addresses for traffic & membership
>
> Hi geode-devs,
>
> I have a question related with Geode & Kubernetes:
> We would like to use Istio with Geode. For that, a sidecar container (Envoy) has to be added in each Geode pod. That sidecar container intercepts and handles all incoming and outgoing traffic for that pod. One of the requirements set by Istio towards applications trying to integrate with it is that the application listening ports need to be bound to either localhost or 0.0.0.0 address (which listens on all interfaces).
>
> Geode binds the locator and server traffic port by default to 0.0.0.0, but the membership ports are bound to the pod IP.
> And with Envoy listening on the pod IP for incoming traffic and proxying everything towards localhost, applications binding to pod IPs won't receive any traffic.
>
> We have tried using the "bind-address" parameter, but that doesn't work for our case. Geode binds the listening ports to the configured address, but it also shares that same address to other members in the system as the address to be used to reach it. If we configure that address to localhost, it just won't work.
>
> Is there any way to configure a bind address to be used only for membership? I have not seen any configuration parameter or property that could be useful to solve this problem, maybe I missed it.
>
> Thanks in advance,
>
> BR/
>
> Alberto Bustamante
Re: Different binding addresses for traffic & membership
Posted by Dan Smith <da...@vmware.com>.
I've been looking into the code a little bit to see if this is possible. I'm not sure it is right now.
Here's some pointers at where to look at. Most of the magic is happening in JGroupsMessenger. JGroupsMessenger wraps jgroups, which we are using to UDP messaging related to membership.
The first thing that happens is that JGroupsMessenger.init creates a jgroups configuration. It does some string replacement on the jgroups-config.xml file that is checked. It puts the configured bind-address into that configuration.
When JGroupsMessenger.start() is called jgroups will bind to that address. Right after that, JGroupsMessenger calls establishLocalAddress, which takes the IP address that jgroups just bound to and creates our local MemberIdentifier.
Later in GMSJoinLeave.attemptToJoin, it sends that local address to the coordinator. Assuming the join is successful, the coordinator will send out a view that includes that MemberIdentifier.
I was really hoping that just setting a bind address of "0.0.0.0" would do the right thing in this case. But it looks like jgroups won't let you bind to that address. I don't currently see a way to get a different address in the MemberIdentifier than the one that jgroups is listening on right now.
Besides the UDP port that jgroups is listening on, there are a couple of other TCP ports used for peer-to-peer messaging. GMSHealthMonitor also starts listening on the same local address returned from jgroups. And the DirectChannel class I think also eventually ends up creating a server socket that listens on the same bind-address. That one might be ok with "0.0.0.0".
-Dan
PS - there is a lot more information on membership on the wiki if it is helpful, but I don't think it gets into this level of detail about what address gets used - https://cwiki.apache.org/confluence/display/GEODE/Membership+Manager+and+Messaging.
________________________________
From: Aaron Lindsey <al...@vmware.com>
Sent: Wednesday, January 20, 2021 2:51 PM
To: dev@geode.apache.org <de...@geode.apache.org>
Subject: Re: Different binding addresses for traffic & membership
> Is there any way to configure a bind address to be used only for membership?
To your first question, I asked around but I’m not aware of anything like what you are looking for. What you are describing does seem like it could become a common setup on Kubernetes, but I personally haven’t tried using Geode with Istio and Envoy. Please share what you learn!
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
Maybe someone with more experience in the membership code could comment on this?
Aaron
> On Jan 20, 2021, at 9:07 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
>
> It seems this is not a trendic topic... 🙂 Let me share my approach by the moment, maybe this will receive more comments:
>
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
>
> Thanks!
>
> BR/
>
> Alberto Bustamante
>
>
>
>
>
> ________________________________
> De: Alberto Bustamante Reyes <al...@est.tech>
> Enviado: martes, 19 de enero de 2021 1:45
> Para: dev@geode.apache.org <de...@geode.apache.org>
> Asunto: Different binding addresses for traffic & membership
>
> Hi geode-devs,
>
> I have a question related with Geode & Kubernetes:
> We would like to use Istio with Geode. For that, a sidecar container (Envoy) has to be added in each Geode pod. That sidecar container intercepts and handles all incoming and outgoing traffic for that pod. One of the requirements set by Istio towards applications trying to integrate with it is that the application listening ports need to be bound to either localhost or 0.0.0.0 address (which listens on all interfaces).
>
> Geode binds the locator and server traffic port by default to 0.0.0.0, but the membership ports are bound to the pod IP.
> And with Envoy listening on the pod IP for incoming traffic and proxying everything towards localhost, applications binding to pod IPs won't receive any traffic.
>
> We have tried using the "bind-address" parameter, but that doesn't work for our case. Geode binds the listening ports to the configured address, but it also shares that same address to other members in the system as the address to be used to reach it. If we configure that address to localhost, it just won't work.
>
> Is there any way to configure a bind address to be used only for membership? I have not seen any configuration parameter or property that could be useful to solve this problem, maybe I missed it.
>
> Thanks in advance,
>
> BR/
>
> Alberto Bustamante
Re: Different binding addresses for traffic & membership
Posted by Aaron Lindsey <al...@vmware.com>.
> Is there any way to configure a bind address to be used only for membership?
To your first question, I asked around but I’m not aware of anything like what you are looking for. What you are describing does seem like it could become a common setup on Kubernetes, but I personally haven’t tried using Geode with Istio and Envoy. Please share what you learn!
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
Maybe someone with more experience in the membership code could comment on this?
Aaron
> On Jan 20, 2021, at 9:07 AM, Alberto Bustamante Reyes <al...@est.tech> wrote:
>
> It seems this is not a trendic topic... 🙂 Let me share my approach by the moment, maybe this will receive more comments:
>
> I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
>
> Thanks!
>
> BR/
>
> Alberto Bustamante
>
>
>
>
>
> ________________________________
> De: Alberto Bustamante Reyes <al...@est.tech>
> Enviado: martes, 19 de enero de 2021 1:45
> Para: dev@geode.apache.org <de...@geode.apache.org>
> Asunto: Different binding addresses for traffic & membership
>
> Hi geode-devs,
>
> I have a question related with Geode & Kubernetes:
> We would like to use Istio with Geode. For that, a sidecar container (Envoy) has to be added in each Geode pod. That sidecar container intercepts and handles all incoming and outgoing traffic for that pod. One of the requirements set by Istio towards applications trying to integrate with it is that the application listening ports need to be bound to either localhost or 0.0.0.0 address (which listens on all interfaces).
>
> Geode binds the locator and server traffic port by default to 0.0.0.0, but the membership ports are bound to the pod IP.
> And with Envoy listening on the pod IP for incoming traffic and proxying everything towards localhost, applications binding to pod IPs won't receive any traffic.
>
> We have tried using the "bind-address" parameter, but that doesn't work for our case. Geode binds the listening ports to the configured address, but it also shares that same address to other members in the system as the address to be used to reach it. If we configure that address to localhost, it just won't work.
>
> Is there any way to configure a bind address to be used only for membership? I have not seen any configuration parameter or property that could be useful to solve this problem, maybe I missed it.
>
> Thanks in advance,
>
> BR/
>
> Alberto Bustamante
RE: Different binding addresses for traffic & membership
Posted by Alberto Bustamante Reyes <al...@est.tech>.
It seems this is not a trendic topic... 🙂 Let me share my approach by the moment, maybe this will receive more comments:
I thought that it will be interesting to take a look at how the membership works (how the distributed system is created), to check if at some point I could decouple how the value of "bind-address" parameter is used to configure binding and to indicate other members that they can reach the new member at that hostname. Any comment about what I should check first is welcome.
Thanks!
BR/
Alberto Bustamante
________________________________
De: Alberto Bustamante Reyes <al...@est.tech>
Enviado: martes, 19 de enero de 2021 1:45
Para: dev@geode.apache.org <de...@geode.apache.org>
Asunto: Different binding addresses for traffic & membership
Hi geode-devs,
I have a question related with Geode & Kubernetes:
We would like to use Istio with Geode. For that, a sidecar container (Envoy) has to be added in each Geode pod. That sidecar container intercepts and handles all incoming and outgoing traffic for that pod. One of the requirements set by Istio towards applications trying to integrate with it is that the application listening ports need to be bound to either localhost or 0.0.0.0 address (which listens on all interfaces).
Geode binds the locator and server traffic port by default to 0.0.0.0, but the membership ports are bound to the pod IP.
And with Envoy listening on the pod IP for incoming traffic and proxying everything towards localhost, applications binding to pod IPs won't receive any traffic.
We have tried using the "bind-address" parameter, but that doesn't work for our case. Geode binds the listening ports to the configured address, but it also shares that same address to other members in the system as the address to be used to reach it. If we configure that address to localhost, it just won't work.
Is there any way to configure a bind address to be used only for membership? I have not seen any configuration parameter or property that could be useful to solve this problem, maybe I missed it.
Thanks in advance,
BR/
Alberto Bustamante