You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Jakov Varenina <ja...@est.tech> on 2020/10/13 14:50:37 UTC
Member that is shutting down initiate removal of other members from
the cluster
Hi all,
sorry for bothering, but we have noticed some differences in behavior
between 1.11 and 1.12 releases and need your help in understanding them.
First I would like to mention that we are running geode in Kubernetes.
We perform shutdown of the worker node that is hosting one member(e.g.
coordinator locator). Shutdown procedure affect member in a following way:
1. TCP shared unordered connections towards other members are terminated
2. Member receives graceful shut-down indication and starts with the
shut-down procedure
Usually connections starting to be terminated first and the shut-down
indication comes short after (e.g. ~10 milliseconds in difference). The
step 1. triggers availability check towards the other members for which
TCP connection has been previously lost. At this point of time
coordinator is unaware of ongoing shut-down and assumes that all other
members are actually having issues due to connection loss. Even after
coordinator receives the graceful shut-down indication this process of
availability check is not stopped. What happens later on is that
availability check fail for all members and coordinator initiates their
removal with RemoveMemberMessage. This message is succesfully received
on the other members forcing them to shut-down.
In geode 1.11 everything is same except the fact that availability check
pass and therefore removals aren't initiated.
In logs it can be seen that for both releases TCP availability check
fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on
1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat
request and heartbeat messages are sent but does not reach their
destination members. RemoveMemberMessage which are sent later on reach
their destination successfully. Does anybody know what was changed in
1.12 that could lead to such difference in behavior?
Additionally, availability check is not stopped when graceful shutdown
is initiated. Do you think that this could be improved, so that member
stops ongoing availability check when detects gracefull shutdown? Just
to add that shutdown procedure is also delayed due to unsuccessful
attempts to estabilsh TCP connections towards the other members.
BRs,
Jakov
Re: Member that is shutting down initiate removal of other members
from the cluster
Posted by Aaron Lindsey <al...@vmware.com>.
Hi Jakov,
Do you see this issue if you drain the worker node before shutting it down? For reference: https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/. I know this doesn’t directly answer your questions, but it seems like draining the node first would help the members receive the shutdown signal before the TCP connections are terminated. For what it’s worth, we also run Geode on Kubernetes but haven’t encountered this issue. Knowing whether it happens when you drain the node first would help us to be able to reproduce it.
Thanks,
Aaron
On Oct 20, 2020, at 7:49 AM, Jakov Varenina <ja...@est.tech>> wrote:
Hi Ernie,
Thank you really much for trying to help.
Unfortunately we haven't been able to reproduce this issue without k8s.
I would like to share some additional info about this issue. We know that this is not the "correct" way to shut down member gracefully, since it's complete TCP communication towards the other members is forcefully terminated even before it receives shutdown indication. This issue with removals from previous mail can be easily avoided by enforcing policy that the TCP connections towards other members aren't terminated until member terminates them by itself during graceful shutdown.
Another thing is that the removals occur only when when we shut down complete k8s node as described in previous mail. When we reproduce the same situation without shutting down k8s node then the removals aren't initiated:
"In parallel terminate member TCP communication and initiate gracefully shut down the member. What happens then is that availability check over UDP using Heartbeat messages towards other members pass, and removals aren't initiated towards other members"
Given the above additional test result when using k8s with 1.12 release, and due to fact that we were unable to reproduce the issue on geode only, we are not sure that the problem is actually in geode. Please feel free to contact me by slack or mail for any additional questions or things you would like to discuss.
BRs,
Jakov
On 19. 10. 2020. 20:26, Ernie Burghardt wrote:
Hi Jakov,
I'm looking into your question(s)... curious if you've run into this in a non-k8s cluster?
Might help focus the investigation...
Thanks,
EB
On 10/13/20, 7:51 AM, "Jakov Varenina" <ja...@est.tech>> wrote:
Hi all,
sorry for bothering, but we have noticed some differences in behavior
between 1.11 and 1.12 releases and need your help in understanding them.
First I would like to mention that we are running geode in Kubernetes.
We perform shutdown of the worker node that is hosting one member(e.g.
coordinator locator). Shutdown procedure affect member in a following way:
1. TCP shared unordered connections towards other members are terminated
2. Member receives graceful shut-down indication and starts with the
shut-down procedure
Usually connections starting to be terminated first and the shut-down
indication comes short after (e.g. ~10 milliseconds in difference). The
step 1. triggers availability check towards the other members for which
TCP connection has been previously lost. At this point of time
coordinator is unaware of ongoing shut-down and assumes that all other
members are actually having issues due to connection loss. Even after
coordinator receives the graceful shut-down indication this process of
availability check is not stopped. What happens later on is that
availability check fail for all members and coordinator initiates their
removal with RemoveMemberMessage. This message is succesfully received
on the other members forcing them to shut-down.
In geode 1.11 everything is same except the fact that availability check
pass and therefore removals aren't initiated.
In logs it can be seen that for both releases TCP availability check
fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on
1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat
request and heartbeat messages are sent but does not reach their
destination members. RemoveMemberMessage which are sent later on reach
their destination successfully. Does anybody know what was changed in
1.12 that could lead to such difference in behavior?
Additionally, availability check is not stopped when graceful shutdown
is initiated. Do you think that this could be improved, so that member
stops ongoing availability check when detects gracefull shutdown? Just
to add that shutdown procedure is also delayed due to unsuccessful
attempts to estabilsh TCP connections towards the other members.
BRs,
Jakov
Re: Member that is shutting down initiate removal of other members
from the cluster
Posted by Jakov Varenina <ja...@est.tech>.
Hi Ernie,
Thank you really much for trying to help.
Unfortunately we haven't been able to reproduce this issue without k8s.
I would like to share some additional info about this issue. We know
that this is not the "correct" way to shut down member gracefully, since
it's complete TCP communication towards the other members is forcefully
terminated even before it receives shutdown indication. This issue with
removals from previous mail can be easily avoided by enforcing policy
that the TCP connections towards other members aren't terminated until
member terminates them by itself during graceful shutdown.
Another thing is that the removals occur only when when we shut down
complete k8s node as described in previous mail. When we reproduce the
same situation without shutting down k8s node then the removals aren't
initiated:
"In parallel terminate member TCP communication and initiate gracefully
shut down the member. What happens then is that availability check over
UDP using Heartbeat messages towards other members pass, and removals
aren't initiated towards other members"
Given the above additional test result when using k8s with 1.12 release,
and due to fact that we were unable to reproduce the issue on geode
only, we are not sure that the problem is actually in geode. Please feel
free to contact me by slack or mail for any additional questions or
things you would like to discuss.
BRs,
Jakov
On 19. 10. 2020. 20:26, Ernie Burghardt wrote:
> Hi Jakov,
>
> I'm looking into your question(s)... curious if you've run into this in a non-k8s cluster?
> Might help focus the investigation...
>
> Thanks,
> EB
>
> On 10/13/20, 7:51 AM, "Jakov Varenina" <ja...@est.tech> wrote:
>
> Hi all,
>
> sorry for bothering, but we have noticed some differences in behavior
> between 1.11 and 1.12 releases and need your help in understanding them.
>
> First I would like to mention that we are running geode in Kubernetes.
> We perform shutdown of the worker node that is hosting one member(e.g.
> coordinator locator). Shutdown procedure affect member in a following way:
>
> 1. TCP shared unordered connections towards other members are terminated
>
> 2. Member receives graceful shut-down indication and starts with the
> shut-down procedure
>
> Usually connections starting to be terminated first and the shut-down
> indication comes short after (e.g. ~10 milliseconds in difference). The
> step 1. triggers availability check towards the other members for which
> TCP connection has been previously lost. At this point of time
> coordinator is unaware of ongoing shut-down and assumes that all other
> members are actually having issues due to connection loss. Even after
> coordinator receives the graceful shut-down indication this process of
> availability check is not stopped. What happens later on is that
> availability check fail for all members and coordinator initiates their
> removal with RemoveMemberMessage. This message is succesfully received
> on the other members forcing them to shut-down.
>
> In geode 1.11 everything is same except the fact that availability check
> pass and therefore removals aren't initiated.
>
> In logs it can be seen that for both releases TCP availability check
> fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on
> 1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat
> request and heartbeat messages are sent but does not reach their
> destination members. RemoveMemberMessage which are sent later on reach
> their destination successfully. Does anybody know what was changed in
> 1.12 that could lead to such difference in behavior?
>
> Additionally, availability check is not stopped when graceful shutdown
> is initiated. Do you think that this could be improved, so that member
> stops ongoing availability check when detects gracefull shutdown? Just
> to add that shutdown procedure is also delayed due to unsuccessful
> attempts to estabilsh TCP connections towards the other members.
>
> BRs,
> Jakov
>
>
Re: Member that is shutting down initiate removal of other members
from the cluster
Posted by Ernie Burghardt <bu...@vmware.com>.
Hi Jakov,
I'm looking into your question(s)... curious if you've run into this in a non-k8s cluster?
Might help focus the investigation...
Thanks,
EB
On 10/13/20, 7:51 AM, "Jakov Varenina" <ja...@est.tech> wrote:
Hi all,
sorry for bothering, but we have noticed some differences in behavior
between 1.11 and 1.12 releases and need your help in understanding them.
First I would like to mention that we are running geode in Kubernetes.
We perform shutdown of the worker node that is hosting one member(e.g.
coordinator locator). Shutdown procedure affect member in a following way:
1. TCP shared unordered connections towards other members are terminated
2. Member receives graceful shut-down indication and starts with the
shut-down procedure
Usually connections starting to be terminated first and the shut-down
indication comes short after (e.g. ~10 milliseconds in difference). The
step 1. triggers availability check towards the other members for which
TCP connection has been previously lost. At this point of time
coordinator is unaware of ongoing shut-down and assumes that all other
members are actually having issues due to connection loss. Even after
coordinator receives the graceful shut-down indication this process of
availability check is not stopped. What happens later on is that
availability check fail for all members and coordinator initiates their
removal with RemoveMemberMessage. This message is succesfully received
on the other members forcing them to shut-down.
In geode 1.11 everything is same except the fact that availability check
pass and therefore removals aren't initiated.
In logs it can be seen that for both releases TCP availability check
fail, but HeartbeatMessageRequest/HearbeatMessage check fails only on
1.12 and pass on 1.11. In 1.12 release it can be seen that heartbeat
request and heartbeat messages are sent but does not reach their
destination members. RemoveMemberMessage which are sent later on reach
their destination successfully. Does anybody know what was changed in
1.12 that could lead to such difference in behavior?
Additionally, availability check is not stopped when graceful shutdown
is initiated. Do you think that this could be improved, so that member
stops ongoing availability check when detects gracefull shutdown? Just
to add that shutdown procedure is also delayed due to unsuccessful
attempts to estabilsh TCP connections towards the other members.
BRs,
Jakov