You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@geode.apache.org by Bruce Schuchardt <bs...@pivotal.io> on 2016/02/11 20:56:18 UTC

Review Request 43497: fixing network-partition detection

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43497/
-----------------------------------------------------------

Review request for geode, Hitesh Khamesra and Udo Kohlmeyer.


Repository: geode


Description
-------

In some network-partition testing a member did not figure out that there was a partition and remained up when it should have shut down.

The problem was in the removeHealthMembers(), and more specifically in its filterMembers() method.  That method was removing members from the suspects collection regardless of whether it was being removed or a shutdown message from it had been received.  It should do neither and merely return a collection of IDs taken from pending-requests that are in the suspects collection.


Diffs
-----

  gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/NetView.java b0ddcc09190f3a91fcea860f28efe6f2c52ea3ad 
  gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 0f16ba97df7396db174dde560cb529709aeb8e43 
  gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeaveJUnitTest.java 5b58c273348b0c03b68ad14193724ae3d5c3f7eb 
  gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/JGroupsMessengerJUnitTest.java 805dd88115a2a13c5fae0c1d5cfce547acc4cd59 

Diff: https://reviews.apache.org/r/43497/diff/


Testing
-------

network-partition testing, precheckin


Thanks,

Bruce Schuchardt


Re: Review Request 43497: fixing network-partition detection

Posted by Udo Kohlmeyer <uk...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43497/#review118907
-----------------------------------------------------------


Ship it!




Ship It!

- Udo Kohlmeyer


On Feb. 11, 2016, 7:56 p.m., Bruce Schuchardt wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43497/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 7:56 p.m.)
> 
> 
> Review request for geode, Hitesh Khamesra and Udo Kohlmeyer.
> 
> 
> Repository: geode
> 
> 
> Description
> -------
> 
> In some network-partition testing a member did not figure out that there was a partition and remained up when it should have shut down.
> 
> The problem was in the removeHealthMembers(), and more specifically in its filterMembers() method.  That method was removing members from the suspects collection regardless of whether it was being removed or a shutdown message from it had been received.  It should do neither and merely return a collection of IDs taken from pending-requests that are in the suspects collection.
> 
> 
> Diffs
> -----
> 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/NetView.java b0ddcc09190f3a91fcea860f28efe6f2c52ea3ad 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 0f16ba97df7396db174dde560cb529709aeb8e43 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeaveJUnitTest.java 5b58c273348b0c03b68ad14193724ae3d5c3f7eb 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/JGroupsMessengerJUnitTest.java 805dd88115a2a13c5fae0c1d5cfce547acc4cd59 
> 
> Diff: https://reviews.apache.org/r/43497/diff/
> 
> 
> Testing
> -------
> 
> network-partition testing, precheckin
> 
> 
> Thanks,
> 
> Bruce Schuchardt
> 
>


Re: Review Request 43497: fixing network-partition detection

Posted by Bruce Schuchardt <bs...@pivotal.io>.

> On Feb. 11, 2016, 9:58 p.m., Hitesh Khamesra wrote:
> > gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java, line 2196
> > <https://reviews.apache.org/r/43497/diff/1/?file=1240369#file1240369line2196>
> >
> >     Here we don't want to verify member if that is removed or leaving, correct? if that's the case then filtermember was updating "suspect" list based on remove/leave status..

Your old code was removing the new removals/leaves from the suspects set and then adding the removals back in.  I've changed this because the IDs in newRemovals do not represent healthy members and shouldn't be removed from the collection.  So, instead of removing all of the removes/leaves in filterMember I changed it to do this in removeHealthyMembers using the newRemovals/newLeaves collections.  Now filterMembers just filters the view requests and update the collection of matching IDs.


> On Feb. 11, 2016, 9:58 p.m., Hitesh Khamesra wrote:
> > gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java, line 2263
> > <https://reviews.apache.org/r/43497/diff/1/?file=1240369#file1240369line2263>
> >
> >     It seems, we should be checking after we come out from while loop, just to be safer side.

The suspects set is updated after every pair of filterMembers invocations in removeHealthyMembers().  It wouldn't do any good to update it a second time with the same newRemovals/newLeaves collections.


- Bruce


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43497/#review118930
-----------------------------------------------------------


On Feb. 11, 2016, 7:56 p.m., Bruce Schuchardt wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43497/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 7:56 p.m.)
> 
> 
> Review request for geode, Hitesh Khamesra and Udo Kohlmeyer.
> 
> 
> Repository: geode
> 
> 
> Description
> -------
> 
> In some network-partition testing a member did not figure out that there was a partition and remained up when it should have shut down.
> 
> The problem was in the removeHealthMembers(), and more specifically in its filterMembers() method.  That method was removing members from the suspects collection regardless of whether it was being removed or a shutdown message from it had been received.  It should do neither and merely return a collection of IDs taken from pending-requests that are in the suspects collection.
> 
> 
> Diffs
> -----
> 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/NetView.java b0ddcc09190f3a91fcea860f28efe6f2c52ea3ad 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 0f16ba97df7396db174dde560cb529709aeb8e43 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeaveJUnitTest.java 5b58c273348b0c03b68ad14193724ae3d5c3f7eb 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/JGroupsMessengerJUnitTest.java 805dd88115a2a13c5fae0c1d5cfce547acc4cd59 
> 
> Diff: https://reviews.apache.org/r/43497/diff/
> 
> 
> Testing
> -------
> 
> network-partition testing, precheckin
> 
> 
> Thanks,
> 
> Bruce Schuchardt
> 
>


Re: Review Request 43497: fixing network-partition detection

Posted by Hitesh Khamesra <hk...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43497/#review118930
-----------------------------------------------------------




gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java (line 2196)
<https://reviews.apache.org/r/43497/#comment180240>

    Here we don't want to verify member if that is removed or leaving, correct? if that's the case then filtermember was updating "suspect" list based on remove/leave status..



gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 
<https://reviews.apache.org/r/43497/#comment180239>

    It seems, we should be checking after we come out from while loop, just to be safer side.


- Hitesh Khamesra


On Feb. 11, 2016, 7:56 p.m., Bruce Schuchardt wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43497/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 7:56 p.m.)
> 
> 
> Review request for geode, Hitesh Khamesra and Udo Kohlmeyer.
> 
> 
> Repository: geode
> 
> 
> Description
> -------
> 
> In some network-partition testing a member did not figure out that there was a partition and remained up when it should have shut down.
> 
> The problem was in the removeHealthMembers(), and more specifically in its filterMembers() method.  That method was removing members from the suspects collection regardless of whether it was being removed or a shutdown message from it had been received.  It should do neither and merely return a collection of IDs taken from pending-requests that are in the suspects collection.
> 
> 
> Diffs
> -----
> 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/NetView.java b0ddcc09190f3a91fcea860f28efe6f2c52ea3ad 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 0f16ba97df7396db174dde560cb529709aeb8e43 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeaveJUnitTest.java 5b58c273348b0c03b68ad14193724ae3d5c3f7eb 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/JGroupsMessengerJUnitTest.java 805dd88115a2a13c5fae0c1d5cfce547acc4cd59 
> 
> Diff: https://reviews.apache.org/r/43497/diff/
> 
> 
> Testing
> -------
> 
> network-partition testing, precheckin
> 
> 
> Thanks,
> 
> Bruce Schuchardt
> 
>


Re: Review Request 43497: fixing network-partition detection

Posted by Hitesh Khamesra <hk...@pivotal.io>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43497/#review118966
-----------------------------------------------------------


Ship it!




Ship It!

- Hitesh Khamesra


On Feb. 11, 2016, 7:56 p.m., Bruce Schuchardt wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43497/
> -----------------------------------------------------------
> 
> (Updated Feb. 11, 2016, 7:56 p.m.)
> 
> 
> Review request for geode, Hitesh Khamesra and Udo Kohlmeyer.
> 
> 
> Repository: geode
> 
> 
> Description
> -------
> 
> In some network-partition testing a member did not figure out that there was a partition and remained up when it should have shut down.
> 
> The problem was in the removeHealthMembers(), and more specifically in its filterMembers() method.  That method was removing members from the suspects collection regardless of whether it was being removed or a shutdown message from it had been received.  It should do neither and merely return a collection of IDs taken from pending-requests that are in the suspects collection.
> 
> 
> Diffs
> -----
> 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/NetView.java b0ddcc09190f3a91fcea860f28efe6f2c52ea3ad 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java 0f16ba97df7396db174dde560cb529709aeb8e43 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeaveJUnitTest.java 5b58c273348b0c03b68ad14193724ae3d5c3f7eb 
>   gemfire-core/src/test/java/com/gemstone/gemfire/distributed/internal/membership/gms/messenger/JGroupsMessengerJUnitTest.java 805dd88115a2a13c5fae0c1d5cfce547acc4cd59 
> 
> Diff: https://reviews.apache.org/r/43497/diff/
> 
> 
> Testing
> -------
> 
> network-partition testing, precheckin
> 
> 
> Thanks,
> 
> Bruce Schuchardt
> 
>