You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@geode.apache.org by Bruce Schuchardt <bs...@pivotal.io> on 2016/01/19 22:03:00 UTC

Review Request 42520: Reducing join/leave time

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42520/
-----------------------------------------------------------

Review request for geode, anilkumar gingade, Hitesh Khamesra, and Jason Huynh.

Repository: geode

Description
-------

The view collection interval in Geode was set to more than 3x the interval that Pivotal GemFire used: 500ms instead of 150ms. This was in part due to failures in testing when a member cycled its cache frequently but was too aggressive. Geode also sends a redundant "leave" message in GMSJoinLeave - redundant because the DistributionManager has already sent a ShutdownMessage that is treated as a GMS "leave" message and is an acked message. This combined to make Geode about 4x slower than GemFire 8.2 at cycling the cache.

This change-set reduces the collection interval to 300ms and removes the pause after sending the "leave" message. Performance testing shows that Geode is about 25% faster with these changes at cycling a cache (close cache, disconnect DistributedSystem, reconnect and rebuild cache) than GemFire 8.2.

The changes exposed a bug in the "join" logic in GMSJoinLeave. If a view is received while waiting for a join response and the view contains the new member's ID it is treated as a join response. However the code waiting for the join response wasn't correctly checking the "isJoined" status and ended up waiting 12000ms before allowing startup to continue. While fixing this I refactored the attemptToJoin method to make it easier to read.

Diffs
-----

gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/ServiceConfig.java a412dfa00d984f9a31e4c291d99f138a57d8ebd7
gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java abdceb418c08e1fe830e619e88d91e39b921c231
gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/mgr/GMSMembershipManager.java 7fe55e5b53d5d9ba3057537d5edec90e5f69c6c4

Diff: https://reviews.apache.org/r/42520/diff/

Testing
-------

Thanks,

Bruce Schuchardt

Re: Review Request 42520: Reducing join/leave time

Posted by Bruce Schuchardt <bs...@pivotal.io>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42520/#review115438
-----------------------------------------------------------



gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/mgr/GMSMembershipManager.java (line 2037)
<https://reviews.apache.org/r/42520/#comment176430>

    This is an unrelated change - I noticed debug output from these threads hanging around from old DistributedSystems.  They sleep for 3 seconds and then close connections to a member, but if the DirectChannel isn't open there is no reason for them to do anything.


- Bruce Schuchardt


On Jan. 20, 2016, 4:35 p.m., Bruce Schuchardt wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42520/
> -----------------------------------------------------------
> 
> (Updated Jan. 20, 2016, 4:35 p.m.)
> 
> 
> Review request for geode, anilkumar gingade, Hitesh Khamesra, and Jason Huynh.
> 
> 
> Repository: geode
> 
> 
> Description
> -------
> 
> The view collection interval in Geode was set to more than 3x the interval that Pivotal GemFire used: 500ms instead of 150ms.  This was in part due to failures in testing when a member cycled its cache frequently but was too aggressive.  Geode also sends a redundant "leave" message in GMSJoinLeave - redundant because the DistributionManager has already sent a ShutdownMessage that is treated as a GMS "leave" message and is an acked message.  This combined to make Geode about 4x slower than GemFire 8.2 at cycling the cache.
> 
> This change-set reduces the collection interval to 300ms and removes the pause after sending the "leave" message.  Performance testing shows that Geode is about 25% faster with these changes at cycling a cache (close cache, disconnect DistributedSystem, reconnect and rebuild cache) than GemFire 8.2.
> 
> The changes exposed a bug in the "join" logic in GMSJoinLeave.  If a view is received while waiting for a join response and the view contains the new member's ID it is treated as a join response.  However the code waiting for the join response wasn't correctly checking the "isJoined" status and ended up waiting 12000ms before allowing startup to continue.  While fixing this I refactored the attemptToJoin method to make it easier to read.
> 
> 
> Diffs
> -----
> 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/ServiceConfig.java a412dfa00d984f9a31e4c291d99f138a57d8ebd7 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/membership/GMSJoinLeave.java abdceb418c08e1fe830e619e88d91e39b921c231 
>   gemfire-core/src/main/java/com/gemstone/gemfire/distributed/internal/membership/gms/mgr/GMSMembershipManager.java 7fe55e5b53d5d9ba3057537d5edec90e5f69c6c4 
> 
> Diff: https://reviews.apache.org/r/42520/diff/
> 
> 
> Testing
> -------
> 
> These changes have passed precheckin and regression testing
> 
> 
> Thanks,
> 
> Bruce Schuchardt
> 
>