[jira] [Resolved] (GEODE-8809) Servers are missing heartbeats from a member

["Bill Burcham (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GEODE-8809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bill Burcham resolved GEODE-8809.
---------------------------------
    Resolution: Won't Fix

This looks like the failure detector did its job correctly. No Geode bugs were found.

> Servers are missing heartbeats from a member
> --------------------------------------------
>
>                 Key: GEODE-8809
>                 URL: https://issues.apache.org/jira/browse/GEODE-8809
>             Project: Geode
>          Issue Type: Bug
>          Components: messaging
>            Reporter: Nabarun Nag
>            Assignee: Bill Burcham
>            Priority: Major
>              Labels: blocks-1.14.0​
>
> We see this characteristic failure in a number of proprietary applications:
>  * member stops sending heartbeats
>  * The coordinator is requesting availability test from a member, 
>  * member gets it after a delay
>  * the delay causes the server to be kicked out (receives FordedDisconnectException)
>  * operations fail.
>  * server reconnects.
> Usually when the failure detector/health monitor kicks a member out of the distributed system it is for one of these reasons:
> 1. Member really was malfunctioning or unreachable (i.e. something outside of health monitoring had a problem)
>   a. Network problems
>     i. Partition: 2-way, N-way
>     ii. Slowdown or error rate increase
>   b. CPU was over-taxed in faulty member. We see gaps on the order of 10s or more in heartbeat generation on that member.
>     i. Geode was running in a virtualized environment and the virtualization system didn’t give the Geode process sufficient CPU
>     ii. JVM memory was over-utilized so garbage collection (pauses) took too long
>     iii. There was simply too much CPU demand and the product failed to reserve enough CPU capacity to keep the heartbeat going
> This ticket captures situations where the failure detector causes a member to be kicked out *but we cannot prove definitively that any of these as a root cause*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)