[jira] [Created] (GEODE-8385) hang recovering from disk with cyclic dependencies

["Bruce J Schuchardt (Jira)" <ji...@apache.org>]

Bruce J Schuchardt created GEODE-8385:
-----------------------------------------

             Summary: hang recovering from disk with cyclic dependencies
                 Key: GEODE-8385
                 URL: https://issues.apache.org/jira/browse/GEODE-8385
             Project: Geode
          Issue Type: Bug
          Components: membership, persistence
            Reporter: Bruce J Schuchardt
             Fix For: 1.13.0


In a test cluster using replicated persistent Regions all of the servers were shut down and restarted.  The restart hung showing a cycle in disk store dependencies.

 {noformat}
[info 2020/05/29 03:02:36.635 PDT <Thread-18> tid=0x8f] Region /Region_14 has potentially stale data. It is waiting for another online member to recover the latest data.My persistent id:
  DiskStore ID: a175354a-d27d-4575-9916-16fd7ff7ea67  Name: persistgemfire4_host1_4194  Location: /10.32.110.100:/var/vcap/data/rundir/concRecoverAllV4O41/concRecoverAll-0529-024642/vm_5_persist4_disk_1
Members with potentially new data:[  
DiskStore ID: 2d77752e-507d-4425-a382-a5856c61938f  Name: persistgemfire10_host1_4208  Location: /10.32.110.100:/var/vcap/data/rundir/concRecoverAllV4O41/concRecoverAll-0529-024642/vm_2_persist10_disk_1]
Use the gfsh show missing-disk-stores command to see all disk stores that are being waited on by other members.
{noformat}

After looking at the logs for all members, the "members with potentially new data" for each member were found to be:

{noformat}
Member | Members with potentially new data
--------+----------------------------------
1 | all
2 | 4
3 | 4
4 | 10
5 | 2, 3, 4, 8, 10
6 | 2, 3, 4, 5, 7, 8, 10
7 | 3, 4, 10
8 | 3, 4, 10
9 | 2, 3, 4, 5, 7, 8, 10
10 | 3
{noformat}

It appears that there is a cycle in this "waiting for another online member" graph between 3 > 4 > 10 > 3.

The problem seems to have cropped up after the fix for GEODE-7196 was merged.  That changed the timing of member-departed notifications such that a server might close a Region's Persistence Advisor before getting notification that another server was shutting down.  We used to do this notification upon receipt of a ShutdownMessage but now we only do it when the membership view has changed.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)