You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Owen Nichols (Jira)" <ji...@apache.org> on 2020/09/17 00:25:00 UTC
[jira] [Updated] (GEODE-8385) hang recovering from disk with cyclic
dependencies
[ https://issues.apache.org/jira/browse/GEODE-8385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen Nichols updated GEODE-8385:
--------------------------------
Fix Version/s: 1.12.1
> hang recovering from disk with cyclic dependencies
> --------------------------------------------------
>
> Key: GEODE-8385
> URL: https://issues.apache.org/jira/browse/GEODE-8385
> Project: Geode
> Issue Type: Bug
> Components: membership, persistence
> Reporter: Bruce J Schuchardt
> Assignee: Bruce J Schuchardt
> Priority: Major
> Labels: no-release-note, pull-request-available
> Fix For: 1.12.1, 1.13.0, 1.14.0
>
>
> In a test cluster using replicated persistent Regions all of the servers were shut down and restarted. The restart hung showing a cycle in disk store dependencies.
> {noformat}
> [info 2020/05/29 03:02:36.635 PDT <Thread-18> tid=0x8f] Region /Region_14 has potentially stale data. It is waiting for another online member to recover the latest data.My persistent id:
> DiskStore ID: a175354a-d27d-4575-9916-16fd7ff7ea67 Name: persistgemfire4_host1_4194 Location: /10.32.110.100:/var/vcap/data/rundir/concRecoverAllV4O41/concRecoverAll-0529-024642/vm_5_persist4_disk_1
> Members with potentially new data:[
> DiskStore ID: 2d77752e-507d-4425-a382-a5856c61938f Name: persistgemfire10_host1_4208 Location: /10.32.110.100:/var/vcap/data/rundir/concRecoverAllV4O41/concRecoverAll-0529-024642/vm_2_persist10_disk_1]
> Use the gfsh show missing-disk-stores command to see all disk stores that are being waited on by other members.
> {noformat}
> After looking at the logs for all members, the "members with potentially new data" for each member were found to be:
> {noformat}
> Member | Members with potentially new data
> --------+----------------------------------
> 1 | all
> 2 | 4
> 3 | 4
> 4 | 10
> 5 | 2, 3, 4, 8, 10
> 6 | 2, 3, 4, 5, 7, 8, 10
> 7 | 3, 4, 10
> 8 | 3, 4, 10
> 9 | 2, 3, 4, 5, 7, 8, 10
> 10 | 3
> {noformat}
> It appears that there is a cycle in this "waiting for another online member" graph between 3 > 4 > 10 > 3.
> The problem seems to have cropped up after the fix for GEODE-7196 was merged. That changed the timing of member-departed notifications such that a server might close a Region's Persistence Advisor before getting notification that another server was shutting down. We used to do this notification upon receipt of a ShutdownMessage but now we only do it when the membership view has changed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)