You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Barry Oglesby (JIRA)" <ji...@apache.org> on 2015/11/16 20:47:11 UTC
[jira] [Comment Edited] (GEODE-514)
DistributionManagerDUnitTest.testKickOutSickMember has suspect string
[ https://issues.apache.org/jira/browse/GEODE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007176#comment-15007176 ]
Barry Oglesby edited comment on GEODE-514 at 11/16/15 7:47 PM:
---------------------------------------------------------------
This failure reproduced in:
Geode_develop_DistributedTests Build #594 (Nov 16, 2015 5:09:12 AM).
Revision: 88da702593157d8a0c014295cab16149fc088dfc
https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/
{noformat}
Suspicious strings were written to the log during this run.
Fix the strings or use DistributedTestCase.addExpectedException to ignore.
-----------------------------------------------------------------------
Found suspect string in log4j at line 981
[fatal 2015/11/16 08:30:55.682 PST <Quorum Lost Notification> tid=0x7b0] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
-----------------------------------------------------------------------
Found suspect string in log4j at line 985
[fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
{noformat}
Here is output from the log file:
{noformat}
[setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember
[locator][warn 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] failed to collect all ACKs (2) for view preparation [timor(5835:locator)<v0>:26739|173] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.210 PST <Failed to ACK member verify Thread-1> tid=0x2f5] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|174] [timor(5835:locator)<v0>:26739/53740] crashed mbrs: [timor(5801)<v163>:50799/3528]] (1 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|174] [timor(5835:locator)<v0>:26739/53740] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:10.365 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|175] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528, timor(putter:5801)<v175>:55014/18309]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:11.812 PST <P2P message reader@1411ccf1> tid=0x2f9] Admitting member <timor(putter:5801)<v175>:55014>. Now there are 2 non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|175] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528, timor(putter:5801)<v175>:55014/18309] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <Failed to ACK member verify Thread-1> tid=0x2fc] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|176] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|176] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:22.807 PST <UDP Loopback Message Handler> tid=0x24] Membership: lead member is now timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:22.808 PST <FD_SOCK Ping thread> tid=0x2fe] GemFire failure detection is now monitoring timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:23.827 PST <Pooled High Priority Message Processor 3> tid=0x102] Member timor(putter:5801)<v175>:55014 is equivalent or in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:24.017 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|177] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069]] (4 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:24.023 PST <P2P message reader@1ddc7ffb> tid=0x300] Admitting member <timor(sleeper:5856)<v177>:10709>. Now there are 3 non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] failed to collect all ACKs (4) for view preparation [timor(5835:locator)<v0>:26739|177] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <Failed to ACK member verify Thread-1> tid=0x302] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:36.459 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|178] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(sleeper:5856)<v177>:10709/55069] crashed mbrs: [timor(5801)<v163>:50799/3528]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|178] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(sleeper:5856)<v177>:10709/55069] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:38.044 PST <Pooled High Priority Message Processor 3> tid=0x102] Member timor(sleeper:5856)<v177>:10709 is equivalent or in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:38.084 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(sleeper:5856)<v177>:10709.
[locator]
[locator][info 2015/11/16 08:30:43.080 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=Failed to respond within ack-wait-threshold
[locator]
[locator][info 2015/11/16 08:30:43.085 PST <VERIFY_SUSPECT.TimerThread> tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response received from timor(sleeper:5856)<v177>:10709 in 5001 milliseconds: I believe it is gone.
[locator]
[locator][info 2015/11/16 08:30:43.238 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|179] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:48.082 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <VERIFY_SUSPECT.TimerThread> tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response received from timor(sleeper:5856)<v177>:10709 in 5002 milliseconds: I believe it is gone.
[locator]
[locator][warn 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|179] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <Failed to ACK member verify Thread-1> tid=0x307] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|180] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39]
[locator] member timor(5835:locator)<v0>:26739 has a weight of 3
[locator] member timor(putter:5801)<v175>:55014 has a weight of 15
[locator] member timor(sleeper:5856)<v177>:10709 has a weight of 10
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] quorum weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure weight must be < 14)
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|180] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069, timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20
[locator]
[locator][info 2015/11/16 08:30:55.682 PST <View Message Processor> tid=0x3c] Member at timor(sleeper:5856)<v177>:10709 unexpectedly left the distributed cache: departed JGroups view
[locator]
[locator][fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.691 PST <Pooled High Priority Message Processor 3> tid=0x102] Member at timor(putter:5801)<v175>:55014 gracefully left the distributed cache: shutdown message received
[locator]
[locator][info 2015/11/16 08:30:55.835 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|181] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.836 PST <FD_SOCK Ping thread> tid=0x2fe] Member timor(putter:5801)<v175>:55014 shut down with normal termination.
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP Connection(25)-10.118.32.89> tid=0x12] Received method: dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP Connection(25)-10.118.32.89> tid=0x12] Got result: null
[locator] from dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 (took 0 ms)
[locator]
{noformat}
was (Author: barry.oglesby):
This failure reproduced in Geode_develop_DistributedTests Build #594 (Nov 16, 2015 5:09:12 AM).
https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/
{noformat}
Suspicious strings were written to the log during this run.
Fix the strings or use DistributedTestCase.addExpectedException to ignore.
-----------------------------------------------------------------------
Found suspect string in log4j at line 981
[fatal 2015/11/16 08:30:55.682 PST <Quorum Lost Notification> tid=0x7b0] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
-----------------------------------------------------------------------
Found suspect string in log4j at line 985
[fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
{noformat}
Here is output from the log file:
{noformat}
[setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember
[locator][warn 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] failed to collect all ACKs (2) for view preparation [timor(5835:locator)<v0>:26739|173] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.210 PST <Failed to ACK member verify Thread-1> tid=0x2f5] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|174] [timor(5835:locator)<v0>:26739/53740] crashed mbrs: [timor(5801)<v163>:50799/3528]] (1 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|174] [timor(5835:locator)<v0>:26739/53740] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:10.365 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|175] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528, timor(putter:5801)<v175>:55014/18309]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:11.812 PST <P2P message reader@1411ccf1> tid=0x2f9] Admitting member <timor(putter:5801)<v175>:55014>. Now there are 2 non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|175] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528, timor(putter:5801)<v175>:55014/18309] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <Failed to ACK member verify Thread-1> tid=0x2fc] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|176] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|176] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:22.807 PST <UDP Loopback Message Handler> tid=0x24] Membership: lead member is now timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:22.808 PST <FD_SOCK Ping thread> tid=0x2fe] GemFire failure detection is now monitoring timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:23.827 PST <Pooled High Priority Message Processor 3> tid=0x102] Member timor(putter:5801)<v175>:55014 is equivalent or in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:24.017 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|177] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069]] (4 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:24.023 PST <P2P message reader@1ddc7ffb> tid=0x300] Admitting member <timor(sleeper:5856)<v177>:10709>. Now there are 3 non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] failed to collect all ACKs (4) for view preparation [timor(5835:locator)<v0>:26739|177] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <Failed to ACK member verify Thread-1> tid=0x302] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:36.459 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|178] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(sleeper:5856)<v177>:10709/55069] crashed mbrs: [timor(5801)<v163>:50799/3528]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|178] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(sleeper:5856)<v177>:10709/55069] crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:38.044 PST <Pooled High Priority Message Processor 3> tid=0x102] Member timor(sleeper:5856)<v177>:10709 is equivalent or in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:38.084 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(sleeper:5856)<v177>:10709.
[locator]
[locator][info 2015/11/16 08:30:43.080 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=Failed to respond within ack-wait-threshold
[locator]
[locator][info 2015/11/16 08:30:43.085 PST <VERIFY_SUSPECT.TimerThread> tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response received from timor(sleeper:5856)<v177>:10709 in 5001 milliseconds: I believe it is gone.
[locator]
[locator][info 2015/11/16 08:30:43.238 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|179] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:48.082 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <UDP ucast receiver> tid=0x25] Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709] from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <VERIFY_SUSPECT.TimerThread> tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response received from timor(sleeper:5856)<v177>:10709 in 5002 milliseconds: I believe it is gone.
[locator]
[locator][warn 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] failed to collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|179] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309, timor(5801)<v163>:50799/3528] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] attempting to contact members that did not respond to view preparation: [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <Failed to ACK member verify Thread-1> tid=0x307] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|180] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39]
[locator] member timor(5835:locator)<v0>:26739 has a weight of 3
[locator] member timor(putter:5801)<v175>:55014 has a weight of 15
[locator] member timor(sleeper:5856)<v177>:10709 has a weight of 10
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] quorum weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure weight must be < 14)
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler> tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|180] [timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309] crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069, timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler> tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20
[locator]
[locator][info 2015/11/16 08:30:55.682 PST <View Message Processor> tid=0x3c] Member at timor(sleeper:5856)<v177>:10709 unexpectedly left the distributed cache: departed JGroups view
[locator]
[locator][fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308] Possible loss of quorum due to the loss of 2 cache processes: [timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.691 PST <Pooled High Priority Message Processor 3> tid=0x102] Member at timor(putter:5801)<v175>:55014 gracefully left the distributed cache: shutdown message received
[locator]
[locator][info 2015/11/16 08:30:55.835 PST <ViewHandler> tid=0x39] Membership: sending new view [[timor(5835:locator)<v0>:26739|181] [timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.836 PST <FD_SOCK Ping thread> tid=0x2fe] Member timor(putter:5801)<v175>:55014 shut down with normal termination.
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP Connection(25)-10.118.32.89> tid=0x12] Received method: dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP Connection(25)-10.118.32.89> tid=0x12] Got result: null
[locator] from dunit.DistributedTestCase$5.run with 0 args on object: dunit.DistributedTestCase$5@7a954221 (took 0 ms)
[locator]
{noformat}
> DistributionManagerDUnitTest.testKickOutSickMember has suspect string
> ---------------------------------------------------------------------
>
> Key: GEODE-514
> URL: https://issues.apache.org/jira/browse/GEODE-514
> Project: Geode
> Issue Type: Bug
> Reporter: xiaojian zhou
> Labels: CI
>
> revision 0cc9d895b9f4465138d0fa223b0a0cadc1107893
> {noformat}
> java.lang.AssertionError: Suspicious strings were written to the log during this run.
> Fix the strings or use DistributedTestCase.addExpectedException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 990
> [fatal 2015/10/29 06:28:31.322 PDT <Quorum Lost Notification> tid=0xada] Possible loss of quorum due to the loss of 2 cache processes: [cc8-rh64(16364)<v658>:1121, cc8-rh64(sleeper:16419)<v680>:30890]
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 992
> [fatal 2015/10/29 06:28:31.323 PDT <Quorum Lost Notification> tid=0xe04] Possible loss of quorum due to the loss of 2 cache processes: [cc8-rh64(16364)<v658>:1121, cc8-rh64(sleeper:16419)<v680>:30890]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)