You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Kirk Lund (JIRA)" <ji...@apache.org> on 2016/01/08 00:10:39 UTC
[jira] [Commented] (GEODE-767) CI failure:
GMSJoinLeaveJUnitTest.testNoViewAckCausesRemovalMessage
[ https://issues.apache.org/jira/browse/GEODE-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088349#comment-15088349 ]
Kirk Lund commented on GEODE-767:
---------------------------------
Standard Output
{noformat}
[info 2016/01/05 16:22:44.340 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:44.349 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8891]
old view is: null
[info 2016/01/05 16:22:44.349 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:44.349 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:44.351 PST <Geode Membership View Creator> tid=0x209] View Creator thread is starting
[info 2016/01/05 16:22:45.354 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:45.354 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8887 from cc8-rh64(22157)<ec>:8889 reason=testing
[info 2016/01/05 16:22:46.356 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:46.357 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:46.357 PST <Geode Membership View Creator> tid=0x20c] View Creator thread is starting
[info 2016/01/05 16:22:46.357 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v12>:8887]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v12>:8887, cc8-rh64(22157)<ec><v12>:8887]
[info 2016/01/05 16:22:47.360 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:47.360 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8887{lead}, cc8-rh64(22157)<ec>:8887{lead}]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8887]
[info 2016/01/05 16:22:47.360 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:47.361 PST <Geode Membership View Creator> tid=0x20e] View Creator thread is starting
[info 2016/01/05 16:22:48.365 PST <Test worker> tid=0xa] Attempting to join the distributed system through coordinator cc8-rh64(22157)<ec>:8888 using address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:48.366 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:49.369 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:49.370 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:22:49.370 PST <Geode Membership View Creator> tid=0x212] View Creator thread is starting
[info 2016/01/05 16:22:49.371 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:49.372 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
[info 2016/01/05 16:22:49.373 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|3] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887] crashed: [cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890]
old view is: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887]
[warn 2016/01/05 16:22:49.373 PST <Test worker> tid=0xa] total weight lost in this view change is 40 of 65. Quorum has been lost!
[info 2016/01/05 16:22:50.376 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:50.376 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
[info 2016/01/05 16:22:50.377 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|3] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887] crashed: [cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890]
old view is: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887]
[warn 2016/01/05 16:22:50.377 PST <Test worker> tid=0xa] total weight lost in this view change is 40 of 65. Quorum has been lost!
[info 2016/01/05 16:22:51.381 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v1>:8887|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8891, cc8-rh64(22157)<ec><v1>:8887]
old view is: null
[info 2016/01/05 16:22:51.382 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec><v1>:8887
[info 2016/01/05 16:22:51.382 PST <Geode Membership View Creator> tid=0x217] View Creator thread is starting
[info 2016/01/05 16:22:51.388 PST <Geode Membership View Creator> tid=0x217] preparing new view View[cc8-rh64(22157)<ec><v1>:8887|7] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8891, cc8-rh64(22157)<ec><v1>:8887]
failure detection ports: -1 -1 -1 -1 -1
[info 2016/01/05 16:22:51.388 PST <Geode Membership View Creator> tid=0x217] finished waiting for responses to view preparation
[warn 2016/01/05 16:22:51.388 PST <Geode Membership View Creator> tid=0x217] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8891, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890]
[info 2016/01/05 16:22:51.408 PST <Geode View Creator verification thread 1> tid=0x218] checking state of member cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:51.408 PST <Geode View Creator verification thread 2> tid=0x219] checking state of member cc8-rh64(22157)<ec>:8891
[info 2016/01/05 16:22:51.408 PST <Geode View Creator verification thread 1> tid=0x218] member cc8-rh64(22157)<ec>:8888 failed availability check
[info 2016/01/05 16:22:51.408 PST <Geode View Creator verification thread 2> tid=0x219] member cc8-rh64(22157)<ec>:8891 failed availability check
[info 2016/01/05 16:22:53.392 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:54.394 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8891]
old view is: null
[info 2016/01/05 16:22:54.395 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8889 for cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:22:54.395 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8890 for cc8-rh64(22157)<ec>:8890
[info 2016/01/05 16:22:54.395 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|3] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8891]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8891]
[info 2016/01/05 16:22:54.395 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:54.395 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:54.396 PST <Geode Membership View Creator> tid=0x21f] View Creator thread is starting
[info 2016/01/05 16:22:55.399 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8891]
old view is: null
[info 2016/01/05 16:22:55.399 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8889 for cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:22:55.399 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8890 for cc8-rh64(22157)<ec>:8890
[info 2016/01/05 16:22:55.399 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:55.400 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:55.400 PST <Geode Membership View Creator> tid=0x221] View Creator thread is starting
[info 2016/01/05 16:22:56.403 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:56.404 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8889 for cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:56.404 PST <Test worker> tid=0xa] I am being told to leave the distributed system
[info 2016/01/05 16:22:57.406 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:57.407 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:22:57.407 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:22:57.407 PST <Geode Membership View Creator> tid=0x224] View Creator thread is starting
[info 2016/01/05 16:22:57.908 PST <Geode Membership View Creator> tid=0x224] View Creator is processing 1 requests for the next membership view
[info 2016/01/05 16:22:57.909 PST <Geode Membership View Creator> tid=0x224] preparing new view View[cc8-rh64(22157)<ec><v0>:8887|1] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec><v1>:8888]
failure detection ports: -1 -1
[info 2016/01/05 16:22:58.408 PST <Geode Membership View Creator> tid=0x224] finished waiting for responses to view preparation
[warn 2016/01/05 16:22:58.409 PST <Geode Membership View Creator> tid=0x224] these members failed to respond to the view change: [cc8-rh64(22157)<ec><v1>:8888]
[info 2016/01/05 16:22:58.410 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|3] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec>:8887, cc8-rh64(22157)<ec>:8891]
old view is: null
[info 2016/01/05 16:22:59.412 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:22:59.413 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:22:59.413 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8888 from cc8-rh64(22157)<ec>:8888 reason=testing
[info 2016/01/05 16:22:59.413 PST <Geode Membership View Creator> tid=0x228] View Creator thread is starting
[info 2016/01/05 16:22:59.413 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8888 from cc8-rh64(22157)<ec>:8888 reason=testing
[info 2016/01/05 16:22:59.416 PST <Geode Membership View Creator> tid=0x228] preparing new view View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1 -1 -1
[info 2016/01/05 16:23:04.419 PST <Geode Membership View Creator> tid=0x228] finished waiting for responses to view preparation
[warn 2016/01/05 16:23:04.419 PST <Geode Membership View Creator> tid=0x228] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:04.420 PST <Geode Membership View Creator> tid=0x228] member cc8-rh64(22157)<ec>:8888 is already scheduled for removal
[info 2016/01/05 16:23:04.420 PST <Geode View Creator verification thread 1> tid=0x229] checking state of member cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:23:04.421 PST <Geode View Creator verification thread 1> tid=0x229] member cc8-rh64(22157)<ec>:8889 failed availability check
[info 2016/01/05 16:23:04.421 PST <Geode Membership View Creator> tid=0x228] adding these unresponsive members to the crash-set for the next view: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:04.421 PST <Geode Membership View Creator> tid=0x228] received new view: View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec><v6>:8887{lead}, cc8-rh64(22157)<ec><v6>:8887{lead}] crashed: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:04.422 PST <Geode Membership View Creator> tid=0x228] cc8-rh64(22157)<ec>:8888 had a weight of 15
[info 2016/01/05 16:23:04.422 PST <Geode Membership View Creator> tid=0x228] cc8-rh64(22157)<ec>:8889 had a weight of 10
[warn 2016/01/05 16:23:04.422 PST <Geode Membership View Creator> tid=0x228] total weight lost in this view change is 25 of 45. Quorum has been lost!
[info 2016/01/05 16:23:04.422 PST <Geode Membership View Creator> tid=0x228] sending new view View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec><v6>:8887{lead}, cc8-rh64(22157)<ec><v6>:8887{lead}] crashed: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1
[info 2016/01/05 16:23:04.923 PST <Geode Membership View Creator> tid=0x228] View Creator is processing 1 requests for the next membership view
[info 2016/01/05 16:23:06.418 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:06.418 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:06.419 PST <Geode Membership View Creator> tid=0x22b] View Creator thread is starting
[info 2016/01/05 16:23:06.419 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec><v0>:8887, cc8-rh64(22157)<ec>:8891]
old view is: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
[info 2016/01/05 16:23:06.419 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8889 from cc8-rh64(22157)<ec><v0>:8887 reason=crashed
[info 2016/01/05 16:23:06.419 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8890 from cc8-rh64(22157)<ec><v0>:8887 reason=crashed
[info 2016/01/05 16:23:06.419 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8891 from cc8-rh64(22157)<ec><v0>:8887 reason=crashed
[info 2016/01/05 16:23:06.920 PST <Geode Membership View Creator> tid=0x22b] View Creator is processing 3 requests for the next membership view
[info 2016/01/05 16:23:06.921 PST <Geode Membership View Creator> tid=0x22b] cc8-rh64(22157)<ec>:8891 had a weight of 10
[info 2016/01/05 16:23:06.921 PST <Geode Membership View Creator> tid=0x22b] cc8-rh64(22157)<ec>:8889 had a weight of 10
[info 2016/01/05 16:23:06.921 PST <Geode Membership View Creator> tid=0x22b] cc8-rh64(22157)<ec>:8890 had a weight of 10
[warn 2016/01/05 16:23:06.921 PST <Geode Membership View Creator> tid=0x22b] total weight lost in this view change is 30 of 55. Quorum has been lost!
[info 2016/01/05 16:23:07.925 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:07.932 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8888 from cc8-rh64(22157)<ec>:8887 reason=removing for test
[info 2016/01/05 16:23:07.932 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:07.932 PST <Geode Membership View Creator> tid=0x22d] View Creator thread is starting
[info 2016/01/05 16:23:07.933 PST <Geode Membership View Creator> tid=0x22d] preparing new view View[cc8-rh64(22157)<ec><v12>:8887|12] members: [cc8-rh64(22157)<ec><v12>:8887{lead}] crashed: [cc8-rh64(22157)<ec>:8888]
failure detection ports: -1
[info 2016/01/05 16:23:07.934 PST <Geode Membership View Creator> tid=0x22d] finished waiting for responses to view preparation
[info 2016/01/05 16:23:07.934 PST <Geode Membership View Creator> tid=0x22d] received new view: View[cc8-rh64(22157)<ec><v12>:8887|12] members: [cc8-rh64(22157)<ec><v12>:8887{lead}] crashed: [cc8-rh64(22157)<ec>:8888]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v12>:8887]
[info 2016/01/05 16:23:07.934 PST <Geode Membership View Creator> tid=0x22d] cc8-rh64(22157)<ec>:8888 had a weight of 15
[warn 2016/01/05 16:23:07.934 PST <Geode Membership View Creator> tid=0x22d] total weight lost in this view change is 15 of 25. Quorum has been lost!
[info 2016/01/05 16:23:07.939 PST <Geode Membership View Creator> tid=0x22d] sending new view View[cc8-rh64(22157)<ec><v12>:8887|12] members: [cc8-rh64(22157)<ec><v12>:8887{lead}] crashed: [cc8-rh64(22157)<ec>:8888]
failure detection ports: -1
[info 2016/01/05 16:23:08.934 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:08.935 PST <Test worker> tid=0xa] Membership ignoring removal request for cc8-rh64(22157)<ec>:8889 from non-member cc8-rh64(22157)<ec>:9000
[info 2016/01/05 16:23:09.937 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:09.937 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:09.937 PST <Geode Membership View Creator> tid=0x230] View Creator thread is starting
[info 2016/01/05 16:23:09.939 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:10.941 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:10.941 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:10.942 PST <Geode Membership View Creator> tid=0x233] View Creator thread is starting
[info 2016/01/05 16:23:10.943 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:10.943 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:10.944 PST <Geode Membership View Creator> tid=0x235] View Creator thread is starting
[info 2016/01/05 16:23:10.948 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:10.948 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:10.948 PST <Geode Membership View Creator> tid=0x237] View Creator thread is starting
[info 2016/01/05 16:23:11.449 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|1] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890]
old view is: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
[info 2016/01/05 16:23:11.449 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8889 for cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:23:11.449 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8889 from cc8-rh64(22157)<ec><v0>:8887 reason=removing for test
[info 2016/01/05 16:23:16.454 PST <Geode Membership View Creator> tid=0x237] View Creator is processing 1 requests for the next membership view
[info 2016/01/05 16:23:16.455 PST <Geode Membership View Creator> tid=0x237] preparing new view View[cc8-rh64(22157)<ec><v0>:8887|2] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec>:8890] shutdown: [cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1
[info 2016/01/05 16:23:21.457 PST <Geode Membership View Creator> tid=0x237] finished waiting for responses to view preparation
[warn 2016/01/05 16:23:21.457 PST <Geode Membership View Creator> tid=0x237] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8890]
[info 2016/01/05 16:23:21.457 PST <Geode View Creator verification thread 1> tid=0x238] checking state of member cc8-rh64(22157)<ec>:8890
[info 2016/01/05 16:23:21.458 PST <Geode View Creator verification thread 1> tid=0x238] member cc8-rh64(22157)<ec>:8890 passed availability check
[info 2016/01/05 16:23:21.458 PST <Geode Membership View Creator> tid=0x237] received new view: View[cc8-rh64(22157)<ec><v0>:8887|2] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec>:8890] shutdown: [cc8-rh64(22157)<ec>:8889]
old view is: View[cc8-rh64(22157)<ec><v0>:8887|1] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890]
[info 2016/01/05 16:23:21.459 PST <Geode Membership View Creator> tid=0x237] sending new view View[cc8-rh64(22157)<ec><v0>:8887|2] members: [cc8-rh64(22157)<ec><v0>:8887{lead}, cc8-rh64(22157)<ec>:8890] shutdown: [cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1
[info 2016/01/05 16:23:22.955 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:22.955 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:22.955 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:22.955 PST <Geode Membership View Creator> tid=0x23a] View Creator thread is starting
[info 2016/01/05 16:23:22.955 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:22.956 PST <Geode Membership View Creator> tid=0x23a] preparing new view View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887]
failure detection ports: -1 -1 -1
[info 2016/01/05 16:23:22.957 PST <Geode Membership View Creator> tid=0x23a] finished waiting for responses to view preparation
[info 2016/01/05 16:23:22.957 PST <Geode Membership View Creator> tid=0x23a] received new view: View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887]
[info 2016/01/05 16:23:22.958 PST <Geode Membership View Creator> tid=0x23a] sending new view View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887]
failure detection ports: -1 -1 -1
[info 2016/01/05 16:23:23.459 PST <Geode Membership View Creator> tid=0x23a] View Creator is processing 2 requests for the next membership view
[info 2016/01/05 16:23:23.459 PST <Geode Membership View Creator> tid=0x23a] no recipients for new view aside from myself
[info 2016/01/05 16:23:23.459 PST <Geode Membership View Creator> tid=0x23a] received new view: View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec><v6>:8887{lead}, cc8-rh64(22157)<ec><v6>:8887{lead}] shutdown: [cc8-rh64(22157)<ec>:8888]
old view is: View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887]
[info 2016/01/05 16:23:23.459 PST <Geode Membership View Creator> tid=0x23a] no recipients for new view aside from myself
[info 2016/01/05 16:23:24.957 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8700
[warn 2016/01/05 16:23:24.958 PST <Test worker> tid=0xa] detected an attempt to start a peer using an older version of the product cc8-rh64(22157)<ec>:8700
[info 2016/01/05 16:23:24.959 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:24.959 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec><v0>:8889 for cc8-rh64(22157)<ec><v0>:8889
[info 2016/01/05 16:23:25.961 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:25.961 PST <Test worker> tid=0xa] Membership received a request to remove cc8-rh64(22157)<ec>:8888 from cc8-rh64(22157)<ec>:8888 reason=testing
[info 2016/01/05 16:23:25.961 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:25.962 PST <Geode Membership View Creator> tid=0x23e] View Creator thread is starting
[info 2016/01/05 16:23:26.964 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:26.964 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:26.964 PST <Geode Membership View Creator> tid=0x240] View Creator thread is starting
[info 2016/01/05 16:23:26.964 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec>:8890, cc8-rh64(22157)<ec><v0>:8887, cc8-rh64(22157)<ec>:8891]
old view is: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
[info 2016/01/05 16:23:28.967 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:28.969 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:28.969 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec><v0>:8887|0] members: [cc8-rh64(22157)<ec><v0>:8887{lead}]
old view is: null
[info 2016/01/05 16:23:28.969 PST <Geode Membership View Creator> tid=0x243] View Creator thread is starting
[info 2016/01/05 16:23:28.971 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:28.971 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|2] members: [cc8-rh64(22157)<ec>:8887{lead}, cc8-rh64(22157)<ec>:8890]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
[info 2016/01/05 16:23:28.971 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:28.972 PST <Geode Membership View Creator> tid=0x245] View Creator thread is starting
[info 2016/01/05 16:23:28.975 PST <Geode Membership View Creator> tid=0x245] preparing new view View[cc8-rh64(22157)<ec><v9>:8887|9] members: [cc8-rh64(22157)<ec><v9>:8887{lead}, cc8-rh64(22157)<ec>:8890]
failure detection ports: -1 -1
[info 2016/01/05 16:23:28.977 PST <Geode Membership View Creator> tid=0x245] finished waiting for responses to view preparation
[warn 2016/01/05 16:23:28.977 PST <Geode Membership View Creator> tid=0x245] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8890]
[info 2016/01/05 16:23:29.980 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:29.981 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:29.981 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8890
[info 2016/01/05 16:23:29.981 PST <Geode Membership View Creator> tid=0x248] View Creator thread is starting
[info 2016/01/05 16:23:29.981 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8890
[info 2016/01/05 16:23:29.982 PST <Geode Membership View Creator> tid=0x248] preparing new view View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1 -1 -1
[info 2016/01/05 16:23:34.983 PST <Geode Membership View Creator> tid=0x248] finished waiting for responses to view preparation
[warn 2016/01/05 16:23:34.983 PST <Geode Membership View Creator> tid=0x248] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:34.984 PST <Geode View Creator verification thread 1> tid=0x249] checking state of member cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:34.984 PST <Geode View Creator verification thread 2> tid=0x24a] checking state of member cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:23:34.984 PST <Geode View Creator verification thread 1> tid=0x249] member cc8-rh64(22157)<ec>:8888 passed availability check
[info 2016/01/05 16:23:34.985 PST <Geode View Creator verification thread 2> tid=0x24a] member cc8-rh64(22157)<ec>:8889 passed availability check
[info 2016/01/05 16:23:34.985 PST <Geode Membership View Creator> tid=0x248] received new view: View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:34.986 PST <Geode Membership View Creator> tid=0x248] sending new view View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
failure detection ports: -1 -1 -1 -1
[info 2016/01/05 16:23:35.487 PST <Geode Membership View Creator> tid=0x248] View Creator is processing 2 requests for the next membership view
[info 2016/01/05 16:23:35.487 PST <Geode Membership View Creator> tid=0x248] preparing new view View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec><v7>:8890]
failure detection ports: -1 -1 -1 -1 -1
[info 2016/01/05 16:23:40.488 PST <Geode Membership View Creator> tid=0x248] finished waiting for responses to view preparation
[warn 2016/01/05 16:23:40.488 PST <Geode Membership View Creator> tid=0x248] these members failed to respond to the view change: [cc8-rh64(22157)<ec>:8888, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec><v7>:8890]
[info 2016/01/05 16:23:40.489 PST <Geode View Creator verification thread 1> tid=0x24b] checking state of member cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:40.489 PST <Geode View Creator verification thread 2> tid=0x24c] checking state of member cc8-rh64(22157)<ec>:8889
[info 2016/01/05 16:23:40.489 PST <Geode View Creator verification thread 3> tid=0x24d] checking state of member cc8-rh64(22157)<ec><v7>:8890
[info 2016/01/05 16:23:40.490 PST <Geode View Creator verification thread 1> tid=0x24b] member cc8-rh64(22157)<ec>:8888 passed availability check
[info 2016/01/05 16:23:40.490 PST <Geode View Creator verification thread 3> tid=0x24d] member cc8-rh64(22157)<ec><v7>:8890 passed availability check
[info 2016/01/05 16:23:40.490 PST <Geode View Creator verification thread 2> tid=0x24c] member cc8-rh64(22157)<ec>:8889 passed availability check
[info 2016/01/05 16:23:40.491 PST <Geode Membership View Creator> tid=0x248] received new view: View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec><v7>:8890]
old view is: View[cc8-rh64(22157)<ec><v6>:8887|6] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889]
[info 2016/01/05 16:23:40.491 PST <Geode Membership View Creator> tid=0x248] sending new view View[cc8-rh64(22157)<ec><v6>:8887|7] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec><v6>:8887, cc8-rh64(22157)<ec>:8889, cc8-rh64(22157)<ec><v7>:8890]
failure detection ports: -1 -1 -1 -1 -1
[info 2016/01/05 16:23:41.985 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:41.986 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:41.986 PST <Geode Membership View Creator> tid=0x24f] View Creator thread is starting
[info 2016/01/05 16:23:41.987 PST <Geode Membership View Creator> tid=0x24f] preparing new view View[cc8-rh64(22157)<ec><v15>:8887|16] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v15>:8887, cc8-rh64(22157)<ec><v6>:8889]
failure detection ports: -1 -1 -1
[info 2016/01/05 16:23:43.986 PST <Geode Membership View Creator> tid=0x24f] finished waiting for responses to view preparation
[info 2016/01/05 16:23:43.987 PST <Geode Membership View Creator> tid=0x24f] received new view: View[cc8-rh64(22157)<ec><v15>:8887|16] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v15>:8887, cc8-rh64(22157)<ec><v6>:8889]
old view is: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v15>:8887]
[info 2016/01/05 16:23:43.987 PST <Geode Membership View Creator> tid=0x24f] sending new view View[cc8-rh64(22157)<ec><v15>:8887|16] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v15>:8887, cc8-rh64(22157)<ec><v6>:8889]
failure detection ports: -1 -1 -1
new view is View[cc8-rh64(22157)<ec><v15>:8887|16] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec><v15>:8887, cc8-rh64(22157)<ec><v6>:8889]
[info 2016/01/05 16:23:45.988 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
[info 2016/01/05 16:23:45.989 PST <Test worker> tid=0xa] received leave request from cc8-rh64(22157)<ec>:8888 for cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:45.989 PST <Test worker> tid=0xa] This member is becoming the membership coordinator with address cc8-rh64(22157)<ec>:8887
[info 2016/01/05 16:23:45.989 PST <Geode Membership View Creator> tid=0x251] View Creator thread is starting
[info 2016/01/05 16:23:46.991 PST <Test worker> tid=0xa] received join request from cc8-rh64(22157)<ec>:8888
[info 2016/01/05 16:23:46.995 PST <Test worker> tid=0xa] received new view: View[cc8-rh64(22157)<ec>:8888|1] members: [cc8-rh64(22157)<ec>:8888{lead}, cc8-rh64(22157)<ec>:8887]
old view is: null
{noformat}
Standard Error
{noformat}
com.gemstone.gemfire.security.AuthenticationFailedException: we want to fail auth here
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processJoinRequest(GMSJoinLeave.java:409)
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1439)
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeaveJUnitTest.testProcessJoinMessageWithAuthenticationButNullCredentials(GMSJoinLeaveJUnitTest.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.gemstone.gemfire.security.AuthenticationFailedException: we want to fail auth here
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processJoinRequest(GMSJoinLeave.java:409)
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1439)
at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeaveJUnitTest.testProcessJoinMessageWithBadAuthentication(GMSJoinLeaveJUnitTest.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
> CI failure: GMSJoinLeaveJUnitTest.testNoViewAckCausesRemovalMessage
> -------------------------------------------------------------------
>
> Key: GEODE-767
> URL: https://issues.apache.org/jira/browse/GEODE-767
> Project: Geode
> Issue Type: Bug
> Components: membership
> Reporter: Kirk Lund
>
> {noformat}
> org.mockito.exceptions.verification.TooManyActualInvocations:
> messenger.send(<any>);
> Wanted 2 times:
> -> at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeaveJUnitTest.testNoViewAckCausesRemovalMessage(GMSJoinLeaveJUnitTest.java:742)
> But was 3 times. Undesired invocation:
> -> at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeave.sendView(GMSJoinLeave.java:736)
> at com.gemstone.gemfire.distributed.internal.membership.gms.membership.GMSJoinLeaveJUnitTest.testNoViewAckCausesRemovalMessage(GMSJoinLeaveJUnitTest.java:742)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)