You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/05/22 23:06:25 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #
12782 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
Error Message:
Some resources were not closed, shutdown, or released.
Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
Error Message:
expected:<st[art]ed> but was:<st[opp]ed>
Stack Trace:
org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A:D0B02EE63F74C6D3]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10551 lines...]
[junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/init-core-data-001
[junit4] 2> 757362 T4034 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 757362 T4034 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 757364 T4034 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 757364 T4035 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 757364 T4035 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 757464 T4034 oasc.ZkTestServer.run start zk server on port:54300
[junit4] 2> 757465 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 757465 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 757467 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a254558 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 757468 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 757468 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 757468 T4034 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 757469 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 757470 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 757471 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34362c7f name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 757471 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 757471 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 757471 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 757472 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 757474 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 757474 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 757475 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 757475 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 757476 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 757476 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 757477 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 757478 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 757479 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 757479 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 757480 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 757480 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 757481 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 757482 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 757482 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 757483 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 757484 T4034 oas.SolrTestCaseJ4.setUp ###Starting doTest
[junit4] 2> 757533 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1
[junit4] 2> 757534 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 757535 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@76300627{/,null,AVAILABLE}
[junit4] 2> 757537 T4034 oejs.AbstractConnector.doStart Started ServerConnector@93aadc7{HTTP/1.1}{127.0.0.1:42803}
[junit4] 2> 757537 T4034 oejs.Server.doStart Started @758455ms
[junit4] 2> 757537 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=42803, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores}
[junit4] 2> 757538 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 757538 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/'
[junit4] 2> 757547 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/solr.xml
[junit4] 2> 757550 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
[junit4] 2> 757550 T4034 oasc.CoreContainer.<init> New CoreContainer 1191155484
[junit4] 2> 757550 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/]
[junit4] 2> 757551 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib
[junit4] 2> 757551 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib).
[junit4] 2> 757555 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 757556 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 757556 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 757556 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 757556 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 757557 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
[junit4] 2> 757557 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 757557 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 757557 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 757559 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@599a8e77 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 757559 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 757559 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 757560 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 757561 T4062 n:127.0.0.1:42803_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7452961a name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 757561 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 757562 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 757563 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 757564 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 757565 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 757566 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 757567 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42803_
[junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42803_
[junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 757571 T4034 n:127.0.0.1:42803_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42803_
[junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 757573 T4034 n:127.0.0.1:42803_ oasc.Overseer.start Overseer (id=93869083152809987-127.0.0.1:42803_-n_0000000000) starting
[junit4] 2> 757574 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 757576 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 757576 T4064 n:127.0.0.1:42803_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 757579 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
[junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/
[junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 757581 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 757581 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42803",
[junit4] 2> "node_name":"127.0.0.1:42803_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42803",
[junit4] 2> "node_name":"127.0.0.1:42803_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
[junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 757583 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
[junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Looking for collection configName
[junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
[junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/'
[junit4] 2> 758598 T4066 n:127.0.0.1:42803_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 758600 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 758603 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 758608 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 758609 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 758611 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 758665 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 758668 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 758673 T4066 n:127.0.0.1:42803_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/], dataDir=[null]
[junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
[junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data
[junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index/
[junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index
[junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
[junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 758677 T4066 n:127.0.0.1:42803_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 758686 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 758687 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 758696 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@d198739[collection1] main
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905331851100160
[junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905331851100160
[junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 758712 T4066 n:127.0.0.1:42803_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
[junit4] 2> 758714 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 758716 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
[junit4] 2> 758726 T4066 n:127.0.0.1:42803_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 758727 T4067 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d198739[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 758727 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42803 collection:tmp_collection shard:shard2
[junit4] 2> 758727 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 758728 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 758728 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
[junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 758730 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 758731 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@16e04228 url=http://127.0.0.1:42803/collection1 node=127.0.0.1:42803_ C983_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42803, node_name=127.0.0.1:42803_, state=down}
[junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42803/collection1/
[junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncToMe http://127.0.0.1:42803/collection1/ has no replicas
[junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42803/collection1/ shard2
[junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
[junit4] 2> 758732 T4062 n:127.0.0.1:42803_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
[junit4] 2> 758733 T4072 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 758734 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:42803",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 758778 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1
[junit4] 2> 758779 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 758780 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@12138b09{/,null,AVAILABLE}
[junit4] 2> 758781 T4034 oejs.AbstractConnector.doStart Started ServerConnector@537374e0{HTTP/1.1}{127.0.0.1:47447}
[junit4] 2> 758781 T4034 oejs.Server.doStart Started @759698ms
[junit4] 2> 758781 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=47447, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores}
[junit4] 2> 758781 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 758781 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/'
[junit4] 2> 758790 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/solr.xml
[junit4] 2> 758793 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
[junit4] 2> 758794 T4034 oasc.CoreContainer.<init> New CoreContainer 957201046
[junit4] 2> 758794 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/]
[junit4] 2> 758794 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib
[junit4] 2> 758794 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib).
[junit4] 2> 758798 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 758799 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 758799 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 758799 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 758799 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 758800 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
[junit4] 2> 758800 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 758800 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 758800 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 758802 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7f7cd8 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 758802 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 758802 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 758803 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 758804 T4089 n:127.0.0.1:47447_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc03e94 name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 758805 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 758807 T4034 n:127.0.0.1:47447_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 758835 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758835 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:42803/collection1/ and leader is http://127.0.0.1:42803/collection1/
[junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42803
[junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 758885 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 758885 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42803",
[junit4] 2> "node_name":"127.0.0.1:42803_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 758886 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42803",
[junit4] 2> "node_name":"127.0.0.1:42803_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 758987 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 758987 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 759808 T4034 n:127.0.0.1:47447_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47447_
[junit4] 2> 759809 T4034 n:127.0.0.1:47447_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47447_
[junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
[junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/
[junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 759813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 759813 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:47447",
[junit4] 2> "node_name":"127.0.0.1:47447_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:47447",
[junit4] 2> "node_name":"127.0.0.1:47447_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 759915 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 759916 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 760813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/'
[junit4] 2> 760826 T4090 n:127.0.0.1:47447_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 760829 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 760831 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 760840 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 760892 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 760893 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 760894 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 760895 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 760900 T4090 n:127.0.0.1:47447_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 760900 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/], dataDir=[null]
[junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
[junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data
[junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index/
[junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index
[junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
[junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 760904 T4090 n:127.0.0.1:47447_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 760905 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 760908 T4090 n:127.0.0.1:47447_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 760909 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5da2fafa[collection1] main
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905334165307392
[junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905334165307392
[junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 760917 T4090 n:127.0.0.1:47447_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 760918 T4091 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5da2fafa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47447 collection:tmp_collection shard:shard1
[junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
[junit4] 2> 760920 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 760921 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@4e21ece7 url=http://127.0.0.1:47447/collection1 node=127.0.0.1:47447_ C984_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:47447, node_name=127.0.0.1:47447_, state=down}
[junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47447/collection1/
[junit4] 2> 760921 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection"} current state version: 4
[junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncToMe http://127.0.0.1:47447/collection1/ has no replicas
[junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47447/collection1/ shard1
[junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
[junit4] 2> 760922 T4089 n:127.0.0.1:47447_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
[junit4] 2> 760923 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 760924 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:47447",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 760968 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores/collection1
[junit4] 2> 760970 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 760971 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4fc5b5e4{/,null,AVAILABLE}
[junit4] 2> 760971 T4034 oejs.AbstractConnector.doStart Started ServerConnector@41389b97{HTTP/1.1}{127.0.0.1:36397}
[junit4] 2> 760971 T4034 oejs.Server.doStart Started @761888ms
[junit4] 2> 760971 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=36397, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores}
[junit4] 2> 760972 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 760972 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/'
[junit4] 2> 760980 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/solr.xml
[junit4] 2> 760983 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores
[junit4] 2> 760984 T4034 oasc.CoreContainer.<init> New CoreContainer 327774697
[junit4] 2> 760984 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/]
[junit4] 2> 760984 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib
[junit4] 2> 760984 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib).
[junit4] 2> 760988 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 760988 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 760989 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 760989 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 760989 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 760989 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
[junit4] 2> 760990 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 760990 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 760990 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 760992 T4109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21f437e1 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 760992 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 760992 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 760994 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 760995 T4112 n:127.0.0.1:36397_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53fdaf5b name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 760995 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 760997 T4034 n:127.0.0.1:36397_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 761025 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761025 T4112 n:127.0.0.1:36397_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761025 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:47447/collection1/ and leader is http://127.0.0.1:47447/collection1/
[junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47447
[junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 761074 T4095 n:127.0.0.1:474
[...truncated too long message...]
I
[junit4] ERROR 0.00s J0 | CdcrReplicationDistributedZkTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: Some resources were not closed, shutdown, or released.
[junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
[junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
[junit4] Completed [381/502] on J0 in 167.35s, 1 test, 2 failures, 2 errors <<< FAILURES!
[...truncated 376 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1985 tests, 3 suite-level errors, 1 failure, 316 ignored (25 assumptions)
Total time: 32 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Mark Miller <ma...@gmail.com>.
Hit this a while ago, simply cleaned up and it went away...
On Sun, Jun 14, 2015 at 9:34 PM Noble Paul <no...@gmail.com> wrote:
> This is a problem still.
>
> I made some changes to my trunk and I get this error always
>
> java.lang.NoSuchFieldError: totalTermCount
>
>
> This gives a false alarm and we waste time. We need to get to the
> bottom of this
>
> On Tue, May 26, 2015 at 8:58 PM, Erick Erickson <er...@gmail.com>
> wrote:
> > Yep. I found this while trying to deal with CDCR, which is only in trunk.
> >
> >
> >
> > On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com>
> wrote:
> >> Do you mean in trunk ?
> >>
> >> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <
> erickerickson@gmail.com> wrote:
> >>> The problem went away for me. I took sledgehammer approach and deleted
> >>> the entire tree locally and checked out the tree fresh.
> >>>
> >>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com>
> wrote:
> >>>> I'v edone a clean checkout I still see these errors
> >>>>
> >>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
> >>>>
> >>>> <title>Error 500 Server Error</title>
> >>>>
> >>>> </head>
> >>>>
> >>>> <body><h2>HTTP ERROR 500</h2>
> >>>>
> >>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
> >>>>
> >>>> <pre> Server Error</pre></p><h3>Caused
> >>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
> >>>>
> >>>> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> >>>>
> >>>> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> >>>>
> >>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> >>>>
> >>>> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> >>>>
> >>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
> >>>> <er...@gmail.com> wrote:
> >>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
> >>>>> everything just fine, my laptop fails all the time.
> >>>>>
> >>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
> >>>>> <ic...@gmail.com> wrote:
> >>>>>> Not sure if it is related or helpful, but while debugging tests for
> >>>>>> SOLR-7468 yesterday, I encountered this
> >>>>>> java.lang.NoSuchFieldError:
> >>>>>> totalTermCount
> >>>>>>
> >>>>>> few times, I had to forcefully clean at root of the project and it
> worked. I
> >>>>>> remember Anshum had to do that clean thing more than once to make
> it work
> >>>>>> and he remarked "don't ask why".
> >>>>>>
> >>>>>> Sent from my Windows Phone
> >>>>>> ________________________________
> >>>>>> From: Erick Erickson
> >>>>>> Sent: ‎5/‎23/‎2015 6:15 AM
> >>>>>> To: dev@lucene.apache.org
> >>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45)
> - Build
> >>>>>> #12782 - Failure!
> >>>>>>
> >>>>>> OK, this is somewhat weird. I still have the original tree that I
> >>>>>> checked in from which was up to date before I committed the code and
> >>>>>> the tests run from there fine. But a current trunk fails every time.
> >>>>>> Now, the machine it works on is my Mac Pro, and the failures are on
> my
> >>>>>> MacBook so there may be something going on there.
> >>>>>>
> >>>>>> I've got to leave for a while, I'll copy the tree that works on the
> >>>>>> Pro, update the copy and see if this test fails when I get back. If
> >>>>>> they fail, I can diff the trees to see what changed and see if I can
> >>>>>> make any sense out of this.
> >>>>>>
> >>>>>> I can always @Ignore this test to cut down on the noise, probably do
> >>>>>> that tonight if I don't have any revelations.
> >>>>>>
> >>>>>> I see this stack trace which makes no sense to me whatsoever (see
> the
> >>>>>> lines with lots of ***** in front). I looked at where the code
> >>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
> >>>>>>
> >>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount !=
> 0) {
> >>>>>>
> >>>>>> And it's telling me there's no such field? Whaaaaatttt????
> >>>>>>
> >>>>>> Which is freaking me out since I don't see how this would trigger
> the
> >>>>>> exception. Is this a red herring? And, of course, this doesn't fail
> in
> >>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
> >>>>>>
> >>>>>> Of course if this were something fundamental to Lucene, it seems
> like
> >>>>>> this would be failing all over the place so I assume it's something
> to
> >>>>>> do with CDCR... But what do I know?
> >>>>>>
> >>>>>>
> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> >>>>>> status=0 QTime=8
> >>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
> >>>>>> c:source_collection s:shard1 r:core_node3
> >>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log
> ERROR
> >>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
> >>>>>> totalTermCount
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> >>>>>> [junit4] 2> at
> org.eclipse.jetty.server.Server.handle(Server.java:497)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> >>>>>> [junit4] 2> at
> >>>>>> org.eclipse.jetty.io
> .AbstractConnection$2.run(AbstractConnection.java:540)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> >>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
> >>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError:
> totalTermCount
> >>>>>> ************* [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> >>>>>> [junit4] 2> at
> >>>>>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
> >>>>>> [junit4] 2> at
> >>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
> >>>>>> [junit4] 2> ... 23 more
> >>>>>>
> >>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <
> erickerickson@gmail.com>
> >>>>>> wrote:
> >>>>>>> Looking......
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> >>>>>>> <je...@thetaphi.de> wrote:
> >>>>>>>> Build:
> http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
> >>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops
> -XX:+UseConcMarkSweepGC
> >>>>>>>>
> >>>>>>>> 4 tests failed.
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> Some resources were not closed, shutdown, or released.
> >>>>>>>>
> >>>>>>>> Stack Trace:
> >>>>>>>> java.lang.AssertionError: Some resources were not closed,
> shutdown, or
> >>>>>>>> released.
> >>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> >>>>>>>> at org.junit.Assert.fail(Assert.java:93)
> >>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
> >>>>>>>> at
> >>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
> >>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown
> Source)
> >>>>>>>> at
> >>>>>>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> >>>>>>>> at
> >>>>>>>>
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >>>>>>>> at
> >>>>>>>>
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> 5 threads leaked from SUITE scope at
> >>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
> >>>>>>>> Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING,
> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> java.lang.Thread.sleep(Native Method) at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
> >>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-EventThread,
> >>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
> >>>>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>>>>> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> >>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
> >>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>>
> >>>>>>>> Stack Trace:
> >>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads
> leaked from
> >>>>>>>> SUITE scope at
> org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
> >>>>>>>> 1) Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at java.lang.Thread.sleep(Native Method)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
> >>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> 3) Thread[id=4676,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-EventThread,
> >>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> >>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
> >>>>>>>> at sun.misc.Unsafe.park(Native Method)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745)
> >>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> FAILED:
> >>>>>>>> junit.framework.TestSuite.org
> .apache.solr.cloud.CdcrReplicationDistributedZkTest
> >>>>>>>>
> >>>>>>>> Error Message:
> >>>>>>>> There are still zombie threads that couldn't be terminated: 1)
> >>>>>>>> Thread[id=4675,
> >>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803
> _}-SendThread(127.0.0.1:54300),
> >>>>>>>> state=TIMED_WAITING,
> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> java.lang.Thread.sleep(Native Method) at
> >>>>>>>>
> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> >>>>>>>> at
> >>>>>>>>
> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> >>>>>>>> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> >>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2,
> state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
> >>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
> >>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
> >>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
> >>>>>>>> sun.misc.Unsafe.park(Native Method) at
> >>>>>>>>
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> >>>>>>>> at
> >>>>>>>>
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> >>>>>>>> at java.ut
> >>>>>>
> >>>>>> [The entire original message is not included.]
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> -----------------------------------------------------
> >>>> Noble Paul
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: dev-help@lucene.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >> -----------------------------------------------------
> >> Noble Paul
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
> --
- Mark
about.me/markrmiller
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
This is a problem still.
I made some changes to my trunk and I get this error always
java.lang.NoSuchFieldError: totalTermCount
This gives a false alarm and we waste time. We need to get to the
bottom of this
On Tue, May 26, 2015 at 8:58 PM, Erick Erickson <er...@gmail.com> wrote:
> Yep. I found this while trying to deal with CDCR, which is only in trunk.
>
>
>
> On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com> wrote:
>> Do you mean in trunk ?
>>
>> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>> The problem went away for me. I took sledgehammer approach and deleted
>>> the entire tree locally and checked out the tree fresh.
>>>
>>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>>>> I'v edone a clean checkout I still see these errors
>>>>
>>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>>>
>>>> <title>Error 500 Server Error</title>
>>>>
>>>> </head>
>>>>
>>>> <body><h2>HTTP ERROR 500</h2>
>>>>
>>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>>>
>>>> <pre> Server Error</pre></p><h3>Caused
>>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>>>
>>>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>
>>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>
>>>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>
>>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>>>> <er...@gmail.com> wrote:
>>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>>>> everything just fine, my laptop fails all the time.
>>>>>
>>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>>>> <ic...@gmail.com> wrote:
>>>>>> Not sure if it is related or helpful, but while debugging tests for
>>>>>> SOLR-7468 yesterday, I encountered this
>>>>>> java.lang.NoSuchFieldError:
>>>>>> totalTermCount
>>>>>>
>>>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>>>> remember Anshum had to do that clean thing more than once to make it work
>>>>>> and he remarked "don't ask why".
>>>>>>
>>>>>> Sent from my Windows Phone
>>>>>> ________________________________
>>>>>> From: Erick Erickson
>>>>>> Sent: ‎5/‎23/‎2015 6:15 AM
>>>>>> To: dev@lucene.apache.org
>>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>>>> #12782 - Failure!
>>>>>>
>>>>>> OK, this is somewhat weird. I still have the original tree that I
>>>>>> checked in from which was up to date before I committed the code and
>>>>>> the tests run from there fine. But a current trunk fails every time.
>>>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>>>> MacBook so there may be something going on there.
>>>>>>
>>>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>>>> Pro, update the copy and see if this test fails when I get back. If
>>>>>> they fail, I can diff the trees to see what changed and see if I can
>>>>>> make any sense out of this.
>>>>>>
>>>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>>>> that tonight if I don't have any revelations.
>>>>>>
>>>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>>>> lines with lots of ***** in front). I looked at where the code
>>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>>>
>>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>>>
>>>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>>>
>>>>>> Which is freaking me out since I don't see how this would trigger the
>>>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>>>
>>>>>> Of course if this were something fundamental to Lucene, it seems like
>>>>>> this would be failing all over the place so I assume it's something to
>>>>>> do with CDCR... But what do I know?
>>>>>>
>>>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>>>> status=0 QTime=8
>>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>>>> c:source_collection s:shard1 r:core_node3
>>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>>>> totalTermCount
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>>> [junit4] 2> at
>>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>>>> ************* [junit4] 2> at
>>>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>>> [junit4] 2> at
>>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>>>> [junit4] 2> at
>>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>>>> [junit4] 2> ... 23 more
>>>>>>
>>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>>>> wrote:
>>>>>>> Looking......
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>>>> <je...@thetaphi.de> wrote:
>>>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>>>
>>>>>>>> 4 tests failed.
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>>>
>>>>>>>> Stack Trace:
>>>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>>>> released.
>>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>>>> at
>>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>>>> at
>>>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>>> at
>>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> 5 threads leaked from SUITE scope at
>>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>>>> Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>>> at
>>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>> Stack Trace:
>>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>>>> 1) Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> 3) Thread[id=4676,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>>> at
>>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>> at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>>
>>>>>>>>
>>>>>>>> FAILED:
>>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>>
>>>>>>>> Error Message:
>>>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>>>> Thread[id=4675,
>>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>>>> at
>>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>>> at
>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>>> at java.ut
>>>>>>
>>>>>> [The entire original message is not included.]
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
Yep. I found this while trying to deal with CDCR, which is only in trunk.
On Mon, May 25, 2015 at 11:51 PM, Noble Paul <no...@gmail.com> wrote:
> Do you mean in trunk ?
>
> On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
>> The problem went away for me. I took sledgehammer approach and deleted
>> the entire tree locally and checked out the tree fresh.
>>
>> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>>> I'v edone a clean checkout I still see these errors
>>>
>>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>>
>>> <title>Error 500 Server Error</title>
>>>
>>> </head>
>>>
>>> <body><h2>HTTP ERROR 500</h2>
>>>
>>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>>
>>> <pre> Server Error</pre></p><h3>Caused
>>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>>
>>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>
>>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>
>>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>
>>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>
>>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>
>>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>
>>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>
>>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>>> <er...@gmail.com> wrote:
>>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>>> everything just fine, my laptop fails all the time.
>>>>
>>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>>> <ic...@gmail.com> wrote:
>>>>> Not sure if it is related or helpful, but while debugging tests for
>>>>> SOLR-7468 yesterday, I encountered this
>>>>> java.lang.NoSuchFieldError:
>>>>> totalTermCount
>>>>>
>>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>>> remember Anshum had to do that clean thing more than once to make it work
>>>>> and he remarked "don't ask why".
>>>>>
>>>>> Sent from my Windows Phone
>>>>> ________________________________
>>>>> From: Erick Erickson
>>>>> Sent: ‎5/‎23/‎2015 6:15 AM
>>>>> To: dev@lucene.apache.org
>>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>>> #12782 - Failure!
>>>>>
>>>>> OK, this is somewhat weird. I still have the original tree that I
>>>>> checked in from which was up to date before I committed the code and
>>>>> the tests run from there fine. But a current trunk fails every time.
>>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>>> MacBook so there may be something going on there.
>>>>>
>>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>>> Pro, update the copy and see if this test fails when I get back. If
>>>>> they fail, I can diff the trees to see what changed and see if I can
>>>>> make any sense out of this.
>>>>>
>>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>>> that tonight if I don't have any revelations.
>>>>>
>>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>>> lines with lots of ***** in front). I looked at where the code
>>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>>
>>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>>
>>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>>
>>>>> Which is freaking me out since I don't see how this would trigger the
>>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>>
>>>>> Of course if this were something fundamental to Lucene, it seems like
>>>>> this would be failing all over the place so I assume it's something to
>>>>> do with CDCR... But what do I know?
>>>>>
>>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>>> status=0 QTime=8
>>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>>> c:source_collection s:shard1 r:core_node3
>>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>>> totalTermCount
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>>> [junit4] 2> at
>>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>>> ************* [junit4] 2> at
>>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>>> [junit4] 2> at
>>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>>> [junit4] 2> at
>>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>>> [junit4] 2> ... 23 more
>>>>>
>>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>>> wrote:
>>>>>> Looking......
>>>>>>
>>>>>>
>>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>>> <je...@thetaphi.de> wrote:
>>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>>
>>>>>>> 4 tests failed.
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>>
>>>>>>> Stack Trace:
>>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>>> released.
>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>>> at
>>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>>> at
>>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>>> at
>>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>>> at
>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>>
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> 5 threads leaked from SUITE scope at
>>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>>> Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>> at
>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> Stack Trace:
>>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>>> 1) Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 3) Thread[id=4676,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>>> at
>>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>> at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>>
>>>>>>>
>>>>>>> FAILED:
>>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>>
>>>>>>> Error Message:
>>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>>> Thread[id=4675,
>>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>>> at
>>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>>> at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>>> at
>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>>> at java.ut
>>>>>
>>>>> [The entire original message is not included.]
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
Do you mean in trunk ?
On Mon, May 25, 2015 at 9:53 PM, Erick Erickson <er...@gmail.com> wrote:
> The problem went away for me. I took sledgehammer approach and deleted
> the entire tree locally and checked out the tree fresh.
>
> On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
>> I'v edone a clean checkout I still see these errors
>>
>> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>>
>> <title>Error 500 Server Error</title>
>>
>> </head>
>>
>> <body><h2>HTTP ERROR 500</h2>
>>
>> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>>
>> <pre> Server Error</pre></p><h3>Caused
>> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>>
>> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>
>> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>
>> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>
>> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>
>> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>
>> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>
>> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>
>> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
>> <er...@gmail.com> wrote:
>>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>>> everything just fine, my laptop fails all the time.
>>>
>>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>>> <ic...@gmail.com> wrote:
>>>> Not sure if it is related or helpful, but while debugging tests for
>>>> SOLR-7468 yesterday, I encountered this
>>>> java.lang.NoSuchFieldError:
>>>> totalTermCount
>>>>
>>>> few times, I had to forcefully clean at root of the project and it worked. I
>>>> remember Anshum had to do that clean thing more than once to make it work
>>>> and he remarked "don't ask why".
>>>>
>>>> Sent from my Windows Phone
>>>> ________________________________
>>>> From: Erick Erickson
>>>> Sent: ‎5/‎23/‎2015 6:15 AM
>>>> To: dev@lucene.apache.org
>>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>>> #12782 - Failure!
>>>>
>>>> OK, this is somewhat weird. I still have the original tree that I
>>>> checked in from which was up to date before I committed the code and
>>>> the tests run from there fine. But a current trunk fails every time.
>>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>>> MacBook so there may be something going on there.
>>>>
>>>> I've got to leave for a while, I'll copy the tree that works on the
>>>> Pro, update the copy and see if this test fails when I get back. If
>>>> they fail, I can diff the trees to see what changed and see if I can
>>>> make any sense out of this.
>>>>
>>>> I can always @Ignore this test to cut down on the noise, probably do
>>>> that tonight if I don't have any revelations.
>>>>
>>>> I see this stack trace which makes no sense to me whatsoever (see the
>>>> lines with lots of ***** in front). I looked at where the code
>>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>>
>>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>>
>>>> And it's telling me there's no such field? Whaaaaatttt????
>>>>
>>>> Which is freaking me out since I don't see how this would trigger the
>>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>>
>>>> Of course if this were something fundamental to Lucene, it seems like
>>>> this would be failing all over the place so I assume it's something to
>>>> do with CDCR... But what do I know?
>>>>
>>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>>> status=0 QTime=8
>>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>>> c:source_collection s:shard1 r:core_node3
>>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>>> totalTermCount
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>>> [junit4] 2> at
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>>> ************* [junit4] 2> at
>>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>>> [junit4] 2> at
>>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>>> [junit4] 2> at
>>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>>> [junit4] 2> at
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>>> [junit4] 2> at
>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>>> [junit4] 2> at
>>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>>> [junit4] 2> ... 23 more
>>>>
>>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>>> wrote:
>>>>> Looking......
>>>>>
>>>>>
>>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>>> <je...@thetaphi.de> wrote:
>>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>>
>>>>>> 4 tests failed.
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> Some resources were not closed, shutdown, or released.
>>>>>>
>>>>>> Stack Trace:
>>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>>> released.
>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>>> at
>>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>>> at
>>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>>> at
>>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>>> at
>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> 5 threads leaked from SUITE scope at
>>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>>> Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>> at
>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> Stack Trace:
>>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>>> 1) Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at java.lang.Thread.sleep(Native Method)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 3) Thread[id=4676,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>> at
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>> at
>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>> at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>>
>>>>>>
>>>>>> FAILED:
>>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>>
>>>>>> Error Message:
>>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>>> Thread[id=4675,
>>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> java.lang.Thread.sleep(Native Method) at
>>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>>> at
>>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>>> at
>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>>> at java.ut
>>>>
>>>> [The entire original message is not included.]
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
The problem went away for me. I took sledgehammer approach and deleted
the entire tree locally and checked out the tree fresh.
On Mon, May 25, 2015 at 4:37 AM, Noble Paul <no...@gmail.com> wrote:
> I'v edone a clean checkout I still see these errors
>
> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
>
> <title>Error 500 Server Error</title>
>
> </head>
>
> <body><h2>HTTP ERROR 500</h2>
>
> <p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
>
> <pre> Server Error</pre></p><h3>Caused
> by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
>
> at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>
> at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>
> at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>
> at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>
> at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>
> at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>
> On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
>> everything just fine, my laptop fails all the time.
>>
>> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
>> <ic...@gmail.com> wrote:
>>> Not sure if it is related or helpful, but while debugging tests for
>>> SOLR-7468 yesterday, I encountered this
>>> java.lang.NoSuchFieldError:
>>> totalTermCount
>>>
>>> few times, I had to forcefully clean at root of the project and it worked. I
>>> remember Anshum had to do that clean thing more than once to make it work
>>> and he remarked "don't ask why".
>>>
>>> Sent from my Windows Phone
>>> ________________________________
>>> From: Erick Erickson
>>> Sent: ‎5/‎23/‎2015 6:15 AM
>>> To: dev@lucene.apache.org
>>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>>> #12782 - Failure!
>>>
>>> OK, this is somewhat weird. I still have the original tree that I
>>> checked in from which was up to date before I committed the code and
>>> the tests run from there fine. But a current trunk fails every time.
>>> Now, the machine it works on is my Mac Pro, and the failures are on my
>>> MacBook so there may be something going on there.
>>>
>>> I've got to leave for a while, I'll copy the tree that works on the
>>> Pro, update the copy and see if this test fails when I get back. If
>>> they fail, I can diff the trees to see what changed and see if I can
>>> make any sense out of this.
>>>
>>> I can always @Ignore this test to cut down on the noise, probably do
>>> that tonight if I don't have any revelations.
>>>
>>> I see this stack trace which makes no sense to me whatsoever (see the
>>> lines with lots of ***** in front). I looked at where the code
>>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>>
>>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>>
>>> And it's telling me there's no such field? Whaaaaatttt????
>>>
>>> Which is freaking me out since I don't see how this would trigger the
>>> exception. Is this a red herring? And, of course, this doesn't fail in
>>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>>
>>> Of course if this were something fundamental to Lucene, it seems like
>>> this would be failing all over the place so I assume it's something to
>>> do with CDCR... But what do I know?
>>>
>>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>>> status=0 QTime=8
>>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>>> c:source_collection s:shard1 r:core_node3
>>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>>> totalTermCount
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>>> [junit4] 2> at
>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>>> [junit4] 2> at
>>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>>> [junit4] 2> at
>>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>>> [junit4] 2> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>>> [junit4] 2> at
>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>>> ************* [junit4] 2> at
>>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>>> [junit4] 2> at
>>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>>> [junit4] 2> at
>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>>> [junit4] 2> at
>>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>>> [junit4] 2> at
>>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>>> [junit4] 2> at
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>>> [junit4] 2> at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>>> [junit4] 2> at
>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>>> [junit4] 2> at
>>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>>> [junit4] 2> ... 23 more
>>>
>>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>>> wrote:
>>>> Looking......
>>>>
>>>>
>>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>>> <je...@thetaphi.de> wrote:
>>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>>
>>>>> 4 tests failed.
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> Some resources were not closed, shutdown, or released.
>>>>>
>>>>> Stack Trace:
>>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>>> released.
>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>> at org.junit.Assert.fail(Assert.java:93)
>>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>>> at
>>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>>> at
>>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>>> at
>>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>>> at
>>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>>
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> 5 threads leaked from SUITE scope at
>>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>>> Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> java.lang.Thread.sleep(Native Method) at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> Stack Trace:
>>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>>> 1) Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at java.lang.Thread.sleep(Native Method)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 3) Thread[id=4676,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> at
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> at
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>>
>>>>>
>>>>> FAILED:
>>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>>
>>>>> Error Message:
>>>>> There are still zombie threads that couldn't be terminated: 1)
>>>>> Thread[id=4675,
>>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> java.lang.Thread.sleep(Native Method) at
>>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>>> at
>>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>>> sun.misc.Unsafe.park(Native Method) at
>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>>> at
>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>>> at java.ut
>>>
>>> [The entire original message is not included.]
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Noble Paul <no...@gmail.com>.
I'v edone a clean checkout I still see these errors
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /solr/.system_shard1_replica2/update. Reason:
<pre> Server Error</pre></p><h3>Caused
by:</h3><pre>java.lang.NoSuchFieldError: totalTermCount
at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
On Sat, May 23, 2015 at 11:52 AM, Erick Erickson
<er...@gmail.com> wrote:
> thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
> everything just fine, my laptop fails all the time.
>
> On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
> <ic...@gmail.com> wrote:
>> Not sure if it is related or helpful, but while debugging tests for
>> SOLR-7468 yesterday, I encountered this
>> java.lang.NoSuchFieldError:
>> totalTermCount
>>
>> few times, I had to forcefully clean at root of the project and it worked. I
>> remember Anshum had to do that clean thing more than once to make it work
>> and he remarked "don't ask why".
>>
>> Sent from my Windows Phone
>> ________________________________
>> From: Erick Erickson
>> Sent: ‎5/‎23/‎2015 6:15 AM
>> To: dev@lucene.apache.org
>> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
>> #12782 - Failure!
>>
>> OK, this is somewhat weird. I still have the original tree that I
>> checked in from which was up to date before I committed the code and
>> the tests run from there fine. But a current trunk fails every time.
>> Now, the machine it works on is my Mac Pro, and the failures are on my
>> MacBook so there may be something going on there.
>>
>> I've got to leave for a while, I'll copy the tree that works on the
>> Pro, update the copy and see if this test fails when I get back. If
>> they fail, I can diff the trees to see what changed and see if I can
>> make any sense out of this.
>>
>> I can always @Ignore this test to cut down on the noise, probably do
>> that tonight if I don't have any revelations.
>>
>> I see this stack trace which makes no sense to me whatsoever (see the
>> lines with lots of ***** in front). I looked at where the code
>> originates (BufferedUpdatesStream[277]) and it looks like this:
>>
>> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>>
>> And it's telling me there's no such field? Whaaaaatttt????
>>
>> Which is freaking me out since I don't see how this would trigger the
>> exception. Is this a red herring? And, of course, this doesn't fail in
>> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>>
>> Of course if this were something fundamental to Lucene, it seems like
>> this would be failing all over the place so I assume it's something to
>> do with CDCR... But what do I know?
>>
>> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
>> status=0 QTime=8
>> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
>> c:source_collection s:shard1 r:core_node3
>> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
>> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
>> totalTermCount
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
>> [junit4] 2> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>> [junit4] 2> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
>> [junit4] 2> at
>> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>> [junit4] 2> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>> [junit4] 2> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>> [junit4] 2> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>> [junit4] 2> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
>> [junit4] 2> at
>> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>> [junit4] 2> at
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>> [junit4] 2> at
>> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>> [junit4] 2> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>> [junit4] 2> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
>> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
>> ************* [junit4] 2> at
>> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
>> [junit4] 2> at
>> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
>> [junit4] 2> at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
>> [junit4] 2> at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
>> [junit4] 2> at
>> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
>> [junit4] 2> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
>> [junit4] 2> at
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
>> [junit4] 2> at
>> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
>> [junit4] 2> at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
>> [junit4] 2> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
>> [junit4] 2> at
>> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
>> [junit4] 2> at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
>> [junit4] 2> ... 23 more
>>
>> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
>> wrote:
>>> Looking......
>>>
>>>
>>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>>> <je...@thetaphi.de> wrote:
>>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>>
>>>> 4 tests failed.
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> Some resources were not closed, shutdown, or released.
>>>>
>>>> Stack Trace:
>>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>>> released.
>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>> at org.junit.Assert.fail(Assert.java:93)
>>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>>> at
>>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>> at
>>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>>> at
>>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>>> at
>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>>> at
>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>> at
>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>> at
>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>> at
>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>> at
>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> 5 threads leaked from SUITE scope at
>>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>>> Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> java.lang.Thread.sleep(Native Method) at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Stack Trace:
>>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>>> 1) Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at java.lang.Thread.sleep(Native Method)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 3) Thread[id=4676,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>> at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>> at
>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at
>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>>
>>>>
>>>> FAILED:
>>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>>
>>>> Error Message:
>>>> There are still zombie threads that couldn't be terminated: 1)
>>>> Thread[id=4675,
>>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> java.lang.Thread.sleep(Native Method) at
>>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>>> at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>>> sun.misc.Unsafe.park(Native Method) at
>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>>> at
>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>>> at java.ut
>>
>> [The entire original message is not included.]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
thanks! I'll give it a whirl. It's particularly weird b/c my pro runs
everything just fine, my laptop fails all the time.
On Fri, May 22, 2015 at 10:10 PM, Ishan Chattopadhyaya
<ic...@gmail.com> wrote:
> Not sure if it is related or helpful, but while debugging tests for
> SOLR-7468 yesterday, I encountered this
> java.lang.NoSuchFieldError:
> totalTermCount
>
> few times, I had to forcefully clean at root of the project and it worked. I
> remember Anshum had to do that clean thing more than once to make it work
> and he remarked "don't ask why".
>
> Sent from my Windows Phone
> ________________________________
> From: Erick Erickson
> Sent: ‎5/‎23/‎2015 6:15 AM
> To: dev@lucene.apache.org
> Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
> #12782 - Failure!
>
> OK, this is somewhat weird. I still have the original tree that I
> checked in from which was up to date before I committed the code and
> the tests run from there fine. But a current trunk fails every time.
> Now, the machine it works on is my Mac Pro, and the failures are on my
> MacBook so there may be something going on there.
>
> I've got to leave for a while, I'll copy the tree that works on the
> Pro, update the copy and see if this test fails when I get back. If
> they fail, I can diff the trees to see what changed and see if I can
> make any sense out of this.
>
> I can always @Ignore this test to cut down on the noise, probably do
> that tonight if I don't have any revelations.
>
> I see this stack trace which makes no sense to me whatsoever (see the
> lines with lots of ***** in front). I looked at where the code
> originates (BufferedUpdatesStream[277]) and it looks like this:
>
> if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
>
> And it's telling me there's no such field? Whaaaaatttt????
>
> Which is freaking me out since I don't see how this would trigger the
> exception. Is this a red herring? And, of course, this doesn't fail in
> IntelliJ but it does fail every time from the shell. Siiiigggghhh.
>
> Of course if this were something fundamental to Lucene, it seems like
> this would be failing all over the place so I assume it's something to
> do with CDCR... But what do I know?
>
> 1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
> status=0 QTime=8
> ************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
> c:source_collection s:shard1 r:core_node3
> x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
> null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
> totalTermCount
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
> [junit4] 2> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> [junit4] 2> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
> [junit4] 2> at
> org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> [junit4] 2> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> [junit4] 2> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> [junit4] 2> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> [junit4] 2> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> [junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
> [junit4] 2> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
> [junit4] 2> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> [junit4] 2> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> [junit4] 2> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> [junit4] 2> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> [junit4] 2> at java.lang.Thread.run(Thread.java:745)
> [junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
> ************* [junit4] 2> at
> org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
> [junit4] 2> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
> [junit4] 2> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
> [junit4] 2> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
> [junit4] 2> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
> [junit4] 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
> [junit4] 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
> [junit4] 2> at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
> [junit4] 2> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> [junit4] 2> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> [junit4] 2> at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
> [junit4] 2> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
> [junit4] 2> ... 23 more
>
> On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com>
> wrote:
>> Looking......
>>
>>
>> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
>> <je...@thetaphi.de> wrote:
>>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>>
>>> 4 tests failed.
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> Some resources were not closed, shutdown, or released.
>>>
>>> Stack Trace:
>>> java.lang.AssertionError: Some resources were not closed, shutdown, or
>>> released.
>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>> at org.junit.Assert.fail(Assert.java:93)
>>> at org.junit.Assert.assertTrue(Assert.java:43)
>>> at
>>> org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>>> at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>>> at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>> at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>> at
>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>> at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>> at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> 5 threads leaked from SUITE scope at
>>> org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1)
>>> Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> java.lang.Thread.sleep(Native Method) at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2)
>>> Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677,
>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Stack Trace:
>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from
>>> SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>>> 1) Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>> at java.lang.Thread.sleep(Native Method)
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 3) Thread[id=4676,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread,
>>> state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>> at
>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>> at
>>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at
>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>>
>>>
>>> FAILED:
>>> junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>>
>>> Error Message:
>>> There are still zombie threads that couldn't be terminated: 1)
>>> Thread[id=4675,
>>> name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300),
>>> state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at
>>> java.lang.Thread.sleep(Native Method) at
>>> org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>>> at
>>> org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720,
>>> name=zkCallback-697-thread-3, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677,
>>> name=zkCallback-697-thread-1, state=TIMED_WAITING,
>>> group=TGRP-CdcrReplicationDistributedZkTest] at
>>> sun.misc.Unsafe.park(Native Method) at
>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>>> at
>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>>> at java.ut
>
> [The entire original message is not included.]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build
#12782 - Failure!
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Not sure if it is related or helpful, but while debugging tests for SOLR-7468 yesterday, I encountered this
java.lang.NoSuchFieldError:
totalTermCount
few times, I had to forcefully clean at root of the project and it worked. I remember Anshum had to do that clean thing more than once to make it work and he remarked "don't ask why".
Sent from my Windows Phone
-----Original Message-----
From: "Erick Erickson" <er...@gmail.com>
Sent: ‎5/‎23/‎2015 6:15 AM
To: "dev@lucene.apache.org" <de...@lucene.apache.org>
Subject: Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #12782 - Failure!
OK, this is somewhat weird. I still have the original tree that I
checked in from which was up to date before I committed the code and
the tests run from there fine. But a current trunk fails every time.
Now, the machine it works on is my Mac Pro, and the failures are on my
MacBook so there may be something going on there.
I've got to leave for a while, I'll copy the tree that works on the
Pro, update the copy and see if this test fails when I get back. If
they fail, I can diff the trees to see what changed and see if I can
make any sense out of this.
I can always @Ignore this test to cut down on the noise, probably do
that tonight if I don't have any revelations.
I see this stack trace which makes no sense to me whatsoever (see the
lines with lots of ***** in front). I looked at where the code
originates (BufferedUpdatesStream[277]) and it looks like this:
if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
And it's telling me there's no such field? Whaaaaatttt????
Which is freaking me out since I don't see how this would trigger the
exception. Is this a red herring? And, of course, this doesn't fail in
IntelliJ but it does fail every time from the shell. Siiiigggghhh.
Of course if this were something fundamental to Lucene, it seems like
this would be failing all over the place so I assume it's something to
do with CDCR... But what do I know?
1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
status=0 QTime=8
************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
c:source_collection s:shard1 r:core_node3
x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
totalTermCount
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
************* [junit4] 2> at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
[junit4] 2> at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
[junit4] 2> at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
[junit4] 2> at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
[junit4] 2> at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
[junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
[junit4] 2> ... 23 more
On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com> wrote:
> Looking......
>
>
> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 4 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> Some resources were not closed, shutdown, or released.
>>
>> Stack Trace:
>> java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at org.junit.Assert.assertTrue(Assert.java:43)
>> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.ut
[The entire original message is not included.]
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #
12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
OK, this is somewhat weird. I still have the original tree that I
checked in from which was up to date before I committed the code and
the tests run from there fine. But a current trunk fails every time.
Now, the machine it works on is my Mac Pro, and the failures are on my
MacBook so there may be something going on there.
I've got to leave for a while, I'll copy the tree that works on the
Pro, update the copy and see if this test fails when I get back. If
they fail, I can diff the trees to see what changed and see if I can
make any sense out of this.
I can always @Ignore this test to cut down on the noise, probably do
that tonight if I don't have any revelations.
I see this stack trace which makes no sense to me whatsoever (see the
lines with lots of ***** in front). I looked at where the code
originates (BufferedUpdatesStream[277]) and it looks like this:
if (coalescedUpdates != null && coalescedUpdates.totalTermCount != 0) {
And it's telling me there's no such field? Whaaaaatttt????
Which is freaking me out since I don't see how this would trigger the
exception. Is this a red herring? And, of course, this doesn't fail in
IntelliJ but it does fail every time from the shell. Siiiigggghhh.
Of course if this were something fundamental to Lucene, it seems like
this would be failing all over the place so I assume it's something to
do with CDCR... But what do I know?
1:56434/source_collection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}
status=0 QTime=8
************* [junit4] 2> 143699 T370 n:127.0.0.1:56443_
c:source_collection s:shard1 r:core_node3
x:source_collection_shard1_replica1 C122 oasc.SolrException.log ERROR
null:java.lang.RuntimeException: java.lang.NoSuchFieldError:
totalTermCount
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:579)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:451)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] 2> at
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:497)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.NoSuchFieldError: totalTermCount
************* [junit4] 2> at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:277)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3032)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3018)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2707)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2852)
[junit4] 2> at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2819)
[junit4] 2> at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:586)
[junit4] 2> at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
[junit4] 2> at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1642)
[junit4] 2> at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1619)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
[junit4] 2> at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
[junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2051)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:640)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:436)
[junit4] 2> ... 23 more
On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com> wrote:
> Looking......
>
>
> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 4 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> Some resources were not closed, shutdown, or released.
>>
>> Stack Trace:
>> java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at org.junit.Assert.assertTrue(Assert.java:43)
>> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
>>
>> Error Message:
>> expected:<st[art]ed> but was:<st[opp]ed>
>>
>> Stack Trace:
>> org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A:D0B02EE63F74C6D3]:0)
>> at org.junit.Assert.assertEquals(Assert.java:125)
>> at org.junit.Assert.assertEquals(Assert.java:147)
>> at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
>> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
>> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
>> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
>> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 10551 lines...]
>> [junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>> [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/init-core-data-001
>> [junit4] 2> 757362 T4034 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
>> [junit4] 2> 757362 T4034 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
>> [junit4] 2> 757364 T4034 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>> [junit4] 2> 757364 T4035 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 757364 T4035 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 757464 T4034 oasc.ZkTestServer.run start zk server on port:54300
>> [junit4] 2> 757465 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757465 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757467 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a254558 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757468 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757468 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757468 T4034 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 757469 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757470 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757471 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34362c7f name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757471 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757471 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757471 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1
>> [junit4] 2> 757472 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>> [junit4] 2> 757474 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
>> [junit4] 2> 757474 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>> [junit4] 2> 757475 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
>> [junit4] 2> 757475 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>> [junit4] 2> 757476 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 757476 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>> [junit4] 2> 757477 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>> [junit4] 2> 757478 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>> [junit4] 2> 757479 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>> [junit4] 2> 757479 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>> [junit4] 2> 757480 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 757480 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 757481 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>> [junit4] 2> 757482 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>> [junit4] 2> 757482 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>> [junit4] 2> 757483 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>> [junit4] 2> 757484 T4034 oas.SolrTestCaseJ4.setUp ###Starting doTest
>> [junit4] 2> 757533 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1
>> [junit4] 2> 757534 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 757535 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@76300627{/,null,AVAILABLE}
>> [junit4] 2> 757537 T4034 oejs.AbstractConnector.doStart Started ServerConnector@93aadc7{HTTP/1.1}{127.0.0.1:42803}
>> [junit4] 2> 757537 T4034 oejs.Server.doStart Started @758455ms
>> [junit4] 2> 757537 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=42803, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores}
>> [junit4] 2> 757538 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 757538 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/'
>> [junit4] 2> 757547 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/solr.xml
>> [junit4] 2> 757550 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
>> [junit4] 2> 757550 T4034 oasc.CoreContainer.<init> New CoreContainer 1191155484
>> [junit4] 2> 757550 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/]
>> [junit4] 2> 757551 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib
>> [junit4] 2> 757551 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib).
>> [junit4] 2> 757555 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 757556 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 757556 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 757556 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 757556 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 757557 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 757557 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 757557 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757557 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757559 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@599a8e77 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757559 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757559 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757560 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757561 T4062 n:127.0.0.1:42803_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7452961a name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757561 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757562 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue
>> [junit4] 2> 757563 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>> [junit4] 2> 757564 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
>> [junit4] 2> 757565 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
>> [junit4] 2> 757566 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
>> [junit4] 2> 757567 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes
>> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /aliases.json
>> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
>> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42803_
>> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42803_
>> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect
>> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>> [junit4] 2> 757571 T4034 n:127.0.0.1:42803_ oasc.Overseer.close Overseer (id=null) closing
>> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42803_
>> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>> [junit4] 2> 757573 T4034 n:127.0.0.1:42803_ oasc.Overseer.start Overseer (id=93869083152809987-127.0.0.1:42803_-n_0000000000) starting
>> [junit4] 2> 757574 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
>> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 757576 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>> [junit4] 2> 757576 T4064 n:127.0.0.1:42803_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
>> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
>> [junit4] 2> 757579 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Found 1 core definitions
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 757581 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForCoreNodeName look for our core node name
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 0
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
>> [junit4] 2> 757583 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Looking for collection configName
>> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
>> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/'
>> [junit4] 2> 758598 T4066 n:127.0.0.1:42803_ oasc.Config.<init> loaded config solrconfig.xml with version 0
>> [junit4] 2> 758600 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
>> [junit4] 2> 758603 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
>> [junit4] 2> 758608 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>> [junit4] 2> 758609 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
>> [junit4] 2> 758611 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema [collection1] Schema name=test
>> [junit4] 2> 758665 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema default search field in schema is text
>> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 758668 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/], dataDir=[null]
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index/
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
>> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 758677 T4066 n:127.0.0.1:42803_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
>> [junit4] 2> 758686 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 758687 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
>> [junit4] 2> 758696 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
>> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
>> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@d198739[collection1] main
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905331851100160
>> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905331851100160
>> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
>> [junit4] 2> 758712 T4066 n:127.0.0.1:42803_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
>> [junit4] 2> 758714 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
>> [junit4] 2> 758716 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
>> [junit4] 2> 758726 T4066 n:127.0.0.1:42803_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
>> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
>> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
>> [junit4] 2> 758727 T4067 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d198739[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
>> [junit4] 2> 758727 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42803 collection:tmp_collection shard:shard2
>> [junit4] 2> 758727 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 758728 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>> [junit4] 2> 758728 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>> [junit4] 2> 758730 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>> [junit4] 2> 758731 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection"} current state version: 1
>> [junit4] 2> ASYNC NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@16e04228 url=http://127.0.0.1:42803/collection1 node=127.0.0.1:42803_ C983_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42803, node_name=127.0.0.1:42803_, state=down}
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42803/collection1/
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncToMe http://127.0.0.1:42803/collection1/ has no replicas
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42803/collection1/ shard2
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
>> [junit4] 2> 758732 T4062 n:127.0.0.1:42803_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
>> [junit4] 2> 758733 T4072 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758734 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "state":"active"} current state version: 1
>> [junit4] 2> 758778 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1
>> [junit4] 2> 758779 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 758780 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@12138b09{/,null,AVAILABLE}
>> [junit4] 2> 758781 T4034 oejs.AbstractConnector.doStart Started ServerConnector@537374e0{HTTP/1.1}{127.0.0.1:47447}
>> [junit4] 2> 758781 T4034 oejs.Server.doStart Started @759698ms
>> [junit4] 2> 758781 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=47447, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores}
>> [junit4] 2> 758781 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 758781 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/'
>> [junit4] 2> 758790 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/solr.xml
>> [junit4] 2> 758793 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.<init> New CoreContainer 957201046
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/]
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib
>> [junit4] 2> 758794 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib).
>> [junit4] 2> 758798 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 758799 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 758799 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 758799 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 758799 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 758800 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 758800 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 758800 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 758800 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 758802 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7f7cd8 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 758802 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 758802 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 758803 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 758804 T4089 n:127.0.0.1:47447_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc03e94 name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 758805 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 758807 T4034 n:127.0.0.1:47447_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 758835 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758835 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:42803/collection1/ and leader is http://127.0.0.1:42803/collection1/
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42803
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 758885 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758885 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "core_node_name":"core_node1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 2
>> [junit4] 2> 758886 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "core_node_name":"core_node1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 758987 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758987 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 759808 T4034 n:127.0.0.1:47447_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47447_
>> [junit4] 2> 759809 T4034 n:127.0.0.1:47447_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47447_
>> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.Overseer.close Overseer (id=null) closing
>> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
>> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
>> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Found 1 core definitions
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForCoreNodeName look for our core node name
>> [junit4] 2> 759813 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "node_name":"127.0.0.1:47447_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 3
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "node_name":"127.0.0.1:47447_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
>> [junit4] 2> 759915 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 759916 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 760813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/'
>> [junit4] 2> 760826 T4090 n:127.0.0.1:47447_ oasc.Config.<init> loaded config solrconfig.xml with version 0
>> [junit4] 2> 760829 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
>> [junit4] 2> 760831 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
>> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
>> [junit4] 2> 760840 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema [collection1] Schema name=test
>> [junit4] 2> 760892 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema default search field in schema is text
>> [junit4] 2> 760893 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 760894 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 760895 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
>> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/], dataDir=[null]
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index/
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 760904 T4090 n:127.0.0.1:47447_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
>> [junit4] 2> 760905 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760908 T4090 n:127.0.0.1:47447_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
>> [junit4] 2> 760909 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5da2fafa[collection1] main
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905334165307392
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905334165307392
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
>> [junit4] 2> 760917 T4090 n:127.0.0.1:47447_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
>> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
>> [junit4] 2> 760918 T4091 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5da2fafa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
>> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
>> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47447 collection:tmp_collection shard:shard1
>> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
>> [junit4] 2> 760920 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>> [junit4] 2> 760921 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>> [junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@4e21ece7 url=http://127.0.0.1:47447/collection1 node=127.0.0.1:47447_ C984_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:47447, node_name=127.0.0.1:47447_, state=down}
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47447/collection1/
>> [junit4] 2> 760921 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"tmp_collection"} current state version: 4
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncToMe http://127.0.0.1:47447/collection1/ has no replicas
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47447/collection1/ shard1
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
>> [junit4] 2> 760922 T4089 n:127.0.0.1:47447_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
>> [junit4] 2> 760923 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 760924 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "state":"active"} current state version: 4
>> [junit4] 2> 760968 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores/collection1
>> [junit4] 2> 760970 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 760971 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4fc5b5e4{/,null,AVAILABLE}
>> [junit4] 2> 760971 T4034 oejs.AbstractConnector.doStart Started ServerConnector@41389b97{HTTP/1.1}{127.0.0.1:36397}
>> [junit4] 2> 760971 T4034 oejs.Server.doStart Started @761888ms
>> [junit4] 2> 760971 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=36397, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores}
>> [junit4] 2> 760972 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 760972 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/'
>> [junit4] 2> 760980 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/solr.xml
>> [junit4] 2> 760983 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.<init> New CoreContainer 327774697
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/]
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib
>> [junit4] 2> 760984 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib).
>> [junit4] 2> 760988 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 760988 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 760989 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 760989 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 760989 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 760989 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 760990 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 760990 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 760990 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 760992 T4109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21f437e1 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 760992 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 760992 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 760994 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 760995 T4112 n:127.0.0.1:36397_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53fdaf5b name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 760995 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 760997 T4034 n:127.0.0.1:36397_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 761025 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761025 T4112 n:127.0.0.1:36397_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761025 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:47447/collection1/ and leader is http://127.0.0.1:47447/collection1/
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47447
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
>> [junit4] 2> 761074 T4095 n:127.0.0.1:474
>>
>> [...truncated too long message...]
>>
>> I
>> [junit4] ERROR 0.00s J0 | CdcrReplicationDistributedZkTest (suite) <<<
>> [junit4] > Throwable #1: java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> [junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at java.lang.Thread.sleep(Native Method)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> [junit4] > 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at java.lang.Thread.sleep(Native Method)
>> [junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> [junit4] Completed [381/502] on J0 in 167.35s, 1 test, 2 failures, 2 errors <<< FAILURES!
>>
>> [...truncated 376 lines...]
>> BUILD FAILED
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1985 tests, 3 suite-level errors, 1 failure, 316 ignored (25 assumptions)
>>
>> Total time: 32 minutes 3 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
On Fri, May 22, 2015 at 2:09 PM, Erick Erickson <er...@gmail.com> wrote:
> Looking......
>
>
> On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
>> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>>
>> 4 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> Some resources were not closed, shutdown, or released.
>>
>> Stack Trace:
>> java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at org.junit.Assert.assertTrue(Assert.java:43)
>> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>>
>>
>> FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
>>
>> Error Message:
>> expected:<st[art]ed> but was:<st[opp]ed>
>>
>> Stack Trace:
>> org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
>> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A:D0B02EE63F74C6D3]:0)
>> at org.junit.Assert.assertEquals(Assert.java:125)
>> at org.junit.Assert.assertEquals(Assert.java:147)
>> at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
>> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
>> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
>> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
>> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>> at java.lang.Thread.run(Thread.java:745)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 10551 lines...]
>> [junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>> [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/init-core-data-001
>> [junit4] 2> 757362 T4034 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
>> [junit4] 2> 757362 T4034 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
>> [junit4] 2> 757364 T4034 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>> [junit4] 2> 757364 T4035 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 757364 T4035 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>> [junit4] 2> 757464 T4034 oasc.ZkTestServer.run start zk server on port:54300
>> [junit4] 2> 757465 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757465 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757467 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a254558 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757468 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757468 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757468 T4034 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 757469 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757470 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757471 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34362c7f name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757471 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757471 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757471 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1
>> [junit4] 2> 757472 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>> [junit4] 2> 757474 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
>> [junit4] 2> 757474 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>> [junit4] 2> 757475 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
>> [junit4] 2> 757475 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>> [junit4] 2> 757476 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 757476 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>> [junit4] 2> 757477 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>> [junit4] 2> 757478 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>> [junit4] 2> 757479 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>> [junit4] 2> 757479 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>> [junit4] 2> 757480 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 757480 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 757481 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>> [junit4] 2> 757482 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>> [junit4] 2> 757482 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>> [junit4] 2> 757483 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>> [junit4] 2> 757484 T4034 oas.SolrTestCaseJ4.setUp ###Starting doTest
>> [junit4] 2> 757533 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1
>> [junit4] 2> 757534 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 757535 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@76300627{/,null,AVAILABLE}
>> [junit4] 2> 757537 T4034 oejs.AbstractConnector.doStart Started ServerConnector@93aadc7{HTTP/1.1}{127.0.0.1:42803}
>> [junit4] 2> 757537 T4034 oejs.Server.doStart Started @758455ms
>> [junit4] 2> 757537 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=42803, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores}
>> [junit4] 2> 757538 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 757538 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/'
>> [junit4] 2> 757547 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/solr.xml
>> [junit4] 2> 757550 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
>> [junit4] 2> 757550 T4034 oasc.CoreContainer.<init> New CoreContainer 1191155484
>> [junit4] 2> 757550 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/]
>> [junit4] 2> 757551 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib
>> [junit4] 2> 757551 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib).
>> [junit4] 2> 757555 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 757556 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 757556 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 757556 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 757556 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 757557 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 757557 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 757557 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 757557 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757559 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@599a8e77 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757559 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757559 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 757560 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 757561 T4062 n:127.0.0.1:42803_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7452961a name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 757561 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 757562 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue
>> [junit4] 2> 757563 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>> [junit4] 2> 757564 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
>> [junit4] 2> 757565 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
>> [junit4] 2> 757566 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
>> [junit4] 2> 757567 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes
>> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /aliases.json
>> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
>> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42803_
>> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42803_
>> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect
>> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>> [junit4] 2> 757571 T4034 n:127.0.0.1:42803_ oasc.Overseer.close Overseer (id=null) closing
>> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42803_
>> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>> [junit4] 2> 757573 T4034 n:127.0.0.1:42803_ oasc.Overseer.start Overseer (id=93869083152809987-127.0.0.1:42803_-n_0000000000) starting
>> [junit4] 2> 757574 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
>> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 757576 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>> [junit4] 2> 757576 T4064 n:127.0.0.1:42803_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
>> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
>> [junit4] 2> 757579 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/
>> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Found 1 core definitions
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 757581 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForCoreNodeName look for our core node name
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 0
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
>> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
>> [junit4] 2> 757583 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
>> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Looking for collection configName
>> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
>> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/'
>> [junit4] 2> 758598 T4066 n:127.0.0.1:42803_ oasc.Config.<init> loaded config solrconfig.xml with version 0
>> [junit4] 2> 758600 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
>> [junit4] 2> 758603 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
>> [junit4] 2> 758608 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>> [junit4] 2> 758609 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
>> [junit4] 2> 758611 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema [collection1] Schema name=test
>> [junit4] 2> 758665 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema default search field in schema is text
>> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 758668 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/], dataDir=[null]
>> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index/
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index
>> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
>> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 758677 T4066 n:127.0.0.1:42803_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
>> [junit4] 2> 758686 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 758687 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
>> [junit4] 2> 758696 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
>> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
>> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
>> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@d198739[collection1] main
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
>> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905331851100160
>> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905331851100160
>> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
>> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
>> [junit4] 2> 758712 T4066 n:127.0.0.1:42803_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
>> [junit4] 2> 758714 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
>> [junit4] 2> 758716 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
>> [junit4] 2> 758726 T4066 n:127.0.0.1:42803_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
>> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
>> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
>> [junit4] 2> 758727 T4067 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d198739[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
>> [junit4] 2> 758727 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42803 collection:tmp_collection shard:shard2
>> [junit4] 2> 758727 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 758728 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>> [junit4] 2> 758728 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>> [junit4] 2> 758730 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>> [junit4] 2> 758731 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection"} current state version: 1
>> [junit4] 2> ASYNC NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@16e04228 url=http://127.0.0.1:42803/collection1 node=127.0.0.1:42803_ C983_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42803, node_name=127.0.0.1:42803_, state=down}
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42803/collection1/
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncToMe http://127.0.0.1:42803/collection1/ has no replicas
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42803/collection1/ shard2
>> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
>> [junit4] 2> 758732 T4062 n:127.0.0.1:42803_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
>> [junit4] 2> 758733 T4072 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758734 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "state":"active"} current state version: 1
>> [junit4] 2> 758778 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1
>> [junit4] 2> 758779 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 758780 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@12138b09{/,null,AVAILABLE}
>> [junit4] 2> 758781 T4034 oejs.AbstractConnector.doStart Started ServerConnector@537374e0{HTTP/1.1}{127.0.0.1:47447}
>> [junit4] 2> 758781 T4034 oejs.Server.doStart Started @759698ms
>> [junit4] 2> 758781 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=47447, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores}
>> [junit4] 2> 758781 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 758781 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/'
>> [junit4] 2> 758790 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/solr.xml
>> [junit4] 2> 758793 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.<init> New CoreContainer 957201046
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/]
>> [junit4] 2> 758794 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib
>> [junit4] 2> 758794 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib).
>> [junit4] 2> 758798 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 758799 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 758799 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 758799 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 758799 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 758800 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 758800 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 758800 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 758800 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 758802 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7f7cd8 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 758802 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 758802 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 758803 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 758804 T4089 n:127.0.0.1:47447_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc03e94 name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 758805 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 758807 T4034 n:127.0.0.1:47447_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 758835 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758835 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:42803/collection1/ and leader is http://127.0.0.1:42803/collection1/
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42803
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
>> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 758885 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 758885 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "core_node_name":"core_node1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 2
>> [junit4] 2> 758886 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "core_node_name":"core_node1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:42803",
>> [junit4] 2> "node_name":"127.0.0.1:42803_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "shard":"shard2",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 758987 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 758987 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 759808 T4034 n:127.0.0.1:47447_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47447_
>> [junit4] 2> 759809 T4034 n:127.0.0.1:47447_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47447_
>> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.Overseer.close Overseer (id=null) closing
>> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
>> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
>> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/
>> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Found 1 core definitions
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForCoreNodeName look for our core node name
>> [junit4] 2> 759813 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "node_name":"127.0.0.1:47447_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"} current state version: 3
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "node_name":"127.0.0.1:47447_",
>> [junit4] 2> "numShards":"2",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "operation":"state"}
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
>> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
>> [junit4] 2> 759915 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 759916 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 760813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/'
>> [junit4] 2> 760826 T4090 n:127.0.0.1:47447_ oasc.Config.<init> loaded config solrconfig.xml with version 0
>> [junit4] 2> 760829 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
>> [junit4] 2> 760831 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
>> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
>> [junit4] 2> 760840 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema [collection1] Schema name=test
>> [junit4] 2> 760892 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema default search field in schema is text
>> [junit4] 2> 760893 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 760894 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 760895 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
>> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/], dataDir=[null]
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index/
>> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
>> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 760904 T4090 n:127.0.0.1:47447_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
>> [junit4] 2> 760905 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 760908 T4090 n:127.0.0.1:47447_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
>> [junit4] 2> 760909 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
>> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
>> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5da2fafa[collection1] main
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
>> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905334165307392
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905334165307392
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
>> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
>> [junit4] 2> 760917 T4090 n:127.0.0.1:47447_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
>> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
>> [junit4] 2> 760918 T4091 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5da2fafa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
>> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
>> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47447 collection:tmp_collection shard:shard1
>> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
>> [junit4] 2> 760920 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>> [junit4] 2> 760921 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>> [junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@4e21ece7 url=http://127.0.0.1:47447/collection1 node=127.0.0.1:47447_ C984_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:47447, node_name=127.0.0.1:47447_, state=down}
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47447/collection1/
>> [junit4] 2> 760921 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"tmp_collection"} current state version: 4
>> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncToMe http://127.0.0.1:47447/collection1/ has no replicas
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47447/collection1/ shard1
>> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
>> [junit4] 2> 760922 T4089 n:127.0.0.1:47447_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
>> [junit4] 2> 760923 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
>> [junit4] 2> 760924 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
>> [junit4] 2> "operation":"leader",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"tmp_collection",
>> [junit4] 2> "base_url":"http://127.0.0.1:47447",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "state":"active"} current state version: 4
>> [junit4] 2> 760968 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores/collection1
>> [junit4] 2> 760970 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
>> [junit4] 2> 760971 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4fc5b5e4{/,null,AVAILABLE}
>> [junit4] 2> 760971 T4034 oejs.AbstractConnector.doStart Started ServerConnector@41389b97{HTTP/1.1}{127.0.0.1:36397}
>> [junit4] 2> 760971 T4034 oejs.Server.doStart Started @761888ms
>> [junit4] 2> 760971 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=36397, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores}
>> [junit4] 2> 760972 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
>> [junit4] 2> 760972 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/'
>> [junit4] 2> 760980 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/solr.xml
>> [junit4] 2> 760983 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.<init> New CoreContainer 327774697
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/]
>> [junit4] 2> 760984 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib
>> [junit4] 2> 760984 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib).
>> [junit4] 2> 760988 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
>> [junit4] 2> 760988 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
>> [junit4] 2> 760989 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 760989 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 760989 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
>> [junit4] 2> 760989 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
>> [junit4] 2> 760990 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
>> [junit4] 2> 760990 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
>> [junit4] 2> 760990 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 760992 T4109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21f437e1 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 760992 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 760992 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
>> [junit4] 2> 760994 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>> [junit4] 2> 760995 T4112 n:127.0.0.1:36397_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53fdaf5b name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 760995 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>> [junit4] 2> 760997 T4034 n:127.0.0.1:36397_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
>> [junit4] 2> 761025 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761025 T4112 n:127.0.0.1:36397_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761025 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:47447/collection1/ and leader is http://127.0.0.1:47447/collection1/
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47447
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
>> [junit4] 2> 761074 T4095 n:127.0.0.1:474
>>
>> [...truncated too long message...]
>>
>> I
>> [junit4] ERROR 0.00s J0 | CdcrReplicationDistributedZkTest (suite) <<<
>> [junit4] > Throwable #1: java.lang.AssertionError: Some resources were not closed, shutdown, or released.
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> [junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
>> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at java.lang.Thread.sleep(Native Method)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
>> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
>> [junit4] > 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at java.lang.Thread.sleep(Native Method)
>> [junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
>> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [junit4] > at java.lang.Thread.run(Thread.java:745)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>> [junit4] Completed [381/502] on J0 in 167.35s, 1 test, 2 failures, 2 errors <<< FAILURES!
>>
>> [...truncated 376 lines...]
>> BUILD FAILED
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
>> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
>> /home/jenkins/workspace/
> ...
>
> [Message clipped]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #
12782 - Failure!
Posted by Erick Erickson <er...@gmail.com>.
Looking......
On Fri, May 22, 2015 at 2:06 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12782/
> Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
>
> 4 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>
> Error Message:
> Some resources were not closed, shutdown, or released.
>
> Stack Trace:
> java.lang.AssertionError: Some resources were not closed, shutdown, or released.
> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>
> Error Message:
> 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated: 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
> 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
>
>
> FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
>
> Error Message:
> expected:<st[art]ed> but was:<st[opp]ed>
>
> Stack Trace:
> org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
> at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A:D0B02EE63F74C6D3]:0)
> at org.junit.Assert.assertEquals(Assert.java:125)
> at org.junit.Assert.assertEquals(Assert.java:147)
> at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
> at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
> at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 10551 lines...]
> [junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
> [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/init-core-data-001
> [junit4] 2> 757362 T4034 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
> [junit4] 2> 757362 T4034 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
> [junit4] 2> 757364 T4034 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 2> 757364 T4035 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 757364 T4035 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4] 2> 757464 T4034 oasc.ZkTestServer.run start zk server on port:54300
> [junit4] 2> 757465 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
> [junit4] 2> 757465 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 757467 T4042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a254558 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 757468 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 757468 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
> [junit4] 2> 757468 T4034 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 757469 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
> [junit4] 2> 757470 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 757471 T4045 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34362c7f name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 757471 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 757471 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
> [junit4] 2> 757471 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4] 2> 757472 T4034 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4] 2> 757473 T4034 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4] 2> 757474 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
> [junit4] 2> 757474 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4] 2> 757475 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
> [junit4] 2> 757475 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4] 2> 757476 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 757476 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4] 2> 757477 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4] 2> 757477 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4] 2> 757478 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4] 2> 757478 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4] 2> 757479 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
> [junit4] 2> 757479 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
> [junit4] 2> 757480 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4] 2> 757480 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 757481 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 757481 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4] 2> 757482 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4] 2> 757482 T4034 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4] 2> 757483 T4034 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4] 2> 757484 T4034 oas.SolrTestCaseJ4.setUp ###Starting doTest
> [junit4] 2> 757533 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1
> [junit4] 2> 757534 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
> [junit4] 2> 757535 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@76300627{/,null,AVAILABLE}
> [junit4] 2> 757537 T4034 oejs.AbstractConnector.doStart Started ServerConnector@93aadc7{HTTP/1.1}{127.0.0.1:42803}
> [junit4] 2> 757537 T4034 oejs.Server.doStart Started @758455ms
> [junit4] 2> 757537 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=42803, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores}
> [junit4] 2> 757538 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
> [junit4] 2> 757538 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/'
> [junit4] 2> 757547 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/solr.xml
> [junit4] 2> 757550 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
> [junit4] 2> 757550 T4034 oasc.CoreContainer.<init> New CoreContainer 1191155484
> [junit4] 2> 757550 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/]
> [junit4] 2> 757551 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib
> [junit4] 2> 757551 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/lib).
> [junit4] 2> 757555 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
> [junit4] 2> 757556 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
> [junit4] 2> 757556 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 757556 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 757556 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
> [junit4] 2> 757557 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
> [junit4] 2> 757557 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 757557 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
> [junit4] 2> 757557 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 757559 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@599a8e77 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 757559 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 757559 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
> [junit4] 2> 757560 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 757561 T4062 n:127.0.0.1:42803_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7452961a name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 757561 T4034 n:127.0.0.1:42803_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 757562 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4] 2> 757563 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4] 2> 757564 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
> [junit4] 2> 757565 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
> [junit4] 2> 757566 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
> [junit4] 2> 757567 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4] 2> 757568 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42803_
> [junit4] 2> 757569 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42803_
> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect
> [junit4] 2> 757570 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4] 2> 757571 T4034 n:127.0.0.1:42803_ oasc.Overseer.close Overseer (id=null) closing
> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:42803_
> [junit4] 2> 757572 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4] 2> 757573 T4034 n:127.0.0.1:42803_ oasc.Overseer.start Overseer (id=93869083152809987-127.0.0.1:42803_-n_0000000000) starting
> [junit4] 2> 757574 T4034 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
> [junit4] 2> 757576 T4034 n:127.0.0.1:42803_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 757576 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4] 2> 757576 T4064 n:127.0.0.1:42803_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
> [junit4] 2> 757578 T4034 n:127.0.0.1:42803_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
> [junit4] 2> 757579 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores
> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/
> [junit4] 2> 757580 T4034 n:127.0.0.1:42803_ oasc.CorePropertiesLocator.discover Found 1 core definitions
> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 757581 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 757581 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:42803",
> [junit4] 2> "node_name":"127.0.0.1:42803_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"down",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"} current state version: 0
> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:42803",
> [junit4] 2> "node_name":"127.0.0.1:42803_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"down",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"}
> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
> [junit4] 2> 757582 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
> [junit4] 2> 757583 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
> [junit4] 2> 758582 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Looking for collection configName
> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
> [junit4] 2> 758583 T4066 n:127.0.0.1:42803_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 758584 T4066 n:127.0.0.1:42803_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/'
> [junit4] 2> 758598 T4066 n:127.0.0.1:42803_ oasc.Config.<init> loaded config solrconfig.xml with version 0
> [junit4] 2> 758600 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
> [junit4] 2> 758603 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
> [junit4] 2> 758608 T4066 n:127.0.0.1:42803_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 758609 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
> [junit4] 2> 758611 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 758665 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 758666 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 758668 T4066 n:127.0.0.1:42803_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/], dataDir=[null]
> [junit4] 2> 758673 T4066 n:127.0.0.1:42803_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data
> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index/
> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-001/cores/collection1/data/index
> [junit4] 2> 758674 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
> [junit4] 2> 758675 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 758677 T4066 n:127.0.0.1:42803_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
> [junit4] 2> 758686 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 758687 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 758688 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 758689 T4066 n:127.0.0.1:42803_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
> [junit4] 2> 758696 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
> [junit4] 2> 758697 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
> [junit4] 2> 758704 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 758705 T4066 n:127.0.0.1:42803_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@36fd8fde lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7508c674),segFN=segments_1,generation=1}
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@d198739[collection1] main
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
> [junit4] 2> 758706 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905331851100160
> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905331851100160
> [junit4] 2> 758708 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
> [junit4] 2> 758709 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 758710 T4066 n:127.0.0.1:42803_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
> [junit4] 2> 758712 T4066 n:127.0.0.1:42803_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
> [junit4] 2> 758714 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
> [junit4] 2> 758716 T4066 n:127.0.0.1:42803_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
> [junit4] 2> 758726 T4066 n:127.0.0.1:42803_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
> [junit4] 2> 758727 T4067 n:127.0.0.1:42803_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d198739[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> [junit4] 2> 758727 T4066 n:127.0.0.1:42803_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 758727 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42803 collection:tmp_collection shard:shard2
> [junit4] 2> 758727 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 758728 T4034 n:127.0.0.1:42803_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 758728 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
> [junit4] 2> 758730 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4] 2> 758730 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4] 2> 758731 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard2",
> [junit4] 2> "collection":"tmp_collection"} current state version: 1
> [junit4] 2> ASYNC NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@16e04228 url=http://127.0.0.1:42803/collection1 node=127.0.0.1:42803_ C983_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42803, node_name=127.0.0.1:42803_, state=down}
> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42803/collection1/
> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 C983 oasc.SyncStrategy.syncToMe http://127.0.0.1:42803/collection1/ has no replicas
> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42803/collection1/ shard2
> [junit4] 2> 758731 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
> [junit4] 2> 758732 T4062 n:127.0.0.1:42803_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
> [junit4] 2> 758733 T4072 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 758734 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard2",
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "base_url":"http://127.0.0.1:42803",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "state":"active"} current state version: 1
> [junit4] 2> 758778 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1
> [junit4] 2> 758779 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
> [junit4] 2> 758780 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@12138b09{/,null,AVAILABLE}
> [junit4] 2> 758781 T4034 oejs.AbstractConnector.doStart Started ServerConnector@537374e0{HTTP/1.1}{127.0.0.1:47447}
> [junit4] 2> 758781 T4034 oejs.Server.doStart Started @759698ms
> [junit4] 2> 758781 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=47447, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores}
> [junit4] 2> 758781 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
> [junit4] 2> 758781 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/'
> [junit4] 2> 758790 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/solr.xml
> [junit4] 2> 758793 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
> [junit4] 2> 758794 T4034 oasc.CoreContainer.<init> New CoreContainer 957201046
> [junit4] 2> 758794 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/]
> [junit4] 2> 758794 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib
> [junit4] 2> 758794 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/lib).
> [junit4] 2> 758798 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
> [junit4] 2> 758799 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
> [junit4] 2> 758799 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 758799 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 758799 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
> [junit4] 2> 758800 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
> [junit4] 2> 758800 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 758800 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
> [junit4] 2> 758800 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 758802 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7f7cd8 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 758802 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 758802 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
> [junit4] 2> 758803 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 758804 T4089 n:127.0.0.1:47447_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc03e94 name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 758805 T4034 n:127.0.0.1:47447_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 758807 T4034 n:127.0.0.1:47447_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 758835 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 758835 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:42803/collection1/ and leader is http://127.0.0.1:42803/collection1/
> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42803
> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
> [junit4] 2> 758884 T4071 n:127.0.0.1:42803_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 758885 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 758885 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "core":"collection1",
> [junit4] 2> "core_node_name":"core_node1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:42803",
> [junit4] 2> "node_name":"127.0.0.1:42803_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"active",
> [junit4] 2> "shard":"shard2",
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"} current state version: 2
> [junit4] 2> 758886 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
> [junit4] 2> "core":"collection1",
> [junit4] 2> "core_node_name":"core_node1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:42803",
> [junit4] 2> "node_name":"127.0.0.1:42803_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"active",
> [junit4] 2> "shard":"shard2",
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"}
> [junit4] 2> 758987 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 758987 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 759808 T4034 n:127.0.0.1:47447_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47447_
> [junit4] 2> 759809 T4034 n:127.0.0.1:47447_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47447_
> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.Overseer.close Overseer (id=null) closing
> [junit4] 2> 759810 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
> [junit4] 2> 759811 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores
> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/
> [junit4] 2> 759812 T4034 n:127.0.0.1:47447_ oasc.CorePropertiesLocator.discover Found 1 core definitions
> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 759813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 759813 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:47447",
> [junit4] 2> "node_name":"127.0.0.1:47447_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"down",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"} current state version: 3
> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "base_url":"http://127.0.0.1:47447",
> [junit4] 2> "node_name":"127.0.0.1:47447_",
> [junit4] 2> "numShards":"2",
> [junit4] 2> "state":"down",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "operation":"state"}
> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
> [junit4] 2> 759814 T4063 n:127.0.0.1:42803_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 759915 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 759916 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 760813 T4090 n:127.0.0.1:47447_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 760814 T4090 n:127.0.0.1:47447_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/'
> [junit4] 2> 760826 T4090 n:127.0.0.1:47447_ oasc.Config.<init> loaded config solrconfig.xml with version 0
> [junit4] 2> 760829 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
> [junit4] 2> 760831 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 760837 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
> [junit4] 2> 760840 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 760892 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 760893 T4090 n:127.0.0.1:47447_ oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 760894 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 760895 T4090 n:127.0.0.1:47447_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
> [junit4] 2> 760900 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/], dataDir=[null]
> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d9ddd47
> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data
> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index/
> [junit4] 2> 760901 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-002/cores/collection1/data/index
> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
> [junit4] 2> 760902 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 760904 T4090 n:127.0.0.1:47447_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
> [junit4] 2> 760905 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 760906 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 760907 T4090 n:127.0.0.1:47447_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 760908 T4090 n:127.0.0.1:47447_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
> [junit4] 2> 760909 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
> [junit4] 2> 760910 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 760911 T4090 n:127.0.0.1:47447_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=36.8544921875, floorSegmentMB=1.3291015625, forceMergeDeletesPctAllowed=8.97263476447414, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
> [junit4] 2> 760912 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=RawDirectoryWrapper(RAMDirectory@750308ec lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad877f),segFN=segments_1,generation=1}
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5da2fafa[collection1] main
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
> [junit4] 2> 760913 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501905334165307392
> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501905334165307392
> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
> [junit4] 2> 760915 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
> [junit4] 2> 760916 T4090 n:127.0.0.1:47447_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
> [junit4] 2> 760917 T4090 n:127.0.0.1:47447_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
> [junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
> [junit4] 2> 760918 T4091 n:127.0.0.1:47447_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5da2fafa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
> [junit4] 2> 760918 T4090 n:127.0.0.1:47447_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47447 collection:tmp_collection shard:shard1
> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 760919 T4034 n:127.0.0.1:47447_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 760919 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
> [junit4] 2> 760920 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
> [junit4] 2> 760921 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@4e21ece7 url=http://127.0.0.1:47447/collection1 node=127.0.0.1:47447_ C984_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:47447, node_name=127.0.0.1:47447_, state=down}
> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47447/collection1/
> [junit4] 2> 760921 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"tmp_collection"} current state version: 4
> [junit4] 2> 760921 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 C984 oasc.SyncStrategy.syncToMe http://127.0.0.1:47447/collection1/ has no replicas
> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47447/collection1/ shard1
> [junit4] 2> 760922 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
> [junit4] 2> 760922 T4089 n:127.0.0.1:47447_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
> [junit4] 2> 760923 T4062 n:127.0.0.1:42803_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
> [junit4] 2> 760924 T4063 n:127.0.0.1:42803_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
> [junit4] 2> "operation":"leader",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"tmp_collection",
> [junit4] 2> "base_url":"http://127.0.0.1:47447",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "state":"active"} current state version: 4
> [junit4] 2> 760968 T4034 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores/collection1
> [junit4] 2> 760970 T4034 oejs.Server.doStart jetty-9.2.10.v20150310
> [junit4] 2> 760971 T4034 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4fc5b5e4{/,null,AVAILABLE}
> [junit4] 2> 760971 T4034 oejs.AbstractConnector.doStart Started ServerConnector@41389b97{HTTP/1.1}{127.0.0.1:36397}
> [junit4] 2> 760971 T4034 oejs.Server.doStart Started @761888ms
> [junit4] 2> 760971 T4034 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=36397, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores}
> [junit4] 2> 760972 T4034 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
> [junit4] 2> 760972 T4034 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/'
> [junit4] 2> 760980 T4034 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/solr.xml
> [junit4] 2> 760983 T4034 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/cores
> [junit4] 2> 760984 T4034 oasc.CoreContainer.<init> New CoreContainer 327774697
> [junit4] 2> 760984 T4034 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/]
> [junit4] 2> 760984 T4034 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib
> [junit4] 2> 760984 T4034 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest 77F4964252CFD56A-001/jetty-003/lib).
> [junit4] 2> 760988 T4034 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
> [junit4] 2> 760988 T4034 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
> [junit4] 2> 760989 T4034 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 760989 T4034 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 760989 T4034 oasc.CoreContainer.load Node Name: 127.0.0.1
> [junit4] 2> 760989 T4034 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54300/solr
> [junit4] 2> 760990 T4034 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 760990 T4034 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
> [junit4] 2> 760990 T4034 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 760992 T4109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21f437e1 name:ZooKeeperConnection Watcher:127.0.0.1:54300 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 760992 T4034 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 760992 T4034 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
> [junit4] 2> 760994 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 760995 T4112 n:127.0.0.1:36397_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53fdaf5b name:ZooKeeperConnection Watcher:127.0.0.1:54300/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 760995 T4034 n:127.0.0.1:36397_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 760997 T4034 n:127.0.0.1:36397_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 761025 T4062 n:127.0.0.1:42803_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 761025 T4112 n:127.0.0.1:36397_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 761025 T4089 n:127.0.0.1:47447_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:47447/collection1/ and leader is http://127.0.0.1:47447/collection1/
> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47447
> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4] 2> 761074 T4095 n:127.0.0.1:47447_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
> [junit4] 2> 761074 T4095 n:127.0.0.1:474
>
> [...truncated too long message...]
>
> I
> [junit4] ERROR 0.00s J0 | CdcrReplicationDistributedZkTest (suite) <<<
> [junit4] > Throwable #1: java.lang.AssertionError: Some resources were not closed, shutdown, or released.
> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> [junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
> [junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:
> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at java.lang.Thread.sleep(Native Method)
> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > 3) Thread[id=4676, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-EventThread, state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> [junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
> [junit4] > 4) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > 5) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
> [junit4] > 1) Thread[id=4675, name=zkCallback-547-thread-1-processing-{node_name=127.0.0.1:42803_}-SendThread(127.0.0.1:54300), state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at java.lang.Thread.sleep(Native Method)
> [junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
> [junit4] > 2) Thread[id=4719, name=zkCallback-697-thread-2, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > 3) Thread[id=4720, name=zkCallback-697-thread-3, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > 4) Thread[id=4677, name=zkCallback-697-thread-1, state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] > at __randomizedtesting.SeedInfo.seed([77F4964252CFD56A]:0)
> [junit4] Completed [381/502] on J0 in 167.35s, 1 test, 2 failures, 2 errors <<< FAILURES!
>
> [...truncated 376 lines...]
> BUILD FAILED
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
> /home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1985 tests, 3 suite-level errors, 1 failure, 316 ignored (25 assumptions)
>
> Total time: 32 minutes 3 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_45) - Build #
12787 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12787/
Java: 32bit/jdk1.8.0_45 -server -XX:+UseSerialGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest
Error Message:
Some resources were not closed, shutdown, or released.
Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest
Error Message:
5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest: 1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=5939, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-EventThread, state=WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 3) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 4) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:
1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=5939, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-EventThread, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
3) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
4) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
5) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrRequestHandlerTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 3) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
3) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
4) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)
FAILED: org.apache.solr.cloud.CdcrRequestHandlerTest.doTest
Error Message:
expected:<st[art]ed> but was:<st[opp]ed>
Stack Trace:
org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB:AB25C14BCEBE6952]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
at org.apache.solr.cloud.CdcrRequestHandlerTest.doTestLifeCycleActions(CdcrRequestHandlerTest.java:55)
at org.apache.solr.cloud.CdcrRequestHandlerTest.doTest(CdcrRequestHandlerTest.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10437 lines...]
[junit4] Suite: org.apache.solr.cloud.CdcrRequestHandlerTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/init-core-data-001
[junit4] 2> 901044 T5765 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 901044 T5765 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /b_qcl/
[junit4] 2> 901047 T5765 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 901047 T5766 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 901048 T5766 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 901148 T5765 oasc.ZkTestServer.run start zk server on port:56621
[junit4] 2> 901148 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 901148 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 901150 T5773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@125162 name:ZooKeeperConnection Watcher:127.0.0.1:56621 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 901150 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 901151 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 901151 T5765 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 901152 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 901153 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 901154 T5776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e8d39e name:ZooKeeperConnection Watcher:127.0.0.1:56621/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 901154 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 901154 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 901154 T5765 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 901155 T5765 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 901156 T5765 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 901156 T5765 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 901157 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 901157 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 901158 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 901158 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 901159 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 901159 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 901160 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 901160 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 901161 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 901161 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 901162 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 901162 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 901163 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 901163 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 901164 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 901164 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 901165 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 901165 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 901166 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 901166 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 901167 T5765 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 901167 T5765 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 901168 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 901168 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 901169 T5779 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d99e96 name:ZooKeeperConnection Watcher:127.0.0.1:56621/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 901169 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 901170 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 901170 T5765 oas.SolrTestCaseJ4.setUp ###Starting doTest
[junit4] 2> 901223 T5765 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1
[junit4] 2> 901224 T5765 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 901235 T5765 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@1454455{/b_qcl,null,AVAILABLE}
[junit4] 2> 901238 T5765 oejs.AbstractConnector.doStart Started ServerConnector@1e2aeaa{SSL-http/1.1}{127.0.0.1:59558}
[junit4] 2> 901238 T5765 oejs.Server.doStart Started @902185ms
[junit4] 2> 901239 T5765 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/b_qcl, hostPort=59558, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores}
[junit4] 2> 901240 T5765 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 901240 T5765 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/'
[junit4] 2> 901249 T5765 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/solr.xml
[junit4] 2> 901251 T5765 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores
[junit4] 2> 901252 T5765 oasc.CoreContainer.<init> New CoreContainer 10161392
[junit4] 2> 901252 T5765 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/]
[junit4] 2> 901252 T5765 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/lib
[junit4] 2> 901253 T5765 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/lib).
[junit4] 2> 901257 T5765 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 901258 T5765 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 901259 T5765 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 901260 T5765 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 901260 T5765 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 901260 T5765 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56621/solr
[junit4] 2> 901260 T5765 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 901261 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 901261 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 901263 T5793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@198fc76 name:ZooKeeperConnection Watcher:127.0.0.1:56621 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 901263 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 901263 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 901265 T5765 n:127.0.0.1:59558_b_qcl oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 901265 T5796 n:127.0.0.1:59558_b_qcl oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cace86 name:ZooKeeperConnection Watcher:127.0.0.1:56621/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 901265 T5765 n:127.0.0.1:59558_b_qcl oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 901266 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 901267 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 901268 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 901269 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 901270 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 901271 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 901272 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 901273 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 901273 T5765 n:127.0.0.1:59558_b_qcl oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59558_b_qcl
[junit4] 2> 901274 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59558_b_qcl
[junit4] 2> 901275 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 901275 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 901276 T5765 n:127.0.0.1:59558_b_qcl oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 901277 T5765 n:127.0.0.1:59558_b_qcl oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59558_b_qcl
[junit4] 2> 901277 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 901278 T5765 n:127.0.0.1:59558_b_qcl oasc.Overseer.start Overseer (id=93871421065199620-127.0.0.1:59558_b_qcl-n_0000000000) starting
[junit4] 2> 901279 T5765 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 901281 T5765 n:127.0.0.1:59558_b_qcl oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 901282 T5798 n:127.0.0.1:59558_b_qcl oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 901282 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 901282 T5765 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 901284 T5765 n:127.0.0.1:59558_b_qcl oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 901284 T5765 n:127.0.0.1:59558_b_qcl oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 901284 T5765 n:127.0.0.1:59558_b_qcl oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores
[junit4] 2> 901285 T5765 n:127.0.0.1:59558_b_qcl oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 901285 T5765 n:127.0.0.1:59558_b_qcl oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/
[junit4] 2> 901286 T5765 n:127.0.0.1:59558_b_qcl oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 901286 T5800 n:127.0.0.1:59558_b_qcl c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 901287 T5800 n:127.0.0.1:59558_b_qcl c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 901287 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 901287 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 901287 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59558/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:59558_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 901288 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59558/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:59558_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 901288 T5797 n:127.0.0.1:59558_b_qcl oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
[junit4] 2> 901288 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 901289 T5796 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 902287 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 902288 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 902288 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
[junit4] 2> 902288 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.getConfName Looking for collection configName
[junit4] 2> 902289 T5800 n:127.0.0.1:59558_b_qcl oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4] 2> 902289 T5800 n:127.0.0.1:59558_b_qcl oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
[junit4] 2> 902289 T5800 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 902290 T5800 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 902290 T5800 n:127.0.0.1:59558_b_qcl oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/'
[junit4] 2> 902296 T5800 n:127.0.0.1:59558_b_qcl oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 902299 T5800 n:127.0.0.1:59558_b_qcl oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 902301 T5800 n:127.0.0.1:59558_b_qcl oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 902305 T5800 n:127.0.0.1:59558_b_qcl oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 902306 T5800 n:127.0.0.1:59558_b_qcl oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 902309 T5800 n:127.0.0.1:59558_b_qcl oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 902379 T5800 n:127.0.0.1:59558_b_qcl oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 902380 T5800 n:127.0.0.1:59558_b_qcl oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 902381 T5800 n:127.0.0.1:59558_b_qcl oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 902382 T5800 n:127.0.0.1:59558_b_qcl oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 902388 T5800 n:127.0.0.1:59558_b_qcl oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 902388 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 902388 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/], dataDir=[null]
[junit4] 2> 902389 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ecaad6
[junit4] 2> 902389 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/data
[junit4] 2> 902389 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/data/index/
[junit4] 2> 902389 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 902390 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-001/cores/collection1/data/index
[junit4] 2> 902390 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.1572265625, floorSegmentMB=1.3603515625, forceMergeDeletesPctAllowed=20.089493827841707, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 902390 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@112c54f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ba052),segFN=segments_1,generation=1}
[junit4] 2> 902390 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 902392 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 902401 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 902402 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 902402 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 902403 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 902403 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 902404 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 902411 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 902411 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 902412 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 902412 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 902418 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 902418 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 902419 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 902419 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 902420 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 902420 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@112c54f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ba052),segFN=segments_1,generation=1}
[junit4] 2> 902420 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 902420 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@5f6d1e[collection1] main
[junit4] 2> 902433 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 902433 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 902434 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 902434 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501942738496520192
[junit4] 2> 902436 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 3 ms to seed version buckets with highest version 1501942738496520192
[junit4] 2> 902436 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 902436 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 902436 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 902436 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 902437 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 902437 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 902437 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 902437 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 902437 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 902439 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
[junit4] 2> 902441 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 902444 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
[junit4] 2> 902453 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 902453 T5801 n:127.0.0.1:59558_b_qcl x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f6d1e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 902453 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 902454 T5800 n:127.0.0.1:59558_b_qcl x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 902454 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:59558/b_qcl collection:tmp_collection shard:shard2
[junit4] 2> 902454 T5765 n:127.0.0.1:59558_b_qcl oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 902455 T5765 n:127.0.0.1:59558_b_qcl oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 902455 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
[junit4] 2> 902457 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 902457 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 902457 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 902457 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 902458 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C1775 name=collection1 org.apache.solr.core.SolrCore@2a80da url=https://127.0.0.1:59558/b_qcl/collection1 node=127.0.0.1:59558_b_qcl C1775_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:59558/b_qcl, node_name=127.0.0.1:59558_b_qcl, state=down}
[junit4] 2> 902458 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 C1775 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:59558/b_qcl/collection1/
[junit4] 2> 902458 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 C1775 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 902458 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 C1775 oasc.SyncStrategy.syncToMe https://127.0.0.1:59558/b_qcl/collection1/ has no replicas
[junit4] 2> 902458 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:59558/b_qcl/collection1/ shard2
[junit4] 2> 902459 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
[junit4] 2> 902460 T5796 n:127.0.0.1:59558_b_qcl oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
[junit4] 2> 902461 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 902461 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"https://127.0.0.1:59558/b_qcl",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 902521 T5765 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1
[junit4] 2> 902523 T5765 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 902524 T5765 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@18bd2e9{/b_qcl,null,AVAILABLE}
[junit4] 2> 902527 T5765 oejs.AbstractConnector.doStart Started ServerConnector@ae6c60{SSL-http/1.1}{127.0.0.1:54028}
[junit4] 2> 902527 T5765 oejs.Server.doStart Started @903474ms
[junit4] 2> 902528 T5765 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/b_qcl, hostPort=54028, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores}
[junit4] 2> 902528 T5765 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 902529 T5765 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/'
[junit4] 2> 902536 T5765 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/solr.xml
[junit4] 2> 902539 T5765 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores
[junit4] 2> 902540 T5765 oasc.CoreContainer.<init> New CoreContainer 15191010
[junit4] 2> 902540 T5765 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/]
[junit4] 2> 902540 T5765 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/lib
[junit4] 2> 902541 T5765 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/lib).
[junit4] 2> 902544 T5765 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 902546 T5765 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 902547 T5765 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 902547 T5765 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 902547 T5765 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 902547 T5765 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56621/solr
[junit4] 2> 902548 T5765 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 902548 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 902548 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 902549 T5819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@393ec3 name:ZooKeeperConnection Watcher:127.0.0.1:56621 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 902550 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 902550 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 902551 T5765 n:127.0.0.1:54028_b_qcl oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 902552 T5822 n:127.0.0.1:54028_b_qcl oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef75b name:ZooKeeperConnection Watcher:127.0.0.1:56621/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 902552 T5765 n:127.0.0.1:54028_b_qcl oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 902554 T5765 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 902562 T5796 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 902562 T5822 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 902611 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are https://127.0.0.1:59558/b_qcl/collection1/ and leader is https://127.0.0.1:59558/b_qcl/collection1/
[junit4] 2> 902612 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:59558/b_qcl
[junit4] 2> 902612 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 902612 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 902612 T5805 n:127.0.0.1:59558_b_qcl c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 902612 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 902613 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59558/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:59558_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 902613 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:59558/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:59558_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 902714 T5796 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 902714 T5822 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 903556 T5765 n:127.0.0.1:54028_b_qcl oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54028_b_qcl
[junit4] 2> 903556 T5765 n:127.0.0.1:54028_b_qcl oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54028_b_qcl
[junit4] 2> 903557 T5765 n:127.0.0.1:54028_b_qcl oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 903558 T5765 n:127.0.0.1:54028_b_qcl oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 903559 T5765 n:127.0.0.1:54028_b_qcl oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 903559 T5765 n:127.0.0.1:54028_b_qcl oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores
[junit4] 2> 903560 T5765 n:127.0.0.1:54028_b_qcl oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 903560 T5765 n:127.0.0.1:54028_b_qcl oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/
[junit4] 2> 903560 T5765 n:127.0.0.1:54028_b_qcl oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 903561 T5823 n:127.0.0.1:54028_b_qcl c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 903561 T5823 n:127.0.0.1:54028_b_qcl c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 903562 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 903562 T5823 n:127.0.0.1:54028_b_qcl oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 903562 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54028/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:54028_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 903562 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"https://127.0.0.1:54028/b_qcl",
[junit4] 2> "node_name":"127.0.0.1:54028_b_qcl",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 903563 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 903563 T5797 n:127.0.0.1:59558_b_qcl oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 903664 T5796 n:127.0.0.1:59558_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 903664 T5822 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 904562 T5823 n:127.0.0.1:54028_b_qcl oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 904562 T5823 n:127.0.0.1:54028_b_qcl oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 904563 T5823 n:127.0.0.1:54028_b_qcl oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 904563 T5823 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 904563 T5823 n:127.0.0.1:54028_b_qcl oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 904564 T5823 n:127.0.0.1:54028_b_qcl oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/'
[junit4] 2> 904570 T5823 n:127.0.0.1:54028_b_qcl oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 904573 T5823 n:127.0.0.1:54028_b_qcl oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 904575 T5823 n:127.0.0.1:54028_b_qcl oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 904579 T5823 n:127.0.0.1:54028_b_qcl oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 904580 T5823 n:127.0.0.1:54028_b_qcl oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 904583 T5823 n:127.0.0.1:54028_b_qcl oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 904638 T5823 n:127.0.0.1:54028_b_qcl oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 904639 T5823 n:127.0.0.1:54028_b_qcl oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 904639 T5823 n:127.0.0.1:54028_b_qcl oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 904641 T5823 n:127.0.0.1:54028_b_qcl oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 904646 T5823 n:127.0.0.1:54028_b_qcl oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 904646 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 904646 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/], dataDir=[null]
[junit4] 2> 904646 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ecaad6
[junit4] 2> 904647 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/data
[junit4] 2> 904647 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/data/index/
[junit4] 2> 904647 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 904647 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-002/cores/collection1/data/index
[junit4] 2> 904648 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.1572265625, floorSegmentMB=1.3603515625, forceMergeDeletesPctAllowed=20.089493827841707, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 904648 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@19c5b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f65aa),segFN=segments_1,generation=1}
[junit4] 2> 904648 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 904650 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 904650 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 904652 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 904652 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 904653 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 904653 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 904654 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 904670 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 904670 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 904670 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 904671 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 904671 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 904671 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 904672 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 904672 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 904673 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=33, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 904673 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@19c5b5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f65aa),segFN=segments_1,generation=1}
[junit4] 2> 904673 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 904674 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@ba5199[collection1] main
[junit4] 2> 904674 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 904674 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 904674 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 904674 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501942740845330432
[junit4] 2> 904676 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501942740845330432
[junit4] 2> 904676 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 904676 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 904677 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 904677 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 904677 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 904677 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 904677 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 904678 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 904678 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 904679 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 904679 T5824 n:127.0.0.1:54028_b_qcl x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ba5199[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 904680 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 904680 T5823 n:127.0.0.1:54028_b_qcl x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 904680 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:54028/b_qcl collection:tmp_collection shard:shard1
[junit4] 2> 904680 T5765 n:127.0.0.1:54028_b_qcl oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 904681 T5765 n:127.0.0.1:54028_b_qcl oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 904681 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
[junit4] 2> 904683 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 904683 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 904683 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 904683 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 904684 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection"} current state version: 4
[junit4] 2> ASYNC NEW_CORE C1776 name=collection1 org.apache.solr.core.SolrCore@ce4388 url=https://127.0.0.1:54028/b_qcl/collection1 node=127.0.0.1:54028_b_qcl C1776_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:54028/b_qcl, node_name=127.0.0.1:54028_b_qcl, state=down}
[junit4] 2> 904684 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 C1776 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:54028/b_qcl/collection1/
[junit4] 2> 904684 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 C1776 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 904684 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 C1776 oasc.SyncStrategy.syncToMe https://127.0.0.1:54028/b_qcl/collection1/ has no replicas
[junit4] 2> 904684 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:54028/b_qcl/collection1/ shard1
[junit4] 2> 904684 T5828 n:127.0.0.1:54028_b_qcl c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
[junit4] 2> 904685 T5822 n:127.0.0.1:54028_b_qcl oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
[junit4] 2> 904686 T5796 n:127.0.0.1:59558_b_qcl oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 904686 T5797 n:127.0.0.1:59558_b_qcl oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"https://127.0.0.1:54028/b_qcl",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 904733 T5765 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/cores/collection1
[junit4] 2> 904734 T5765 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 904735 T5765 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@11e79b3{/b_qcl,null,AVAILABLE}
[junit4] 2> 904737 T5765 oejs.AbstractConnector.doStart Started ServerConnector@18c37e{SSL-http/1.1}{127.0.0.1:38465}
[junit4] 2> 904738 T5765 oejs.Server.doStart Started @905684ms
[junit4] 2> 904738 T5765 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/b_qcl, hostPort=38465, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/cores}
[junit4] 2> 904739 T5765 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 904739 T5765 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/'
[junit4] 2> 904747 T5765 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/solr.xml
[junit4] 2> 904750 T5765 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/cores
[junit4] 2> 904750 T5765 oasc.CoreContainer.<init> New CoreContainer 13971017
[junit4] 2> 904750 T5765 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/]
[junit4] 2> 904750 T5765 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/lib
[junit4] 2> 904751 T5765 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrRequestHandlerTest C6179EFA3057AEB-001/jetty-003/lib).
[junit4] 2> 904754 T5765 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 904756 T5765 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 904757 T5765 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 904757 T5765 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 904757 T5765 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 904758 T5765 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56621/solr
[junit4] 2> 904758 T5765 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 904758 T5765 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 904759 T5765 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 904760 T5842 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e5b878 name:ZooKeeperConnection Watcher:127.0.0.1:56621 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 904760 T5765 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 904760 T5765 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 904762 T5765 n:127.0.0.1:38465_b_qcl oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 904763 T5845 n:127.0.0.1:38465_b_qcl oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14e0c4c name:ZooKeeperConnection Watcher:127.0.0.1:56621/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4]
[...truncated too long message...]
nError: Some resources were not closed, shutdown, or released.
[junit4] > at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrRequestHandlerTest:
[junit4] > 1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 2) Thread[id=5939, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-EventThread, state=WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > 3) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 4) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 5) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=5940, name=zkCallback-696-thread-1, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 2) Thread[id=5938, name=zkCallback-667-thread-1-processing-{node_name=127.0.0.1:59558_b_qcl}-SendThread(127.0.0.1:56621), state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 3) Thread[id=5966, name=zkCallback-696-thread-2, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 4) Thread[id=5967, name=zkCallback-696-thread-3, state=TIMED_WAITING, group=TGRP-CdcrRequestHandlerTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at __randomizedtesting.SeedInfo.seed([C6179EFA3057AEB]:0)
[junit4] Completed [388/502] on J0 in 54.75s, 1 test, 2 failures, 2 errors <<< FAILURES!
[...truncated 371 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1986 tests, 3 suite-level errors, 1 failure, 344 ignored (25 assumptions)
Total time: 38 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_60-ea-b12) -
Build # 12786 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12786/
Java: 32bit/jdk1.8.0_60-ea-b12 -server -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
Error Message:
expected:<st[art]ed> but was:<st[opp]ed>
Stack Trace:
org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
at __randomizedtesting.SeedInfo.seed([EF101301FFD92986:4854ABA592623A3F]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10422 lines...]
[junit4] Suite: org.apache.solr.cloud.CdcrReplicationDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/init-core-data-001
[junit4] 2> 995425 T5954 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 995425 T5954 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 995427 T5954 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 995427 T5955 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 995427 T5955 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 995527 T5954 oasc.ZkTestServer.run start zk server on port:49382
[junit4] 2> 995528 T5954 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 995528 T5954 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995530 T5962 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@512b74 name:ZooKeeperConnection Watcher:127.0.0.1:49382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995530 T5954 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995531 T5954 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 995531 T5954 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 995532 T5954 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 995533 T5954 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995534 T5965 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b90dc6 name:ZooKeeperConnection Watcher:127.0.0.1:49382/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995534 T5954 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995534 T5954 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 995535 T5954 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 995536 T5954 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 995537 T5954 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 995537 T5954 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 995538 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 995539 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 995540 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 995541 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 995541 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 995542 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 995543 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 995543 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 995544 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 995544 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 995545 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 995545 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 995546 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 995547 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 995547 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 995548 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 995549 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 995549 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 995550 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 995550 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 995551 T5954 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 995552 T5954 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 995553 T5954 oas.SolrTestCaseJ4.setUp ###Starting doTest
[junit4] 2> 995608 T5954 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1
[junit4] 2> 995609 T5954 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 995610 T5954 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@22c7a7{/,null,AVAILABLE}
[junit4] 2> 995611 T5954 oejs.AbstractConnector.doStart Started ServerConnector@35eb00{HTTP/1.1}{127.0.0.1:52895}
[junit4] 2> 995611 T5954 oejs.Server.doStart Started @996521ms
[junit4] 2> 995611 T5954 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=52895, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores}
[junit4] 2> 995612 T5954 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 995612 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/'
[junit4] 2> 995622 T5954 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/solr.xml
[junit4] 2> 995625 T5954 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores
[junit4] 2> 995626 T5954 oasc.CoreContainer.<init> New CoreContainer 24097979
[junit4] 2> 995626 T5954 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/]
[junit4] 2> 995626 T5954 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/lib
[junit4] 2> 995627 T5954 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/lib).
[junit4] 2> 995632 T5954 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 995633 T5954 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 995634 T5954 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 995635 T5954 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 995636 T5954 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 995636 T5954 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49382/solr
[junit4] 2> 995636 T5954 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 995637 T5954 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 995637 T5954 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995638 T5979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16bddc7 name:ZooKeeperConnection Watcher:127.0.0.1:49382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995638 T5954 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995639 T5954 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 995640 T5954 n:127.0.0.1:52895_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 995641 T5982 n:127.0.0.1:52895_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d8f64 name:ZooKeeperConnection Watcher:127.0.0.1:49382/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 995641 T5954 n:127.0.0.1:52895_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 995643 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 995644 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 995645 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 995647 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 995649 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 995651 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 995652 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 995653 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 995654 T5954 n:127.0.0.1:52895_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52895_
[junit4] 2> 995655 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52895_
[junit4] 2> 995656 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 995657 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 995657 T5954 n:127.0.0.1:52895_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 995658 T5954 n:127.0.0.1:52895_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52895_
[junit4] 2> 995659 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 995660 T5954 n:127.0.0.1:52895_ oasc.Overseer.start Overseer (id=93871025939808259-127.0.0.1:52895_-n_0000000000) starting
[junit4] 2> 995661 T5954 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 995663 T5954 n:127.0.0.1:52895_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 995664 T5984 n:127.0.0.1:52895_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 995664 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 995664 T5954 n:127.0.0.1:52895_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 995666 T5954 n:127.0.0.1:52895_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 995666 T5954 n:127.0.0.1:52895_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 995667 T5954 n:127.0.0.1:52895_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores
[junit4] 2> 995668 T5954 n:127.0.0.1:52895_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 995668 T5954 n:127.0.0.1:52895_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/
[junit4] 2> 995669 T5954 n:127.0.0.1:52895_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 995669 T5986 n:127.0.0.1:52895_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 995669 T5986 n:127.0.0.1:52895_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 995670 T5986 n:127.0.0.1:52895_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 995670 T5982 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 995670 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:52895",
[junit4] 2> "node_name":"127.0.0.1:52895_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 995671 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:52895",
[junit4] 2> "node_name":"127.0.0.1:52895_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 995671 T5983 n:127.0.0.1:52895_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
[junit4] 2> 995671 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 995672 T5982 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 996670 T5986 n:127.0.0.1:52895_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 996670 T5986 n:127.0.0.1:52895_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 996671 T5986 n:127.0.0.1:52895_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
[junit4] 2> 996671 T5986 n:127.0.0.1:52895_ oasc.ZkController.getConfName Looking for collection configName
[junit4] 2> 996672 T5986 n:127.0.0.1:52895_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4] 2> 996672 T5986 n:127.0.0.1:52895_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
[junit4] 2> 996672 T5986 n:127.0.0.1:52895_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 996673 T5986 n:127.0.0.1:52895_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 996673 T5986 n:127.0.0.1:52895_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/'
[junit4] 2> 996684 T5986 n:127.0.0.1:52895_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 996688 T5986 n:127.0.0.1:52895_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 996692 T5986 n:127.0.0.1:52895_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 996699 T5986 n:127.0.0.1:52895_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 996700 T5986 n:127.0.0.1:52895_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 996703 T5986 n:127.0.0.1:52895_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 996772 T5986 n:127.0.0.1:52895_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 996773 T5986 n:127.0.0.1:52895_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 996774 T5986 n:127.0.0.1:52895_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 996776 T5986 n:127.0.0.1:52895_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 996783 T5986 n:127.0.0.1:52895_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 996783 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 996784 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/], dataDir=[null]
[junit4] 2> 996784 T5986 n:127.0.0.1:52895_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@859418
[junit4] 2> 996784 T5986 n:127.0.0.1:52895_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/data
[junit4] 2> 996785 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/data/index/
[junit4] 2> 996785 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 996785 T5986 n:127.0.0.1:52895_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-001/cores/collection1/data/index
[junit4] 2> 996785 T5986 n:127.0.0.1:52895_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2000306261, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 996786 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@17e5863 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12836d4),segFN=segments_1,generation=1}
[junit4] 2> 996786 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 996789 T5986 n:127.0.0.1:52895_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 996789 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 996790 T5986 n:127.0.0.1:52895_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 996791 T5986 n:127.0.0.1:52895_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 996792 T5986 n:127.0.0.1:52895_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 996792 T5986 n:127.0.0.1:52895_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 996793 T5986 n:127.0.0.1:52895_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 996794 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 996795 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 996795 T5986 n:127.0.0.1:52895_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 996796 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 996796 T5986 n:127.0.0.1:52895_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 996796 T5986 n:127.0.0.1:52895_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 996797 T5986 n:127.0.0.1:52895_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 996797 T5986 n:127.0.0.1:52895_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 996798 T5986 n:127.0.0.1:52895_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=74.111328125, floorSegmentMB=1.8125, forceMergeDeletesPctAllowed=18.09961478163088, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21562580592977343
[junit4] 2> 996798 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@17e5863 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@12836d4),segFN=segments_1,generation=1}
[junit4] 2> 996798 T5986 n:127.0.0.1:52895_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 996798 T5986 n:127.0.0.1:52895_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@331e7[collection1] main
[junit4] 2> 996799 T5986 n:127.0.0.1:52895_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 996799 T5986 n:127.0.0.1:52895_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 996799 T5986 n:127.0.0.1:52895_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 996799 T5986 n:127.0.0.1:52895_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501936416473481216
[junit4] 2> 996801 T5986 n:127.0.0.1:52895_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501936416473481216
[junit4] 2> 996801 T5986 n:127.0.0.1:52895_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 996802 T5986 n:127.0.0.1:52895_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 996802 T5986 n:127.0.0.1:52895_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 996802 T5986 n:127.0.0.1:52895_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 996802 T5986 n:127.0.0.1:52895_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 996803 T5986 n:127.0.0.1:52895_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 996803 T5986 n:127.0.0.1:52895_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 996803 T5986 n:127.0.0.1:52895_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 996803 T5986 n:127.0.0.1:52895_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 996804 T5986 n:127.0.0.1:52895_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
[junit4] 2> 996805 T5986 n:127.0.0.1:52895_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 996806 T5986 n:127.0.0.1:52895_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
[junit4] 2> 996806 T5986 n:127.0.0.1:52895_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 996807 T5986 n:127.0.0.1:52895_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 996807 T5987 n:127.0.0.1:52895_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@331e7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 996807 T5986 n:127.0.0.1:52895_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 996808 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52895 collection:tmp_collection shard:shard2
[junit4] 2> 996808 T5954 n:127.0.0.1:52895_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 996808 T5954 n:127.0.0.1:52895_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 996808 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
[junit4] 2> 996811 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 996812 T5982 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 996812 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 996812 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 996813 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C2061 name=collection1 org.apache.solr.core.SolrCore@1561f4c url=http://127.0.0.1:52895/collection1 node=127.0.0.1:52895_ C2061_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:52895, node_name=127.0.0.1:52895_, state=down}
[junit4] 2> 996813 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 C2061 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52895/collection1/
[junit4] 2> 996813 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 C2061 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 996813 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 C2061 oasc.SyncStrategy.syncToMe http://127.0.0.1:52895/collection1/ has no replicas
[junit4] 2> 996814 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52895/collection1/ shard2
[junit4] 2> 996814 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
[junit4] 2> 996824 T5982 n:127.0.0.1:52895_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
[junit4] 2> 996825 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 996826 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:52895",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 996873 T5954 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1
[junit4] 2> 996874 T5954 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 996875 T5954 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@1eb596f{/,null,AVAILABLE}
[junit4] 2> 996876 T5954 oejs.AbstractConnector.doStart Started ServerConnector@1759b83{HTTP/1.1}{127.0.0.1:33270}
[junit4] 2> 996876 T5954 oejs.Server.doStart Started @997787ms
[junit4] 2> 996876 T5954 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=33270, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores}
[junit4] 2> 996877 T5954 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 996877 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/'
[junit4] 2> 996887 T5954 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/solr.xml
[junit4] 2> 996890 T5954 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores
[junit4] 2> 996891 T5954 oasc.CoreContainer.<init> New CoreContainer 33303385
[junit4] 2> 996891 T5954 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/]
[junit4] 2> 996892 T5954 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/lib
[junit4] 2> 996892 T5954 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/lib).
[junit4] 2> 996897 T5954 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 996898 T5954 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 996898 T5954 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 996899 T5954 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 996899 T5954 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 996899 T5954 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49382/solr
[junit4] 2> 996900 T5954 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 996900 T5954 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 996901 T5954 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 996901 T6006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@132c388 name:ZooKeeperConnection Watcher:127.0.0.1:49382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 996902 T5954 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 996902 T5954 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 996904 T5954 n:127.0.0.1:33270_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 996904 T6009 n:127.0.0.1:33270_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@140cf15 name:ZooKeeperConnection Watcher:127.0.0.1:49382/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 996905 T5954 n:127.0.0.1:33270_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 996907 T5954 n:127.0.0.1:33270_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 996928 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 996928 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 996975 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:52895/collection1/ and leader is http://127.0.0.1:52895/collection1/
[junit4] 2> 996976 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52895
[junit4] 2> 996976 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 996976 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 996976 T5991 n:127.0.0.1:52895_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 996977 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 996977 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:52895",
[junit4] 2> "node_name":"127.0.0.1:52895_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 996978 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:52895",
[junit4] 2> "node_name":"127.0.0.1:52895_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 997079 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997079 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 997910 T5954 n:127.0.0.1:33270_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33270_
[junit4] 2> 997910 T5954 n:127.0.0.1:33270_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33270_
[junit4] 2> 997911 T5954 n:127.0.0.1:33270_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 997912 T5954 n:127.0.0.1:33270_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 997913 T5954 n:127.0.0.1:33270_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 997913 T5954 n:127.0.0.1:33270_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores
[junit4] 2> 997914 T5954 n:127.0.0.1:33270_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 997914 T5954 n:127.0.0.1:33270_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/
[junit4] 2> 997914 T5954 n:127.0.0.1:33270_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 997915 T6010 n:127.0.0.1:33270_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 997915 T6010 n:127.0.0.1:33270_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 997916 T6010 n:127.0.0.1:33270_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 997916 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 997916 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:33270",
[junit4] 2> "node_name":"127.0.0.1:33270_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 997917 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:33270",
[junit4] 2> "node_name":"127.0.0.1:33270_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 997917 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 997917 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 998018 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 998018 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 998916 T6010 n:127.0.0.1:33270_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 998916 T6010 n:127.0.0.1:33270_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 998917 T6010 n:127.0.0.1:33270_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 998917 T6010 n:127.0.0.1:33270_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 998917 T6010 n:127.0.0.1:33270_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 998917 T6010 n:127.0.0.1:33270_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/'
[junit4] 2> 998924 T6010 n:127.0.0.1:33270_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 998927 T6010 n:127.0.0.1:33270_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 998930 T6010 n:127.0.0.1:33270_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 998934 T6010 n:127.0.0.1:33270_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 998935 T6010 n:127.0.0.1:33270_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 998937 T6010 n:127.0.0.1:33270_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 998994 T6010 n:127.0.0.1:33270_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 998995 T6010 n:127.0.0.1:33270_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 998996 T6010 n:127.0.0.1:33270_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 998997 T6010 n:127.0.0.1:33270_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 999003 T6010 n:127.0.0.1:33270_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 999003 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 999003 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/], dataDir=[null]
[junit4] 2> 999003 T6010 n:127.0.0.1:33270_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@859418
[junit4] 2> 999004 T6010 n:127.0.0.1:33270_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/data
[junit4] 2> 999004 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/data/index/
[junit4] 2> 999004 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 999004 T6010 n:127.0.0.1:33270_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-002/cores/collection1/data/index
[junit4] 2> 999005 T6010 n:127.0.0.1:33270_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2000306261, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 999005 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@27a759 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ebf5dc),segFN=segments_1,generation=1}
[junit4] 2> 999006 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 999008 T6010 n:127.0.0.1:33270_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 999008 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 999009 T6010 n:127.0.0.1:33270_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 999010 T6010 n:127.0.0.1:33270_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 999011 T6010 n:127.0.0.1:33270_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 999026 T6010 n:127.0.0.1:33270_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 999027 T6010 n:127.0.0.1:33270_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 999030 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 999030 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 999031 T6010 n:127.0.0.1:33270_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 999031 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 999032 T6010 n:127.0.0.1:33270_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 999032 T6010 n:127.0.0.1:33270_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 999033 T6010 n:127.0.0.1:33270_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 999033 T6010 n:127.0.0.1:33270_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 999034 T6010 n:127.0.0.1:33270_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=74.111328125, floorSegmentMB=1.8125, forceMergeDeletesPctAllowed=18.09961478163088, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21562580592977343
[junit4] 2> 999035 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@27a759 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ebf5dc),segFN=segments_1,generation=1}
[junit4] 2> 999035 T6010 n:127.0.0.1:33270_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 999035 T6010 n:127.0.0.1:33270_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@188f16[collection1] main
[junit4] 2> 999035 T6010 n:127.0.0.1:33270_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 999036 T6010 n:127.0.0.1:33270_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 999036 T6010 n:127.0.0.1:33270_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 999036 T6010 n:127.0.0.1:33270_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501936418819145728
[junit4] 2> 999038 T6010 n:127.0.0.1:33270_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501936418819145728
[junit4] 2> 999038 T6010 n:127.0.0.1:33270_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 999039 T6010 n:127.0.0.1:33270_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 999039 T6010 n:127.0.0.1:33270_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 999039 T6010 n:127.0.0.1:33270_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 999040 T6010 n:127.0.0.1:33270_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 999040 T6010 n:127.0.0.1:33270_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 999040 T6010 n:127.0.0.1:33270_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 999040 T6010 n:127.0.0.1:33270_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 999040 T6010 n:127.0.0.1:33270_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 999041 T6010 n:127.0.0.1:33270_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 999042 T6011 n:127.0.0.1:33270_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@188f16[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 999042 T6010 n:127.0.0.1:33270_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 999042 T6010 n:127.0.0.1:33270_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 999043 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33270 collection:tmp_collection shard:shard1
[junit4] 2> 999043 T5954 n:127.0.0.1:33270_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 999043 T5954 n:127.0.0.1:33270_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 999043 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
[junit4] 2> 999045 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 999045 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 999045 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 999046 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 999046 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection"} current state version: 4
[junit4] 2> ASYNC NEW_CORE C2062 name=collection1 org.apache.solr.core.SolrCore@18c3213 url=http://127.0.0.1:33270/collection1 node=127.0.0.1:33270_ C2062_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:33270, node_name=127.0.0.1:33270_, state=down}
[junit4] 2> 999046 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 C2062 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33270/collection1/
[junit4] 2> 999046 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 C2062 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 999046 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 C2062 oasc.SyncStrategy.syncToMe http://127.0.0.1:33270/collection1/ has no replicas
[junit4] 2> 999046 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33270/collection1/ shard1
[junit4] 2> 999047 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
[junit4] 2> 999047 T6009 n:127.0.0.1:33270_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
[junit4] 2> 999048 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 999049 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:33270",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 999093 T5954 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1
[junit4] 2> 999094 T5954 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 999095 T5954 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@19d1cc4{/,null,AVAILABLE}
[junit4] 2> 999096 T5954 oejs.AbstractConnector.doStart Started ServerConnector@12a23ed{HTTP/1.1}{127.0.0.1:47313}
[junit4] 2> 999096 T5954 oejs.Server.doStart Started @1000006ms
[junit4] 2> 999096 T5954 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=47313, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores}
[junit4] 2> 999097 T5954 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
[junit4] 2> 999097 T5954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/'
[junit4] 2> 999105 T5954 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/solr.xml
[junit4] 2> 999108 T5954 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores
[junit4] 2> 999108 T5954 oasc.CoreContainer.<init> New CoreContainer 23191402
[junit4] 2> 999108 T5954 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/]
[junit4] 2> 999109 T5954 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/lib
[junit4] 2> 999109 T5954 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/lib).
[junit4] 2> 999113 T5954 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 999114 T5954 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 999114 T5954 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 999114 T5954 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 999115 T5954 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 999115 T5954 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49382/solr
[junit4] 2> 999115 T5954 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 999115 T5954 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 999116 T5954 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 999117 T6029 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@136d9b name:ZooKeeperConnection Watcher:127.0.0.1:49382 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 999117 T5954 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 999118 T5954 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 999118 T5954 n:127.0.0.1:47313_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 999119 T6032 n:127.0.0.1:47313_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@140f9c6 name:ZooKeeperConnection Watcher:127.0.0.1:49382/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 999119 T5954 n:127.0.0.1:47313_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 999121 T5954 n:127.0.0.1:47313_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 999150 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 999150 T6032 n:127.0.0.1:47313_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 999150 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 999199 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:33270/collection1/ and leader is http://127.0.0.1:33270/collection1/
[junit4] 2> 999199 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33270
[junit4] 2> 999199 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 999199 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 999199 T6015 n:127.0.0.1:33270_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 999200 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 999200 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:33270",
[junit4] 2> "node_name":"127.0.0.1:33270_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 5
[junit4] 2> 999200 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:33270",
[junit4] 2> "node_name":"127.0.0.1:33270_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 999301 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 999301 T6032 n:127.0.0.1:47313_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 999301 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1000122 T5954 n:127.0.0.1:47313_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47313_
[junit4] 2> 1000123 T5954 n:127.0.0.1:47313_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47313_
[junit4] 2> 1000124 T5954 n:127.0.0.1:47313_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1000125 T5954 n:127.0.0.1:47313_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 1000126 T5954 n:127.0.0.1:47313_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 1000126 T5954 n:127.0.0.1:47313_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores
[junit4] 2> 1000127 T5954 n:127.0.0.1:47313_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 1000127 T5954 n:127.0.0.1:47313_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/
[junit4] 2> 1000128 T5954 n:127.0.0.1:47313_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 1000128 T6033 n:127.0.0.1:47313_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 1000129 T6033 n:127.0.0.1:47313_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1000129 T6033 n:127.0.0.1:47313_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1000129 T5992 n:127.0.0.1:52895_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 1000130 T5983 n:127.0.0.1:52895_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:47313",
[junit4] 2> "node_name":"127.0.0.1:47313_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 6
[junit4] 2> 1000130 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:47313",
[junit4] 2> "node_name":"127.0.0.1:47313_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 1000130 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 1000130 T5983 n:127.0.0.1:52895_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1000231 T6032 n:127.0.0.1:47313_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1000231 T6009 n:127.0.0.1:33270_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1000231 T5992 n:127.0.0.1:52895_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1001129 T6033 n:127.0.0.1:47313_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1001130 T6033 n:127.0.0.1:47313_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 1001130 T6033 n:127.0.0.1:47313_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1001130 T6033 n:127.0.0.1:47313_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 1001130 T6033 n:127.0.0.1:47313_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1001131 T6033 n:127.0.0.1:47313_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/'
[junit4] 2> 1001138 T6033 n:127.0.0.1:47313_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 1001140 T6033 n:127.0.0.1:47313_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 1001142 T6033 n:127.0.0.1:47313_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1001147 T6033 n:127.0.0.1:47313_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1001148 T6033 n:127.0.0.1:47313_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 1001151 T6033 n:127.0.0.1:47313_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1001211 T6033 n:127.0.0.1:47313_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1001212 T6033 n:127.0.0.1:47313_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1001213 T6033 n:127.0.0.1:47313_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1001214 T6033 n:127.0.0.1:47313_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1001220 T6033 n:127.0.0.1:47313_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 1001220 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1001221 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/], dataDir=[null]
[junit4] 2> 1001221 T6033 n:127.0.0.1:47313_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@859418
[junit4] 2> 1001221 T6033 n:127.0.0.1:47313_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/data
[junit4] 2> 1001222 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/data/index/
[junit4] 2> 1001222 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1001222 T6033 n:127.0.0.1:47313_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-003/cores/collection1/data/index
[junit4] 2> 1001222 T6033 n:127.0.0.1:47313_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=2000306261, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 1001223 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11e5e48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@94dbe1),segFN=segments_1,generation=1}
[junit4] 2> 1001223 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1001226 T6033 n:127.0.0.1:47313_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 1001226 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1001227 T6033 n:127.0.0.1:47313_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1001228 T6033 n:127.0.0.1:47313_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1001229 T6033 n:127.0.0.1:47313_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1001229 T6033 n:127.0.0.1:47313_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1001230 T6033 n:127.0.0.1:47313_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 1001232 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 1001232 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 1001233 T6033 n:127.0.0.1:47313_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 1001233 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 1001234 T6033 n:127.0.0.1:47313_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 1001234 T6033 n:127.0.0.1:47313_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1001235 T6033 n:127.0.0.1:47313_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1001235 T6033 n:127.0.0.1:47313_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1001235 T6033 n:127.0.0.1:47313_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=74.111328125, floorSegmentMB=1.8125, forceMergeDeletesPctAllowed=18.09961478163088, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21562580592977343
[junit4] 2> 1001236 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11e5e48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@94dbe1),segFN=segments_1,generation=1}
[junit4] 2> 1001236 T6033 n:127.0.0.1:47313_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1001236 T6033 n:127.0.0.1:47313_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@e3c792[collection1] main
[junit4] 2> 1001236 T6033 n:127.0.0.1:47313_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 1001237 T6033 n:127.0.0.1:47313_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 1001237 T6033 n:127.0.0.1:47313_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 1001237 T6033 n:127.0.0.1:47313_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501936421127061504
[junit4] 2> 1001239 T6033 n:127.0.0.1:47313_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with
[...truncated too long message...]
_shard1_replica1 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001/jetty-008/cores/target_collection_shard1_replica2/data/index
[junit4] 2> 1119357 T5954 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.Overseer.close Overseer (id=93871025939808303-127.0.0.1:50443_-n_0000000009) closing
[junit4] 2> 1119357 T6616 n:127.0.0.1:50443_ oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50443_
[junit4] 2> 1119358 T6415 n:127.0.0.1:50443_ oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1119359 T5954 oejs.AbstractConnector.doStop Stopped ServerConnector@3804a3{HTTP/1.1}{127.0.0.1:50443}
[junit4] 2> 1119360 T5954 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@16bd4e7{/,null,UNAVAILABLE}
[junit4] 2> 1119360 T5954 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.tearDown ###Ending doTest
[junit4] 2> 1119361 T5954 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49382 49382
[junit4] 2> 1119383 T5955 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49382 49382
[junit4] 2> 1119384 T5955 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 54 /solr/aliases.json
[junit4] 2> 54 /solr/clusterstate.json
[junit4] 2> 34 /solr/configs/conf1
[junit4] 2> 12 /solr/collections/source_collection/state.json
[junit4] 2> 12 /solr/collections/target_collection/state.json
[junit4] 2> 2 /solr/collections/target_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/target_collection/leaders/shard2
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard2
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard2
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/collections/source_collection/cdcr/state/process
[junit4] 2> 13 /solr/collections/target_collection/cdcr/state/buffer
[junit4] 2> 13 /solr/collections/source_collection/cdcr/state/buffer
[junit4] 2> 13 /solr/collections/target_collection/cdcr/state/process
[junit4] 2> 8 /solr/collections/tmp_collection/cdcr/state/process
[junit4] 2> 8 /solr/collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 3 /solr/overseer_elect/election/93871025939808277-127.0.0.1:45940_-n_0000000006
[junit4] 2> 2 /solr/overseer_elect/election/93871025939808259-127.0.0.1:52895_-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 52 /solr/live_nodes
[junit4] 2> 7 /solr/overseer/collection-queue-work
[junit4] 2> 6 /solr/overseer/queue
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CdcrReplicationDistributedZkTest -Dtests.method=doTest -Dtests.seed=EF101301FFD92986 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi -Dtests.timezone=Egypt -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 124s J0 | CdcrReplicationDistributedZkTest.doTest <<<
[junit4] > Throwable #1: org.junit.ComparisonFailure: expected:<st[art]ed> but was:<st[opp]ed>
[junit4] > at __randomizedtesting.SeedInfo.seed([EF101301FFD92986:4854ABA592623A3F]:0)
[junit4] > at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:255)
[junit4] > at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestTargetCollectionNotAvailable(CdcrReplicationDistributedZkTest.java:113)
[junit4] > at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:42)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1119398 T5954 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrReplicationDistributedZkTest EF101301FFD92986-001
[junit4] 2> 123975 T5953 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=PostingsFormat(name=LuceneFixedGap), id=Lucene50(blocksize=128)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=fi, timezone=Egypt
[junit4] 2> NOTE: Linux 3.13.0-53-generic i386/Oracle Corporation 1.8.0_60-ea (32-bit)/cpus=12,threads=1,free=110307128,total=358506496
[junit4] 2> NOTE: All tests run in this JVM: [MinimalSchemaTest, TestFieldSortValues, SolrIndexConfigTest, TestExpandComponent, TestReversedWildcardFilterFactory, TestFunctionQuery, UUIDUpdateProcessorFallbackTest, AssignTest, DistributedDebugComponentTest, TestCSVLoader, TestJsonRequest, DateRangeFieldTest, TestLazyCores, CdcrReplicationHandlerTest, DistributedSpellCheckComponentTest, RemoteQueryErrorTest, TestRandomDVFaceting, TestIndexingPerformance, TestSolrIndexConfig, TestBulkSchemaAPI, HighlighterTest, RuleEngineTest, TestManagedSynonymFilterFactory, TestReplicationHandler, SimpleMLTQParserTest, TermsComponentTest, TestSolrDeletionPolicy1, EnumFieldTest, DistributedQueryComponentCustomSortTest, BJQParserTest, IndexBasedSpellCheckerTest, TestSchemaResource, CurrencyFieldXmlFileTest, JavabinLoaderTest, BlockCacheTest, ReplicationFactorTest, TestJmxMonitoredMap, TermVectorComponentTest, HdfsDirectoryFactoryTest, ExternalFileFieldSortTest, TestSweetSpotSimilarityFactory, ResponseHeaderTest, ChaosMonkeyNothingIsSafeTest, TestManagedSchemaFieldTypeResource, DistributedFacetPivotLongTailTest, TestDocBasedVersionConstraints, TestRandomFaceting, TestRangeQuery, LeaderFailoverAfterPartitionTest, HdfsWriteToMultipleCollectionsTest, SchemaVersionSpecificBehaviorTest, DistributedFacetPivotSmallTest, TestCursorMarkWithoutUniqueKey, TestJmxIntegration, TestFieldResource, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestBadConfig, LeaderElectionIntegrationTest, TestDFRSimilarityFactory, TestTrackingShardHandlerFactory, TestSolr4Spatial, DateMathParserTest, CSVRequestHandlerTest, SyncSliceTest, TestBlendedInfixSuggestions, TestQuerySenderListener, TestCollectionAPI, URLClassifyProcessorTest, TestDistributedStatsComponentCardinality, CoreAdminCreateDiscoverTest, TestBinaryField, TestMergePolicyConfig, IndexSchemaTest, DistribDocExpirationUpdateProcessorTest, PreAnalyzedUpdateProcessorTest, MBeansHandlerTest, PeerSyncTest, TestHighFrequencyDictionaryFactory, HighlighterConfigTest, TestDynamicLoading, TestJettySolrRunner, TestUpdate, FullSolrCloudDistribCmdsTest, TestDistributedSearch, BasicFunctionalityTest, SolrIndexSplitterTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTest, SolrRequestParserTest, TestStressLucene, SolrCoreCheckLockOnStartupTest, DefaultValueUpdateProcessorTest, DocValuesMultiTest, DistanceFunctionTest, TestSurroundQueryParser, DirectSolrSpellCheckerTest, TestOmitPositions, MoreLikeThisHandlerTest, FastVectorHighlighterTest, TestCollationField, UpdateRequestProcessorFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestQuerySenderNoQuery, TestSolrDeletionPolicy2, TestElisionMultitermQuery, OutputWriterTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, ResourceLoaderTest, TestFastLRUCache, DateFieldTest, TestDocumentBuilder, SliceStateTest, UUIDFieldTest, CircularListTest, DistributedIntervalFacetingTest, CdcrReplicationDistributedZkTest]
[junit4] Completed [388/502] on J0 in 124.24s, 1 test, 1 failure <<< FAILURES!
[...truncated 367 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1986 tests, 1 failure, 57 ignored (25 assumptions)
Total time: 43 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b60) - Build #
12785 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12785/
Java: 64bit/jdk1.9.0-ea-b60 -XX:-UseCompressedOops -XX:+UseSerialGC
All tests passed
Build Log:
[...truncated 10763 lines...]
[junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/temp/junit4-J1-20150523_014702_234.syserr
[junit4] >>> JVM J1: stderr (verbatim) ----
[junit4] WARN: Unhandled exception in event serialization. -> java.lang.StackOverflowError
[junit4] at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:691)
[junit4] at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:578)
[junit4] at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:274)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
[junit4] at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
[junit4] at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
[junit4] at java.io.Writer.write(Writer.java:157)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.newline(JsonWriter.java:548)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.beforeValue(JsonWriter.java:589)
[junit4] at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:363)
[junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:86)
[junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
[junit4] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
[junit4] at java.io.PrintStream.flush(PrintStream.java:338)
[junit4] at java.io.FilterOutputStream.flush(FilterOutputStream.java:142)
[junit4] at java.io.PrintStream.write(PrintStream.java:482)
[junit4] at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:294)
[junit4] at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:298)
[junit4] at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
[junit4] at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
[junit4] at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59)
[junit4] at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324)
[junit4] at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
[junit4] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
[junit4] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
[junit4] at org.apache.log4j.Category.callAppenders(Category.java:206)
[junit4] at org.apache.log4j.Category.forcedLog(Category.java:391)
[junit4] at org.apache.log4j.Category.log(Category.java:856)
[junit4] at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:207)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:248)
[junit4] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
[junit4] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:348)
[junit4] at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:256)
[junit4] at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:492)
[junit4] at org.apache.solr.cloud.ShardLeaderE
[...truncated too long message...]
b/fastutil-6.5.11.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/guava-14.0.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hadoop-annotations-2.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hadoop-auth-2.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hadoop-common-2.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hadoop-hdfs-2.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hll-1.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/hppc-0.5.2.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/htrace-core-3.0.4.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/joda-time-2.2.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/log4j-1.2.17.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/org.restlet-2.3.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/protobuf-java-2.5.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/slf4j-log4j12-1.7.7.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/spatial4j-0.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/lib/t-digest-3.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/commons-io-2.4.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/httpclient-4.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/httpcore-4.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/httpmime-4.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/noggit-0.6.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/slf4j-api-1.7.7.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/stax2-api-3.1.4.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/lib/zookeeper-3.4.6.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/javax.servlet-api-3.1.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-continuation-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-deploy-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-http-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-io-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-jmx-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-rewrite-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-security-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-server-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-servlet-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-servlets-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-util-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-webapp-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/server/lib/jetty-xml-9.2.10.v20150310.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/core/classes/java:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/junit-4.10.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/test-framework/lib/randomizedtesting-runner-2.1.13.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/apacheds-all-2.0.0-M15.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/cglib-nodep-2.2.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/commons-collections-3.2.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/commons-math3-3.4.1.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/easymock-3.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/jcl-over-slf4j-1.7.7.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/test-lib/objenesis-1.2.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/analysis/icu/lucene-analyzers-icu-6.0.0-SNAPSHOT.jar:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/contrib/analysis-extras/lib/icu4j-54.1.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/var/lib/jenkins/.ant/lib/ivy-2.3.0.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/var/lib/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.13.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/temp/junit4-J1-20150523_014702_234.events @/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/temp/junit4-J1-20150523_014702_234.suites
[junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1504)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:964)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:961)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:265)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] at java.lang.Thread.run(Thread.java:745)
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
Total time: 130 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build #
12784 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12784/
Java: 64bit/jdk1.8.0_45 -XX:+UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.CdcrRequestHandlerTest.doTest
Error Message:
expected:<[dis]abled> but was:<[en]abled>
Stack Trace:
org.junit.ComparisonFailure: expected:<[dis]abled> but was:<[en]abled>
at __randomizedtesting.SeedInfo.seed([16BD265503763F07:B1F99EF16ECD2CBE]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:256)
at org.apache.solr.cloud.CdcrRequestHandlerTest.doTestBufferActions(CdcrRequestHandlerTest.java:139)
at org.apache.solr.cloud.CdcrRequestHandlerTest.doTest(CdcrRequestHandlerTest.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9898 lines...]
[junit4] Suite: org.apache.solr.cloud.CdcrRequestHandlerTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/init-core-data-001
[junit4] 2> 524627 T2904 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 524627 T2904 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 524628 T2904 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 524629 T2905 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 524629 T2905 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 524729 T2904 oasc.ZkTestServer.run start zk server on port:59405
[junit4] 2> 524729 T2904 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 524729 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 524731 T2912 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e30de37 name:ZooKeeperConnection Watcher:127.0.0.1:59405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 524731 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 524732 T2904 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 524732 T2904 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 524733 T2904 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 524734 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 524734 T2915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c3fbdb2 name:ZooKeeperConnection Watcher:127.0.0.1:59405/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 524734 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 524735 T2904 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 524735 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 524735 T2904 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 524736 T2904 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 524736 T2904 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 524737 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-cdcr.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 524737 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 524738 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 524738 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 524739 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 524740 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 524740 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 524740 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 524741 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 524741 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 524742 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 524742 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 524743 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 524743 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 524744 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 524744 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 524745 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 524745 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 524745 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 524746 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 524746 T2904 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 524746 T2904 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 524747 T2904 oas.SolrTestCaseJ4.setUp ###Starting doTest
[junit4] 2> 524796 T2904 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1
[junit4] 2> 524797 T2904 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 524798 T2904 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5a31014c{/,null,AVAILABLE}
[junit4] 2> 524800 T2904 oejs.AbstractConnector.doStart Started ServerConnector@7c225049{HTTP/1.1}{127.0.0.1:50630}
[junit4] 2> 524800 T2904 oejs.Server.doStart Started @525802ms
[junit4] 2> 524800 T2904 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard1, hostContext=/, hostPort=50630, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores}
[junit4] 2> 524801 T2904 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 524801 T2904 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/'
[junit4] 2> 524808 T2904 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/solr.xml
[junit4] 2> 524811 T2904 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores
[junit4] 2> 524812 T2904 oasc.CoreContainer.<init> New CoreContainer 1422739884
[junit4] 2> 524812 T2904 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/]
[junit4] 2> 524812 T2904 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/lib
[junit4] 2> 524812 T2904 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/lib).
[junit4] 2> 524816 T2904 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 524817 T2904 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 524817 T2904 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 524817 T2904 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 524817 T2904 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 524818 T2904 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59405/solr
[junit4] 2> 524818 T2904 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 524818 T2904 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 524818 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 524819 T2929 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d1a7dc4 name:ZooKeeperConnection Watcher:127.0.0.1:59405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 524820 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 524820 T2904 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 524821 T2904 n:127.0.0.1:50630_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 524821 T2932 n:127.0.0.1:50630_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3772aba name:ZooKeeperConnection Watcher:127.0.0.1:59405/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 524822 T2904 n:127.0.0.1:50630_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 524822 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 524823 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 524824 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 524824 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 524825 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 524826 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 524826 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 524827 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 524827 T2904 n:127.0.0.1:50630_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50630_
[junit4] 2> 524828 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50630_
[junit4] 2> 524828 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 524829 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 524829 T2904 n:127.0.0.1:50630_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 524829 T2904 n:127.0.0.1:50630_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50630_
[junit4] 2> 524830 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 524830 T2904 n:127.0.0.1:50630_ oasc.Overseer.start Overseer (id=93869833944891395-127.0.0.1:50630_-n_0000000000) starting
[junit4] 2> 524831 T2904 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 524833 T2904 n:127.0.0.1:50630_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 524834 T2904 n:127.0.0.1:50630_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 524834 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 524834 T2934 n:127.0.0.1:50630_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 524835 T2904 n:127.0.0.1:50630_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 524835 T2904 n:127.0.0.1:50630_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 524836 T2904 n:127.0.0.1:50630_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores
[junit4] 2> 524845 T2904 n:127.0.0.1:50630_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 524845 T2904 n:127.0.0.1:50630_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/
[junit4] 2> 524845 T2904 n:127.0.0.1:50630_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 524846 T2936 n:127.0.0.1:50630_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 524846 T2936 n:127.0.0.1:50630_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 524846 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 524846 T2936 n:127.0.0.1:50630_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 524847 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:50630",
[junit4] 2> "node_name":"127.0.0.1:50630_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 524847 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:50630",
[junit4] 2> "node_name":"127.0.0.1:50630_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 524847 T2933 n:127.0.0.1:50630_ oasco.ClusterStateMutator.createCollection building a new cName: tmp_collection
[junit4] 2> 524847 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 524848 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 525847 T2936 n:127.0.0.1:50630_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 525847 T2936 n:127.0.0.1:50630_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 525847 T2936 n:127.0.0.1:50630_ oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:tmp_collection
[junit4] 2> 525847 T2936 n:127.0.0.1:50630_ oasc.ZkController.getConfName Looking for collection configName
[junit4] 2> 525848 T2936 n:127.0.0.1:50630_ oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4] 2> 525848 T2936 n:127.0.0.1:50630_ oascc.SolrZkClient.makePath makePath: /collections/tmp_collection
[junit4] 2> 525848 T2936 n:127.0.0.1:50630_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 525849 T2936 n:127.0.0.1:50630_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 525849 T2936 n:127.0.0.1:50630_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/'
[junit4] 2> 525856 T2936 n:127.0.0.1:50630_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 525859 T2936 n:127.0.0.1:50630_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 525861 T2936 n:127.0.0.1:50630_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 525865 T2936 n:127.0.0.1:50630_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 525866 T2936 n:127.0.0.1:50630_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 525868 T2936 n:127.0.0.1:50630_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 525913 T2936 n:127.0.0.1:50630_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 525914 T2936 n:127.0.0.1:50630_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 525915 T2936 n:127.0.0.1:50630_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 525916 T2936 n:127.0.0.1:50630_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 525921 T2936 n:127.0.0.1:50630_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 525921 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 525921 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/], dataDir=[null]
[junit4] 2> 525921 T2936 n:127.0.0.1:50630_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@72d91d9e
[junit4] 2> 525922 T2936 n:127.0.0.1:50630_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/data
[junit4] 2> 525922 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/data/index/
[junit4] 2> 525922 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 525922 T2936 n:127.0.0.1:50630_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-001/cores/collection1/data/index
[junit4] 2> 525922 T2936 n:127.0.0.1:50630_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3154041884000785]
[junit4] 2> 525923 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@2d816cb4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ea5a721),segFN=segments_1,generation=1}
[junit4] 2> 525923 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 525925 T2936 n:127.0.0.1:50630_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 525933 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 525934 T2936 n:127.0.0.1:50630_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525934 T2936 n:127.0.0.1:50630_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525935 T2936 n:127.0.0.1:50630_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525935 T2936 n:127.0.0.1:50630_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525935 T2936 n:127.0.0.1:50630_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 525940 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 525940 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 525941 T2936 n:127.0.0.1:50630_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 525941 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 525948 T2936 n:127.0.0.1:50630_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 525948 T2936 n:127.0.0.1:50630_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20611698469899686]
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@2d816cb4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ea5a721),segFN=segments_1,generation=1}
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 525949 T2936 n:127.0.0.1:50630_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@2514ead3[collection1] main
[junit4] 2> 525950 T2936 n:127.0.0.1:50630_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 525950 T2936 n:127.0.0.1:50630_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 525950 T2936 n:127.0.0.1:50630_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 525950 T2936 n:127.0.0.1:50630_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501917344502382592
[junit4] 2> 525952 T2936 n:127.0.0.1:50630_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501917344502382592
[junit4] 2> 525952 T2936 n:127.0.0.1:50630_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 525952 T2936 n:127.0.0.1:50630_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 525952 T2936 n:127.0.0.1:50630_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 525952 T2936 n:127.0.0.1:50630_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 525953 T2936 n:127.0.0.1:50630_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 525953 T2936 n:127.0.0.1:50630_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 525953 T2936 n:127.0.0.1:50630_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 525953 T2936 n:127.0.0.1:50630_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 525953 T2936 n:127.0.0.1:50630_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 525954 T2936 n:127.0.0.1:50630_ x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/cdcr/state
[junit4] 2> 525955 T2936 n:127.0.0.1:50630_ x:collection1 oash.CdcrBufferStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 525956 T2936 n:127.0.0.1:50630_ x:collection1 oash.CdcrProcessStateManager.createStateNode Created znode /collections/tmp_collection/cdcr/state/process
[junit4] 2> 525959 T2936 n:127.0.0.1:50630_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 525960 T2936 n:127.0.0.1:50630_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 525960 T2937 n:127.0.0.1:50630_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2514ead3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 525960 T2936 n:127.0.0.1:50630_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 525961 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50630 collection:tmp_collection shard:shard2
[junit4] 2> 525961 T2904 n:127.0.0.1:50630_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2
[junit4] 2> 525961 T2904 n:127.0.0.1:50630_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 525961 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard2/election
[junit4] 2> 525962 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 525963 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 525963 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 525963 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C631 name=collection1 org.apache.solr.core.SolrCore@472c2de url=http://127.0.0.1:50630/collection1 node=127.0.0.1:50630_ C631_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:50630, node_name=127.0.0.1:50630_, state=down}
[junit4] 2> 525964 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 C631 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50630/collection1/
[junit4] 2> 525964 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection"} current state version: 1
[junit4] 2> 525964 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 C631 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 525964 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 C631 oasc.SyncStrategy.syncToMe http://127.0.0.1:50630/collection1/ has no replicas
[junit4] 2> 525964 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50630/collection1/ shard2
[junit4] 2> 525964 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard2
[junit4] 2> 525965 T2932 n:127.0.0.1:50630_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard2
[junit4] 2> 525966 T2942 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 525967 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:50630",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 526015 T2904 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1
[junit4] 2> 526016 T2904 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 526017 T2904 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4459a37e{/,null,AVAILABLE}
[junit4] 2> 526018 T2904 oejs.AbstractConnector.doStart Started ServerConnector@7f5a3d51{HTTP/1.1}{127.0.0.1:51997}
[junit4] 2> 526018 T2904 oejs.Server.doStart Started @527020ms
[junit4] 2> 526018 T2904 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard2, hostContext=/, hostPort=51997, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores}
[junit4] 2> 526018 T2904 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 526019 T2904 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/'
[junit4] 2> 526027 T2904 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/solr.xml
[junit4] 2> 526030 T2904 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores
[junit4] 2> 526030 T2904 oasc.CoreContainer.<init> New CoreContainer 1070840218
[junit4] 2> 526031 T2904 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/]
[junit4] 2> 526031 T2904 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/lib
[junit4] 2> 526031 T2904 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/lib).
[junit4] 2> 526036 T2904 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 526036 T2904 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 526037 T2904 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 526037 T2904 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 526037 T2904 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 526037 T2904 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59405/solr
[junit4] 2> 526037 T2904 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 526038 T2904 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 526038 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 526039 T2956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31c3aea0 name:ZooKeeperConnection Watcher:127.0.0.1:59405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 526040 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 526040 T2904 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 526041 T2904 n:127.0.0.1:51997_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 526041 T2959 n:127.0.0.1:51997_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a5cbc72 name:ZooKeeperConnection Watcher:127.0.0.1:59405/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 526042 T2904 n:127.0.0.1:51997_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 526044 T2904 n:127.0.0.1:51997_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 526069 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 526069 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 526117 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register We are http://127.0.0.1:50630/collection1/ and leader is http://127.0.0.1:50630/collection1/
[junit4] 2> 526118 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50630
[junit4] 2> 526118 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 526118 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 526118 T2941 n:127.0.0.1:50630_ c:tmp_collection s:shard2 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 526119 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 526119 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:50630",
[junit4] 2> "node_name":"127.0.0.1:50630_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 526119 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:50630",
[junit4] 2> "node_name":"127.0.0.1:50630_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 526221 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 526221 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 527045 T2904 n:127.0.0.1:51997_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51997_
[junit4] 2> 527046 T2904 n:127.0.0.1:51997_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51997_
[junit4] 2> 527047 T2904 n:127.0.0.1:51997_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 527048 T2904 n:127.0.0.1:51997_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 527048 T2904 n:127.0.0.1:51997_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 527049 T2904 n:127.0.0.1:51997_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores
[junit4] 2> 527049 T2904 n:127.0.0.1:51997_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 527049 T2904 n:127.0.0.1:51997_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/
[junit4] 2> 527050 T2904 n:127.0.0.1:51997_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 527050 T2960 n:127.0.0.1:51997_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 527050 T2960 n:127.0.0.1:51997_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 527051 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527051 T2960 n:127.0.0.1:51997_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 527051 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51997",
[junit4] 2> "node_name":"127.0.0.1:51997_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 527051 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51997",
[junit4] 2> "node_name":"127.0.0.1:51997_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 527051 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 527052 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 527156 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 527156 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 528051 T2960 n:127.0.0.1:51997_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 528051 T2960 n:127.0.0.1:51997_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 528052 T2960 n:127.0.0.1:51997_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 528052 T2960 n:127.0.0.1:51997_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 528052 T2960 n:127.0.0.1:51997_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 528052 T2960 n:127.0.0.1:51997_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/'
[junit4] 2> 528059 T2960 n:127.0.0.1:51997_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 528061 T2960 n:127.0.0.1:51997_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 528063 T2960 n:127.0.0.1:51997_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 528074 T2960 n:127.0.0.1:51997_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 528074 T2960 n:127.0.0.1:51997_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 528077 T2960 n:127.0.0.1:51997_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 528120 T2960 n:127.0.0.1:51997_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 528121 T2960 n:127.0.0.1:51997_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 528121 T2960 n:127.0.0.1:51997_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 528123 T2960 n:127.0.0.1:51997_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 528128 T2960 n:127.0.0.1:51997_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 528128 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/], dataDir=[null]
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@72d91d9e
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/data
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/data/index/
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 528129 T2960 n:127.0.0.1:51997_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-002/cores/collection1/data/index
[junit4] 2> 528130 T2960 n:127.0.0.1:51997_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3154041884000785]
[junit4] 2> 528130 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1760d72b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cbfaf60),segFN=segments_1,generation=1}
[junit4] 2> 528130 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 528133 T2960 n:127.0.0.1:51997_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 528133 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 528135 T2960 n:127.0.0.1:51997_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 528136 T2960 n:127.0.0.1:51997_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 528137 T2960 n:127.0.0.1:51997_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 528137 T2960 n:127.0.0.1:51997_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 528138 T2960 n:127.0.0.1:51997_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 528140 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 528140 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 528140 T2960 n:127.0.0.1:51997_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 528140 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 528141 T2960 n:127.0.0.1:51997_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 528141 T2960 n:127.0.0.1:51997_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 528141 T2960 n:127.0.0.1:51997_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 528142 T2960 n:127.0.0.1:51997_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 528142 T2960 n:127.0.0.1:51997_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20611698469899686]
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1760d72b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1cbfaf60),segFN=segments_1,generation=1}
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@6e7803a[collection1] main
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 528143 T2960 n:127.0.0.1:51997_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501917346801909760
[junit4] 2> 528145 T2960 n:127.0.0.1:51997_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501917346801909760
[junit4] 2> 528145 T2960 n:127.0.0.1:51997_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 528146 T2960 n:127.0.0.1:51997_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 528148 T2960 n:127.0.0.1:51997_ x:collection1 oasha.AdminHandlers.inform WARN <requestHandler name="/admin/"
[junit4] 2> class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
[junit4] 2> 528148 T2961 n:127.0.0.1:51997_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e7803a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 528149 T2960 n:127.0.0.1:51997_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 528149 T2960 n:127.0.0.1:51997_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 528149 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51997 collection:tmp_collection shard:shard1
[junit4] 2> 528149 T2904 n:127.0.0.1:51997_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2
[junit4] 2> 528150 T2904 n:127.0.0.1:51997_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 528150 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leader_elect/shard1/election
[junit4] 2> 528151 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 528152 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C632 name=collection1 org.apache.solr.core.SolrCore@59ac843d url=http://127.0.0.1:51997/collection1 node=127.0.0.1:51997_ C632_STATE=coll:tmp_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:51997, node_name=127.0.0.1:51997_, state=down}
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 C632 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51997/collection1/
[junit4] 2> 528152 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection"} current state version: 4
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 C632 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 C632 oasc.SyncStrategy.syncToMe http://127.0.0.1:51997/collection1/ has no replicas
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51997/collection1/ shard1
[junit4] 2> 528152 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oascc.SolrZkClient.makePath makePath: /collections/tmp_collection/leaders/shard1
[junit4] 2> 528153 T2959 n:127.0.0.1:51997_ oash.CdcrLeaderStateManager$LeaderStateWatcher.process Received new leader state @ tmp_collection:shard1
[junit4] 2> 528154 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 528154 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "base_url":"http://127.0.0.1:51997",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 528223 T2904 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1
[junit4] 2> 528224 T2904 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 528226 T2904 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4106eab5{/,null,AVAILABLE}
[junit4] 2> 528226 T2904 oejs.AbstractConnector.doStart Started ServerConnector@d517fbf{HTTP/1.1}{127.0.0.1:51316}
[junit4] 2> 528227 T2904 oejs.Server.doStart Started @529229ms
[junit4] 2> 528227 T2904 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {shards=shard3, hostContext=/, hostPort=51316, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores}
[junit4] 2> 528227 T2904 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 528228 T2904 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/'
[junit4] 2> 528241 T2904 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/solr.xml
[junit4] 2> 528245 T2904 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores
[junit4] 2> 528245 T2904 oasc.CoreContainer.<init> New CoreContainer 2730665
[junit4] 2> 528246 T2904 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/]
[junit4] 2> 528246 T2904 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/lib
[junit4] 2> 528246 T2904 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/lib).
[junit4] 2> 528252 T2904 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 528254 T2904 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 528254 T2904 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 528255 T2904 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 528255 T2904 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 528255 T2904 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59405/solr
[junit4] 2> 528256 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 528256 T2904 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 528256 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 528257 T2904 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 528258 T2904 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 528258 T2979 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ffe8d6f name:ZooKeeperConnection Watcher:127.0.0.1:59405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 528258 T2904 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 528259 T2904 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 528260 T2904 n:127.0.0.1:51316_ oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 528260 T2982 n:127.0.0.1:51316_ oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9e35353 name:ZooKeeperConnection Watcher:127.0.0.1:59405/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 528260 T2904 n:127.0.0.1:51316_ oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 528262 T2904 n:127.0.0.1:51316_ oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 528305 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:51997/collection1/ and leader is http://127.0.0.1:51997/collection1/
[junit4] 2> 528305 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51997
[junit4] 2> 528305 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 528305 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=tmp_collection
[junit4] 2> 528305 T2965 n:127.0.0.1:51997_ c:tmp_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 528306 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 528306 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51997",
[junit4] 2> "node_name":"127.0.0.1:51997_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 5
[junit4] 2> 528306 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51997",
[junit4] 2> "node_name":"127.0.0.1:51997_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 528408 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 528408 T2982 n:127.0.0.1:51316_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 528408 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 529263 T2904 n:127.0.0.1:51316_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51316_
[junit4] 2> 529264 T2904 n:127.0.0.1:51316_ oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51316_
[junit4] 2> 529265 T2904 n:127.0.0.1:51316_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 529266 T2904 n:127.0.0.1:51316_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 529266 T2904 n:127.0.0.1:51316_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 529266 T2904 n:127.0.0.1:51316_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores
[junit4] 2> 529267 T2904 n:127.0.0.1:51316_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1, collection=tmp_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 529267 T2904 n:127.0.0.1:51316_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/
[junit4] 2> 529268 T2904 n:127.0.0.1:51316_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 529268 T2983 n:127.0.0.1:51316_ c:tmp_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=tmp_collection
[junit4] 2> 529268 T2983 n:127.0.0.1:51316_ c:tmp_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 529269 T2932 n:127.0.0.1:50630_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 529269 T2983 n:127.0.0.1:51316_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 529269 T2933 n:127.0.0.1:50630_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51316",
[junit4] 2> "node_name":"127.0.0.1:51316_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"} current state version: 6
[junit4] 2> 529269 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Update state numShards=2 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:51316",
[junit4] 2> "node_name":"127.0.0.1:51316_",
[junit4] 2> "numShards":"2",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"tmp_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 529269 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Collection already exists with numShards=2
[junit4] 2> 529269 T2933 n:127.0.0.1:50630_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
[junit4] 2> 529370 T2982 n:127.0.0.1:51316_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 529370 T2959 n:127.0.0.1:51997_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 529370 T2932 n:127.0.0.1:50630_ oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 530269 T2983 n:127.0.0.1:51316_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 530269 T2983 n:127.0.0.1:51316_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:tmp_collection
[junit4] 2> 530269 T2983 n:127.0.0.1:51316_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 530269 T2983 n:127.0.0.1:51316_ oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 530270 T2983 n:127.0.0.1:51316_ oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 530270 T2983 n:127.0.0.1:51316_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/'
[junit4] 2> 530277 T2983 n:127.0.0.1:51316_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 530280 T2983 n:127.0.0.1:51316_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 530283 T2983 n:127.0.0.1:51316_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 530290 T2983 n:127.0.0.1:51316_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 530290 T2983 n:127.0.0.1:51316_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 530294 T2983 n:127.0.0.1:51316_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 530339 T2983 n:127.0.0.1:51316_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 530339 T2983 n:127.0.0.1:51316_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 530340 T2983 n:127.0.0.1:51316_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 530341 T2983 n:127.0.0.1:51316_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 530346 T2983 n:127.0.0.1:51316_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection tmp_collection
[junit4] 2> 530347 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 530347 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/], dataDir=[null]
[junit4] 2> 530347 T2983 n:127.0.0.1:51316_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@72d91d9e
[junit4] 2> 530347 T2983 n:127.0.0.1:51316_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/data
[junit4] 2> 530348 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/data/index/
[junit4] 2> 530348 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 530348 T2983 n:127.0.0.1:51316_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-003/cores/collection1/data/index
[junit4] 2> 530348 T2983 n:127.0.0.1:51316_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3154041884000785]
[junit4] 2> 530348 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@5346ba86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@669acdf0),segFN=segments_1,generation=1}
[junit4] 2> 530349 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 530351 T2983 n:127.0.0.1:51316_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "cdcr-processor-chain"
[junit4] 2> 530351 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 530352 T2983 n:127.0.0.1:51316_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 530352 T2983 n:127.0.0.1:51316_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 530353 T2983 n:127.0.0.1:51316_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 530353 T2983 n:127.0.0.1:51316_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 530354 T2983 n:127.0.0.1:51316_ x:collection1 oasc.PluginBag.createPlugin requestHandler : '/replication' created with startup=lazy
[junit4] 2> 530356 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 530356 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [org.apache.solr.handler.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
[junit4] 2> 530356 T2983 n:127.0.0.1:51316_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin,/admin/threads,/admin/ping,/cdcr,/update,/admin/file
[junit4] 2> 530356 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 530357 T2983 n:127.0.0.1:51316_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 530357 T2983 n:127.0.0.1:51316_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 530358 T2983 n:127.0.0.1:51316_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 530358 T2983 n:127.0.0.1:51316_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 530358 T2983 n:127.0.0.1:51316_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20611698469899686]
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@5346ba86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@669acdf0),segFN=segments_1,generation=1}
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@52a91a32[collection1] main
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 530359 T2983 n:127.0.0.1:51316_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501917349125554176
[junit4] 2> 530361 T2983 n:127.0.0.1:51316_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501917349125554176
[junit4] 2> 530361 T2983 n:127.0.0.1:51316_ x:collection1 oascc.ZkStateReader.readConfigName Load collection config from:/collections/tmp_collection
[junit4] 2> 530362 T2983 n:127.0.0.1:51316_ x:collection1 oascc.ZkStateReader.readConfigName path=/collections/tmp_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 530362 T2983 n:127.0.0.1:51316_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 530362 T2983 n:127.0.0.1:51316_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 530
[...truncated too long message...]
33 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 562633 T3034 n:127.0.0.1:42897_ oascc.ZkStateReader.updateWatchedCollection Updating data for source_collection to ver 16
[junit4] 2> 562633 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.SolrCore.closeSearcher [source_collection_shard1_replica2] Closing main searcher on request.
[junit4] 2> 562640 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 562640 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data/index [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data/index;done=false>>]
[junit4] 2> 562641 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data/index
[junit4] 2> 562641 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data;done=false>>]
[junit4] 2> 562641 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001/jetty-004/cores/source_collection_shard1_replica2/data
[junit4] 2> 562641 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.Overseer.close Overseer (id=93869833944891401-127.0.0.1:42897_-n_0000000003) closing
[junit4] 2> 562642 T3184 n:127.0.0.1:42897_ oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:42897_
[junit4] 2> 564143 T3034 n:127.0.0.1:42897_ oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 564147 T2904 oejs.AbstractConnector.doStop Stopped ServerConnector@7f45d825{HTTP/1.1}{127.0.0.1:0}
[junit4] 2> 564147 T2904 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@2c215962{/,null,UNAVAILABLE}
[junit4] 2> 564148 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.tearDown ###Ending doTest
[junit4] 2> 564149 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59405 59405
[junit4] 2> 564171 T2905 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59405 59405
[junit4] 2> 564172 T2905 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 19 /solr/aliases.json
[junit4] 2> 19 /solr/clusterstate.json
[junit4] 2> 9 /solr/configs/conf1
[junit4] 2> 5 /solr/collections/source_collection/state.json
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard2
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard2
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/collections/source_collection/cdcr/state/process
[junit4] 2> 6 /solr/collections/source_collection/cdcr/state/buffer
[junit4] 2> 4 /solr/collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 4 /solr/collections/tmp_collection/cdcr/state/process
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/live_nodes
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 4 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CdcrRequestHandlerTest -Dtests.method=doTest -Dtests.seed=16BD265503763F07 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi -Dtests.timezone=Asia/Dhaka -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 39.6s J2 | CdcrRequestHandlerTest.doTest <<<
[junit4] > Throwable #1: org.junit.ComparisonFailure: expected:<[dis]abled> but was:<[en]abled>
[junit4] > at __randomizedtesting.SeedInfo.seed([16BD265503763F07:B1F99EF16ECD2CBE]:0)
[junit4] > at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertState(BaseCdcrDistributedZkTest.java:256)
[junit4] > at org.apache.solr.cloud.CdcrRequestHandlerTest.doTestBufferActions(CdcrRequestHandlerTest.java:139)
[junit4] > at org.apache.solr.cloud.CdcrRequestHandlerTest.doTest(CdcrRequestHandlerTest.java:41)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 564181 T2904 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrRequestHandlerTest 16BD265503763F07-001
[junit4] 2> 39557 T2903 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=2, maxDocsPerChunk=680, blockSize=61), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=2, blockSize=61)), sim=DefaultSimilarity, locale=vi, timezone=Asia/Dhaka
[junit4] 2> NOTE: Linux 3.13.0-53-generic amd64/Oracle Corporation 1.8.0_45 (64-bit)/cpus=12,threads=1,free=134969976,total=518520832
[junit4] 2> NOTE: All tests run in this JVM: [DistributedFacetPivotWhiteBoxTest, MoreLikeThisHandlerTest, TestRestoreCore, TestDynamicFieldCollectionResource, BadIndexSchemaTest, HardAutoCommitTest, QueryEqualityTest, BasicDistributedZk2Test, TestSurroundQueryParser, TestFieldCollectionResource, TestSolrXml, TestBinaryResponseWriter, TestReqParamsAPI, TestImplicitCoreProperties, OpenExchangeRatesOrgProviderTest, TestObjectReleaseTracker, CursorPagingTest, SolrInfoMBeanTest, FacetPivotSmallTest, TestSchemaManager, TestPerFieldSimilarity, SearchHandlerTest, PluginInfoTest, RankQueryTest, TestNoOpRegenerator, SSLMigrationTest, TestInitParams, TestCSVResponseWriter, TestUtils, HdfsUnloadDistributedZkTest, TestExactSharedStatsCache, SortByFunctionTest, ActionThrottleTest, TestReload, TestSchemaSimilarityResource, TestCopyFieldCollectionResource, SimpleCollectionCreateDeleteTest, TestRestManager, TestIntervalFaceting, DistributedIntervalFacetingTest, SynonymTokenizerTest, TestPartialUpdateDeduplication, PolyFieldTest, TestShortCircuitedRequests, AutoCommitTest, TestDistributedMissingSort, ShardSplitTest, NumericFieldsTest, TestRequestStatusCollectionAPI, OutOfBoxZkACLAndCredentialsProvidersTest, StatsComponentTest, TestMiniSolrCloudCluster, OutputWriterTest, TestMissingGroups, TestSort, BufferStoreTest, ChangedSchemaMergeTest, AliasIntegrationTest, TestMacros, DirectUpdateHandlerTest, SolrIndexSplitterTest, CdcrRequestHandlerTest]
[junit4] Completed [215/502] on J2 in 40.57s, 1 test, 1 failure <<< FAILURES!
[...truncated 893 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1986 tests, 1 failure, 57 ignored (25 assumptions)
Total time: 42 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_60-ea-b12) -
Build # 12783 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12783/
Java: 64bit/jdk1.8.0_60-ea-b12 -XX:-UseCompressedOops -XX:+UseParallelGC
3 tests failed.
FAILED: org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest
Error Message:
There are still nodes recoverying - waited for 330 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
at __randomizedtesting.SeedInfo.seed([35642EE5937987FB:92209641FEC29442]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:172)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:133)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:128)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForRecoveriesToFinish(BaseCdcrDistributedZkTest.java:419)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.clearSourceCollection(BaseCdcrDistributedZkTest.java:285)
at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTestPartialReplicationAfterPeerSync(CdcrReplicationHandlerTest.java:159)
at org.apache.solr.cloud.CdcrReplicationHandlerTest.doTest(CdcrReplicationHandlerTest.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
Error Message:
Shard split did not complete. Last recorded state: running expected:<[completed]> but was:<[running]>
Stack Trace:
org.junit.ComparisonFailure: Shard split did not complete. Last recorded state: running expected:<[completed]> but was:<[running]>
at __randomizedtesting.SeedInfo.seed([35642EE5937987FB:6D00A28495132F2F]:0)
at org.junit.Assert.assertEquals(Assert.java:125)
at org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls(CollectionsAPIAsyncDistributedZkTest.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest
Error Message:
Number of tlogs on replica: 54 is superior to: 50.
Stack Trace:
java.lang.AssertionError: Number of tlogs on replica: 54 is superior to: 50.
at __randomizedtesting.SeedInfo.seed([35642EE5937987FB:92209641FEC29442]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertUpdateLogs(BaseCdcrDistributedZkTest.java:720)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestUpdateLogSynchronisation(CdcrReplicationDistributedZkTest.java:372)
at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9731 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIAsyncDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/init-core-data-001
[junit4] 2> 524069 T6041 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 524069 T6041 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 524071 T6041 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 524071 T6042 oasc.ZkTestServer$2$1.setClientPort client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 524071 T6042 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 524171 T6041 oasc.ZkTestServer.run start zk server on port:50889
[junit4] 2> 524177 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 524177 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 524178 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 524179 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 524179 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 524180 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 524180 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 524181 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 524181 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 524182 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 524182 T6041 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 524225 T6041 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1
[junit4] 2> 524226 T6041 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 524227 T6041 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@36e1ecb{/,null,AVAILABLE}
[junit4] 2> 524228 T6041 oejs.AbstractConnector.doStart Started ServerConnector@c1bc03c{HTTP/1.1}{127.0.0.1:55304}
[junit4] 2> 524228 T6041 oejs.Server.doStart Started @525083ms
[junit4] 2> 524228 T6041 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/, hostPort=55304, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores}
[junit4] 2> 524228 T6041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 524228 T6041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/'
[junit4] 2> 524235 T6041 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/solr.xml
[junit4] 2> 524238 T6041 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores
[junit4] 2> 524239 T6041 oasc.CoreContainer.<init> New CoreContainer 1045639281
[junit4] 2> 524239 T6041 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/]
[junit4] 2> 524239 T6041 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/lib
[junit4] 2> 524239 T6041 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/lib).
[junit4] 2> 524242 T6041 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 524243 T6041 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 524243 T6041 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 524244 T6041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 524244 T6041 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 524244 T6041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50889/solr
[junit4] 2> 524244 T6041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 524250 T6041 n:127.0.0.1:55304_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55304_
[junit4] 2> 524251 T6041 n:127.0.0.1:55304_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 524252 T6041 n:127.0.0.1:55304_ oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55304_
[junit4] 2> 524253 T6041 n:127.0.0.1:55304_ oasc.Overseer.start Overseer (id=93869437308108803-127.0.0.1:55304_-n_0000000000) starting
[junit4] 2> 524255 T6041 n:127.0.0.1:55304_ oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 524255 T6071 n:127.0.0.1:55304_ oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 524255 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 524256 T6041 n:127.0.0.1:55304_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 524257 T6041 n:127.0.0.1:55304_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 524257 T6041 n:127.0.0.1:55304_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores
[junit4] 2> 524258 T6041 n:127.0.0.1:55304_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 524258 T6041 n:127.0.0.1:55304_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/
[junit4] 2> 524258 T6041 n:127.0.0.1:55304_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 524259 T6073 n:127.0.0.1:55304_ c:control_collection x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 524259 T6073 n:127.0.0.1:55304_ c:control_collection x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 524259 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 524259 T6073 n:127.0.0.1:55304_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 524259 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 0
[junit4] 2> 524260 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 524260 T6070 n:127.0.0.1:55304_ oasco.ClusterStateMutator.createCollection building a new cName: control_collection
[junit4] 2> 524260 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 525259 T6073 n:127.0.0.1:55304_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 525260 T6073 n:127.0.0.1:55304_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 525260 T6073 n:127.0.0.1:55304_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 525260 T6073 n:127.0.0.1:55304_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/'
[junit4] 2> 525266 T6073 n:127.0.0.1:55304_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 525268 T6073 n:127.0.0.1:55304_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 525271 T6073 n:127.0.0.1:55304_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 525283 T6073 n:127.0.0.1:55304_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 525284 T6073 n:127.0.0.1:55304_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 525287 T6073 n:127.0.0.1:55304_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 525358 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 525362 T6073 n:127.0.0.1:55304_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 525363 T6073 n:127.0.0.1:55304_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 525367 T6073 n:127.0.0.1:55304_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 525368 T6073 n:127.0.0.1:55304_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 525369 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 525369 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 525370 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 525371 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/], dataDir=[null]
[junit4] 2> 525371 T6073 n:127.0.0.1:55304_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f22cf55
[junit4] 2> 525371 T6073 n:127.0.0.1:55304_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/data
[junit4] 2> 525371 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/data/index/
[junit4] 2> 525371 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 525372 T6073 n:127.0.0.1:55304_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/collection1/data/index
[junit4] 2> 525372 T6073 n:127.0.0.1:55304_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3401501224447425]
[junit4] 2> 525372 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3170f661 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@24d67963),segFN=segments_1,generation=1}
[junit4] 2> 525372 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 525374 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 525374 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 525375 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 525377 T6073 n:127.0.0.1:55304_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525377 T6073 n:127.0.0.1:55304_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525378 T6073 n:127.0.0.1:55304_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525378 T6073 n:127.0.0.1:55304_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 525380 T6073 n:127.0.0.1:55304_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
[junit4] 2> 525380 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 525381 T6073 n:127.0.0.1:55304_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 525381 T6073 n:127.0.0.1:55304_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 525381 T6073 n:127.0.0.1:55304_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=55.82421875, floorSegmentMB=0.431640625, forceMergeDeletesPctAllowed=19.65456556101773, segmentsPerTier=33.0, maxCFSSegmentSizeMB=1.021484375, noCFSRatio=0.0
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3170f661 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@24d67963),segFN=segments_1,generation=1}
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@4d5b0ab3[collection1] main
[junit4] 2> 525382 T6073 n:127.0.0.1:55304_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 525383 T6073 n:127.0.0.1:55304_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 525383 T6073 n:127.0.0.1:55304_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 525383 T6073 n:127.0.0.1:55304_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501910998304423936
[junit4] 2> 525384 T6073 n:127.0.0.1:55304_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 1 ms to seed version buckets with highest version 1501910998304423936
[junit4] 2> 525385 T6073 n:127.0.0.1:55304_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 525385 T6073 n:127.0.0.1:55304_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 525385 T6073 n:127.0.0.1:55304_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 525385 T6073 n:127.0.0.1:55304_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 525386 T6074 n:127.0.0.1:55304_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d5b0ab3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 525386 T6073 n:127.0.0.1:55304_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 525387 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55304 collection:control_collection shard:shard1
[junit4] 2> 525387 T6041 n:127.0.0.1:55304_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 525387 T6041 n:127.0.0.1:55304_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 525388 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 525389 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 525389 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 525390 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 525390 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection"} current state version: 1
[junit4] 2> ASYNC NEW_CORE C1162 name=collection1 org.apache.solr.core.SolrCore@7ededb8 url=http://127.0.0.1:55304/collection1 node=127.0.0.1:55304_ C1162_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:55304, node_name=127.0.0.1:55304_, state=down}
[junit4] 2> 525390 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 C1162 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55304/collection1/
[junit4] 2> 525390 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 C1162 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 525390 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 C1162 oasc.SyncStrategy.syncToMe http://127.0.0.1:55304/collection1/ has no replicas
[junit4] 2> 525390 T6041 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 525390 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55304/collection1/ shard1
[junit4] 2> 525391 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 525392 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 1
[junit4] 2> 525433 T6041 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1
[junit4] 2> 525433 T6041 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001
[junit4] 2> 525434 T6041 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2> 525435 T6041 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@dff7c04{/,null,AVAILABLE}
[junit4] 2> 525435 T6041 oejs.AbstractConnector.doStart Started ServerConnector@34a05c29{HTTP/1.1}{127.0.0.1:48678}
[junit4] 2> 525435 T6041 oejs.Server.doStart Started @526291ms
[junit4] 2> 525436 T6041 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solrconfig=solrconfig.xml, hostContext=/, hostPort=48678, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores}
[junit4] 2> 525436 T6041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2> 525436 T6041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/'
[junit4] 2> 525443 T6041 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/solr.xml
[junit4] 2> 525446 T6041 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores
[junit4] 2> 525446 T6041 oasc.CoreContainer.<init> New CoreContainer 441787764
[junit4] 2> 525446 T6041 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/]
[junit4] 2> 525446 T6041 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/lib
[junit4] 2> 525447 T6041 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/lib).
[junit4] 2> 525450 T6041 oashc.HttpShardHandlerFactory.init created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
[junit4] 2> 525451 T6041 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 525451 T6041 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 525451 T6041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 525451 T6041 oasc.CoreContainer.load Node Name: 127.0.0.1
[junit4] 2> 525451 T6041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50889/solr
[junit4] 2> 525452 T6041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 525542 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:55304/collection1/ and leader is http://127.0.0.1:55304/collection1/
[junit4] 2> 525542 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55304
[junit4] 2> 525542 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 525542 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 525542 T6077 n:127.0.0.1:55304_ c:control_collection s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 525543 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 525543 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"} current state version: 2
[junit4] 2> 525543 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 526456 T6041 n:127.0.0.1:48678_ oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48678_
[junit4] 2> 526457 T6041 n:127.0.0.1:48678_ oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 526458 T6041 n:127.0.0.1:48678_ oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
[junit4] 2> 526458 T6041 n:127.0.0.1:48678_ oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. Skipping setup for authorization module.
[junit4] 2> 526459 T6041 n:127.0.0.1:48678_ oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores
[junit4] 2> 526459 T6041 n:127.0.0.1:48678_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
[junit4] 2> 526459 T6041 n:127.0.0.1:48678_ oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/
[junit4] 2> 526460 T6041 n:127.0.0.1:48678_ oasc.CorePropertiesLocator.discover Found 1 core definitions
[junit4] 2> 526460 T6098 n:127.0.0.1:48678_ c:collection1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 526460 T6098 n:127.0.0.1:48678_ c:collection1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 526461 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 526461 T6098 n:127.0.0.1:48678_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 526461 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:48678",
[junit4] 2> "node_name":"127.0.0.1:48678_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 3
[junit4] 2> 526461 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:48678",
[junit4] 2> "node_name":"127.0.0.1:48678_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 526461 T6070 n:127.0.0.1:55304_ oasco.ClusterStateMutator.createCollection building a new cName: collection1
[junit4] 2> 526461 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
[junit4] 2> 527461 T6098 n:127.0.0.1:48678_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 527461 T6098 n:127.0.0.1:48678_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 527462 T6098 n:127.0.0.1:48678_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 527462 T6098 n:127.0.0.1:48678_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/'
[junit4] 2> 527467 T6098 n:127.0.0.1:48678_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 527470 T6098 n:127.0.0.1:48678_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 527473 T6098 n:127.0.0.1:48678_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 527479 T6098 n:127.0.0.1:48678_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 527479 T6098 n:127.0.0.1:48678_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 527482 T6098 n:127.0.0.1:48678_ oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 527555 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 527559 T6098 n:127.0.0.1:48678_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 527559 T6098 n:127.0.0.1:48678_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 527563 T6098 n:127.0.0.1:48678_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 527564 T6098 n:127.0.0.1:48678_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 527565 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 527565 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 527565 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 527565 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 527565 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 527566 T6098 n:127.0.0.1:48678_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 527566 T6098 n:127.0.0.1:48678_ oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 527566 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 527566 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/], dataDir=[null]
[junit4] 2> 527566 T6098 n:127.0.0.1:48678_ x:collection1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f22cf55
[junit4] 2> 527567 T6098 n:127.0.0.1:48678_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/data
[junit4] 2> 527567 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/data/index/
[junit4] 2> 527567 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 527567 T6098 n:127.0.0.1:48678_ x:collection1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/shard-1-001/cores/collection1/data/index
[junit4] 2> 527567 T6098 n:127.0.0.1:48678_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3401501224447425]
[junit4] 2> 527568 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@55925056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@279bc771),segFN=segments_1,generation=1}
[junit4] 2> 527568 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 527570 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 527571 T6098 n:127.0.0.1:48678_ x:collection1 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 527571 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 527572 T6098 n:127.0.0.1:48678_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 527572 T6098 n:127.0.0.1:48678_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 527573 T6098 n:127.0.0.1:48678_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 527573 T6098 n:127.0.0.1:48678_ x:collection1 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 527576 T6098 n:127.0.0.1:48678_ x:collection1 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
[junit4] 2> 527576 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
[junit4] 2> 527577 T6098 n:127.0.0.1:48678_ x:collection1 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 527577 T6098 n:127.0.0.1:48678_ x:collection1 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 527577 T6098 n:127.0.0.1:48678_ x:collection1 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 527577 T6098 n:127.0.0.1:48678_ x:collection1 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 527578 T6098 n:127.0.0.1:48678_ x:collection1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=55.82421875, floorSegmentMB=0.431640625, forceMergeDeletesPctAllowed=19.65456556101773, segmentsPerTier=33.0, maxCFSSegmentSizeMB=1.021484375, noCFSRatio=0.0
[junit4] 2> 527578 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@55925056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@279bc771),segFN=segments_1,generation=1}
[junit4] 2> 527578 T6098 n:127.0.0.1:48678_ x:collection1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 527579 T6098 n:127.0.0.1:48678_ x:collection1 oass.SolrIndexSearcher.<init> Opening Searcher@37839bc9[collection1] main
[junit4] 2> 527579 T6098 n:127.0.0.1:48678_ x:collection1 oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value of version field
[junit4] 2> 527579 T6098 n:127.0.0.1:48678_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ for 65536 version buckets from index
[junit4] 2> 527579 T6098 n:127.0.0.1:48678_ x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index
[junit4] 2> 527579 T6098 n:127.0.0.1:48678_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501911000607096832
[junit4] 2> 527581 T6098 n:127.0.0.1:48678_ x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets with highest version 1501911000607096832
[junit4] 2> 527581 T6098 n:127.0.0.1:48678_ x:collection1 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 527581 T6098 n:127.0.0.1:48678_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 527581 T6098 n:127.0.0.1:48678_ x:collection1 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 527581 T6098 n:127.0.0.1:48678_ x:collection1 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 527582 T6098 n:127.0.0.1:48678_ x:collection1 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 527582 T6098 n:127.0.0.1:48678_ x:collection1 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 527582 T6098 n:127.0.0.1:48678_ x:collection1 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 527582 T6098 n:127.0.0.1:48678_ x:collection1 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 527582 T6099 n:127.0.0.1:48678_ x:collection1 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37839bc9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 527582 T6098 n:127.0.0.1:48678_ x:collection1 oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
[junit4] 2> 527583 T6098 n:127.0.0.1:48678_ x:collection1 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 527583 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48678 collection:collection1 shard:shard1
[junit4] 2> 527583 T6041 n:127.0.0.1:48678_ oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 527583 T6041 n:127.0.0.1:48678_ oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 527585 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 527585 T6041 oas.SolrTestCaseJ4.setUp ###Starting testSolrJAPICalls
[junit4] 2> 527585 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527585 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 527585 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 527586 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1"} current state version: 4
[junit4] 2> ASYNC NEW_CORE C1163 name=collection1 org.apache.solr.core.SolrCore@2d14096c url=http://127.0.0.1:48678/collection1 node=127.0.0.1:48678_ C1163_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:48678, node_name=127.0.0.1:48678_, state=down}
[junit4] 2> 527586 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 C1163 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48678/collection1/
[junit4] 2> 527586 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 C1163 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 527586 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 C1163 oasc.SyncStrategy.syncToMe http://127.0.0.1:48678/collection1/ has no replicas
[junit4] 2> 527586 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48678/collection1/ shard1
[junit4] 2> 527587 T6087 n:127.0.0.1:48678_ oasha.CollectionsHandler.handleRequestBody Invoked Collection Action :create with paramsasync=1001&collection.configName=conf1&name=testasynccollectioncreation&action=CREATE&numShards=1&wt=javabin&version=2
[junit4] 2> 527587 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527587 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
[junit4] 2> 527588 T6087 n:127.0.0.1:48678_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/collections params={async=1001&collection.configName=conf1&name=testasynccollectioncreation&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 527588 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:48678",
[junit4] 2> "core":"collection1",
[junit4] 2> "state":"active"} current state version: 4
[junit4] 2> 527588 T6071 n:127.0.0.1:55304_ oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4] 2> "name":"testasynccollectioncreation",
[junit4] 2> "fromApi":"true",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"1",
[junit4] 2> "async":"1001",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"}
[junit4] 2> 527588 T6104 n:127.0.0.1:55304_ c:testasynccollectioncreation oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
[junit4] 2> "name":"testasynccollectioncreation",
[junit4] 2> "fromApi":"true",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"1",
[junit4] 2> "async":"1001",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"}
[junit4] 2> 527589 T6104 n:127.0.0.1:55304_ c:testasynccollectioncreation oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testasynccollectioncreation
[junit4] 2> 527590 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527590 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "name":"testasynccollectioncreation",
[junit4] 2> "fromApi":"true",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"1",
[junit4] 2> "async":"1001",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"} current state version: 4
[junit4] 2> 527590 T6088 n:127.0.0.1:48678_ oasha.CollectionsHandler.handleRequestBody Invoked Collection Action :requeststatus with paramsrequestid=1001&action=REQUESTSTATUS&wt=javabin&version=2
[junit4] 2> 527590 T6070 n:127.0.0.1:55304_ oasco.ClusterStateMutator.createCollection building a new cName: testasynccollectioncreation
[junit4] 2> 527591 T6088 n:127.0.0.1:48678_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/collections params={requestid=1001&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 527638 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.register We are http://127.0.0.1:48678/collection1/ and leader is http://127.0.0.1:48678/collection1/
[junit4] 2> 527638 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48678
[junit4] 2> 527638 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 527638 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 527638 T6102 n:127.0.0.1:48678_ c:collection1 s:shard1 x:collection1 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 527639 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527639 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:48678",
[junit4] 2> "node_name":"127.0.0.1:48678_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"} current state version: 5
[junit4] 2> 527639 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:48678",
[junit4] 2> "node_name":"127.0.0.1:48678_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 527639 T6070 n:127.0.0.1:55304_ oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/testasynccollectioncreation/state.json
[junit4] 2> 527690 T6104 n:127.0.0.1:55304_ c:testasynccollectioncreation oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection testasynccollectioncreation, shardNames [shard1] , replicationFactor : 1
[junit4] 2> 527690 T6104 n:127.0.0.1:55304_ c:testasynccollectioncreation oasc.OverseerCollectionProcessor.createCollection Creating core testasynccollectioncreation_shard1_replica1 as part of shard shard1 of collection testasynccollectioncreation on 127.0.0.1:55304_
[junit4] 2> 527693 T6059 n:127.0.0.1:55304_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/cores params={async=1001182016192633248&qt=/admin/cores&collection.configName=conf1&name=testasynccollectioncreation_shard1_replica1&action=CREATE&numShards=1&collection=testasynccollectioncreation&shard=shard1&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 527693 T6107 n:127.0.0.1:55304_ oasha.CoreAdminHandler.handleCreateAction core create command async=1001182016192633248&qt=/admin/cores&collection.configName=conf1&name=testasynccollectioncreation_shard1_replica1&action=CREATE&numShards=1&collection=testasynccollectioncreation&shard=shard1&wt=javabin&version=2
[junit4] 2> 527693 T6107 n:127.0.0.1:55304_ oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=testasynccollectioncreation_shard1_replica1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=testasynccollectioncreation_shard1_replica1, collection=testasynccollectioncreation, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/, numShards=1, dataDir=data/, shard=shard1}
[junit4] 2> 527694 T6107 n:127.0.0.1:55304_ c:testasynccollectioncreation x:testasynccollectioncreation_shard1_replica1 oasc.ZkController.publish publishing core=testasynccollectioncreation_shard1_replica1 state=down collection=testasynccollectioncreation
[junit4] 2> 527694 T6060 n:127.0.0.1:55304_ oasha.CoreAdminHandler.handleRequestActionStatus Checking request status for : 1001182016192633248
[junit4] 2> 527695 T6069 n:127.0.0.1:55304_ oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
[junit4] 2> 527695 T6107 n:127.0.0.1:55304_ oasc.ZkController.preRegister Registering watch for external collection testasynccollectioncreation
[junit4] 2> 527695 T6060 n:127.0.0.1:55304_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=1001182016192633248&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 527695 T6070 n:127.0.0.1:55304_ oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
[junit4] 2> "core":"testasynccollectioncreation_shard1_replica1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testasynccollectioncreation",
[junit4] 2> "operation":"state"} current state version: 6
[junit4] 2> 527695 T6107 n:127.0.0.1:55304_ oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 527695 T6070 n:127.0.0.1:55304_ oasco.ReplicaMutator.updateState Update state numShards=1 message={
[junit4] 2> "core":"testasynccollectioncreation_shard1_replica1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:55304",
[junit4] 2> "node_name":"127.0.0.1:55304_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"testasynccollectioncreation",
[junit4] 2> "operation":"state"}
[junit4] 2> 527797 T6070 n:127.0.0.1:55304_ oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/testasynccollectioncreation/state.json version: 0
[junit4] 2> 528593 T6089 n:127.0.0.1:48678_ oasha.CollectionsHandler.handleRequestBody Invoked Collection Action :requeststatus with paramsrequestid=1001&action=REQUESTSTATUS&wt=javabin&version=2
[junit4] 2> 528593 T6089 n:127.0.0.1:48678_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/collections params={requestid=1001&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 528695 T6107 n:127.0.0.1:55304_ oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testasynccollectioncreation_shard1_replica1
[junit4] 2> 528696 T6107 n:127.0.0.1:55304_ oasc.ZkController.createCollectionZkNode Check for collection zkNode:testasynccollectioncreation
[junit4] 2> 528696 T6107 n:127.0.0.1:55304_ oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 528696 T6107 n:127.0.0.1:55304_ oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/'
[junit4] 2> 528697 T6061 n:127.0.0.1:55304_ oasha.CoreAdminHandler.handleRequestActionStatus Checking request status for : 1001182016192633248
[junit4] 2> 528697 T6061 n:127.0.0.1:55304_ oass.HttpSolrCall.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=1001182016192633248&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 528702 T6107 n:127.0.0.1:55304_ oasc.Config.<init> loaded config solrconfig.xml with version 0
[junit4] 2> 528704 T6107 n:127.0.0.1:55304_ oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
[junit4] 2> 528707 T6107 n:127.0.0.1:55304_ oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 528714 T6107 n:127.0.0.1:55304_ oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 528714 T6107 n:127.0.0.1:55304_ oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
[junit4] 2> 528717 T6107 n:127.0.0.1:55304_ oass.IndexSchema.readSchema [testasynccollectioncreation_shard1_replica1] Schema name=test
[junit4] 2> 528794 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 528797 T6107 n:127.0.0.1:55304_ oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 528798 T6107 n:127.0.0.1:55304_ oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 528801 T6107 n:127.0.0.1:55304_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 528802 T6107 n:127.0.0.1:55304_ oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 528804 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 528805 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 528805 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 528805 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 528805 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 528805 T6107 n:127.0.0.1:55304_ oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 528806 T6107 n:127.0.0.1:55304_ oasc.CoreContainer.create Creating SolrCore 'testasynccollectioncreation_shard1_replica1' using configuration from collection testasynccollectioncreation
[junit4] 2> 528806 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 528806 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrCore.<init> [[testasynccollectioncreation_shard1_replica1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/], dataDir=[null]
[junit4] 2> 528806 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f22cf55
[junit4] 2> 528807 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/data
[junit4] 2> 528807 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/data/index/
[junit4] 2> 528807 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrCore.initIndex WARN [testasynccollectioncreation_shard1_replica1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4] 2> 528807 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIAsyncDistributedZkTest 35642EE5937987FB-001/control-001/cores/testasynccollectioncreation_shard1_replica1/data/index
[junit4] 2> 528807 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3401501224447425]
[junit4] 2> 528808 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@a1e6d17 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62baf57a),segFN=segments_1,generation=1}
[junit4] 2> 528808 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 528810 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 528810 T6107 n:127.0.0.1:55304_ x:testasynccollectioncreation_shard1_replica1 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorC
[...truncated too long message...]
ollection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest 35642EE5937987FB-001/jetty-008/cores/target_collection_shard2_replica2/data/index
[junit4] 2> 901622 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest 35642EE5937987FB-001/jetty-008/cores/target_collection_shard2_replica2/data [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest 35642EE5937987FB-001/jetty-008/cores/target_collection_shard2_replica2/data;done=false>>]
[junit4] 2> 901623 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest 35642EE5937987FB-001/jetty-008/cores/target_collection_shard2_replica2/data
[junit4] 2> 901623 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.Overseer.close Overseer (id=93869442226913423-127.0.0.1:49909_-n_0000000012) closing
[junit4] 2> 901624 T5299 n:127.0.0.1:49909_ oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49909_
[junit4] 2> 903125 T4650 n:127.0.0.1:49909_ oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 903126 T3378 oejs.AbstractConnector.doStop Stopped ServerConnector@7882e5b5{HTTP/1.1}{127.0.0.1:49909}
[junit4] 2> 903126 T3378 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@342ce6f2{/,null,UNAVAILABLE}
[junit4] 2> 903127 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.tearDown ###Ending doTest
[junit4] 2> 903128 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57946 57946
[junit4] 2> 903306 T3379 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:57946 57946
[junit4] 2> 903307 T3379 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 257 /solr/aliases.json
[junit4] 2> 257 /solr/clusterstate.json
[junit4] 2> 72 /solr/configs/conf1
[junit4] 2> 31 /solr/collections/source_collection/state.json
[junit4] 2> 30 /solr/collections/target_collection/state.json
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/source_collection/leaders/shard2
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/tmp_collection/leaders/shard2
[junit4] 2> 2 /solr/collections/target_collection/leaders/shard1
[junit4] 2> 2 /solr/collections/target_collection/leaders/shard2
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 41 /solr/collections/source_collection/cdcr/state/process
[junit4] 2> 34 /solr/collections/source_collection/cdcr/state/buffer
[junit4] 2> 31 /solr/collections/target_collection/cdcr/state/buffer
[junit4] 2> 31 /solr/collections/target_collection/cdcr/state/process
[junit4] 2> 8 /solr/collections/tmp_collection/cdcr/state/buffer
[junit4] 2> 8 /solr/collections/tmp_collection/cdcr/state/process
[junit4] 2> 3 /solr/overseer_elect/election/93869442226913299-127.0.0.1:45503_-n_0000000005
[junit4] 2> 3 /solr/overseer_elect/election/93869442226913283-127.0.0.1:58843_-n_0000000000
[junit4] 2> 2 /solr/collections/source_collection/leader_elect/shard2/election/93869442226913289-core_node1-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/93869442226913289-127.0.0.1:60101_-n_0000000003
[junit4] 2> 2 /solr/overseer_elect/election/93869442226913303-127.0.0.1:49909_-n_0000000007
[junit4] 2> 2 /solr/overseer_elect/election/93869442226913395-127.0.0.1:43614_-n_0000000010
[junit4] 2> 2 /solr/collections/target_collection/leader_elect/shard2/election/93869442226913299-core_node1-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/93869442226913327-127.0.0.1:57568_-n_0000000009
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 249 /solr/live_nodes
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 4 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CdcrReplicationDistributedZkTest -Dtests.method=doTest -Dtests.seed=35642EE5937987FB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_EG -Dtests.timezone=Asia/Pontianak -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 304s J2 | CdcrReplicationDistributedZkTest.doTest <<<
[junit4] > Throwable #1: java.lang.AssertionError: Number of tlogs on replica: 54 is superior to: 50.
[junit4] > at __randomizedtesting.SeedInfo.seed([35642EE5937987FB:92209641FEC29442]:0)
[junit4] > at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertUpdateLogs(BaseCdcrDistributedZkTest.java:720)
[junit4] > at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTestUpdateLogSynchronisation(CdcrReplicationDistributedZkTest.java:372)
[junit4] > at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.doTest(CdcrReplicationDistributedZkTest.java:46)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 903316 T3378 c:source_collection s:shard1 x:source_collection_shard1_replica1 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.CdcrReplicationDistributedZkTest 35642EE5937987FB-001
[junit4] 2> 304179 T3377 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=Direct)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_EG, timezone=Asia/Pontianak
[junit4] 2> NOTE: Linux 3.13.0-53-generic amd64/Oracle Corporation 1.8.0_60-ea (64-bit)/cpus=12,threads=1,free=282632056,total=528482304
[junit4] 2> NOTE: All tests run in this JVM: [TestFieldCollectionResource, HardAutoCommitTest, ShardSplitTest, CircularListTest, TestSolrXml, TestNoOpRegenerator, TestShortCircuitedRequests, TestClusterStateMutator, AliasIntegrationTest, TestLuceneMatchVersion, CachingDirectoryFactoryTest, SSLMigrationTest, OverriddenZkACLAndCredentialsProvidersTest, DOMUtilTest, DistanceUnitsTest, TestFieldTypeResource, DirectUpdateHandlerOptimizeTest, TestStressRecovery, XsltUpdateRequestHandlerTest, CursorPagingTest, TestInitParams, ChangedSchemaMergeTest, HdfsUnloadDistributedZkTest, LeaderElectionTest, TestMissingGroups, TestNRTOpen, PolyFieldTest, HdfsChaosMonkeySafeLeaderTest, CdcrRequestHandlerTest, TestBinaryResponseWriter, ActionThrottleTest, TestXIncludeConfig, MoreLikeThisHandlerTest, TestObjectReleaseTracker, TestCloudInspectUtil, SimpleCollectionCreateDeleteTest, StressHdfsTest, TermVectorComponentDistributedTest, MergeStrategyTest, OutputWriterTest, TestSolrQueryParser, TestSolrJ, TestUtils, TestDistributedMissingSort, TestHashQParserPlugin, DistributedIntervalFacetingTest, TestStressReorder, TestCSVResponseWriter, TestSurroundQueryParser, TestSort, TestCopyFieldCollectionResource, AsyncMigrateRouteKeyTest, TestCollationField, LoggingHandlerTest, TestRecovery, MultiThreadedOCPTest, TestFastOutputStream, SOLR749Test, TestImplicitCoreProperties, DeleteShardTest, TestIntervalFaceting, CdcrReplicationDistributedZkTest]
[junit4] Completed [175/502] on J2 in 304.69s, 1 test, 1 failure <<< FAILURES!
[...truncated 1015 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:526: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:229: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:512: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1415: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:973: There were test failures: 502 suites, 1985 tests, 3 failures, 57 ignored (25 assumptions)
Total time: 44 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any