You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/08 04:06:49 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20-ea-b11) -
Build # 10257 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10257/
Java: 64bit/jdk1.8.0_20-ea-b11 -XX:-UseCompressedOops -XX:+UseParallelGC
All tests passed
Build Log:
[...truncated 20655 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 59 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_20-ea-b11 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk7) - Build #
10261 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10261/
Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
All tests passed
Build Log:
[...truncated 20412 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 53 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_20-ea-b11) -
Build # 10260 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10260/
Java: 32bit/jdk1.8.0_20-ea-b11 -client -XX:+UseG1GC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest
Error Message:
ERROR: SolrZkClient opens=21 closes=20
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=21 closes=20
at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 1) Thread[id=3041, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) 2) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:
1) Thread[id=3041, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
2) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)
REGRESSION: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
Error Message:
some core start times did not change on reload
Stack Trace:
java.lang.AssertionError: some core start times did not change on reload
at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354:ECEC003DA98BD368]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:836)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:203)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11314 lines...]
[junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/init-core-data-001
[junit4] 2> 703456 T2890 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 703456 T2890 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 703462 T2890 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 703463 T2890 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 703463 T2891 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 703563 T2890 oasc.ZkTestServer.run start zk server on port:46497
[junit4] 2> 703565 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 713803 T2894 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 10235ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 713806 T2892 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db8e1b220000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 715564 T2897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fa4265 name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715564 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 715565 T2890 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 715574 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 715576 T2899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cfdd10 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715576 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 715577 T2890 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 715583 T2890 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 715588 T2890 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 715592 T2890 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 715597 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 715598 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 715605 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 715606 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 715612 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 715613 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 715617 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 715619 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 715624 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 715625 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 715631 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 715632 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 715636 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 715637 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 715641 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 715642 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 715646 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 715647 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 715652 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 715652 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 715655 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 715656 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 715659 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 715660 T2901 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16649b6 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715660 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> Using old style solr.xml
[junit4] 2> 715664 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 715665 T2903 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13716b1 name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715665 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 715665 T2890 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 715669 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 715671 T2905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a48118 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715671 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 715672 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf2/solrconfig.xml
[junit4] 2> 715672 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
[junit4] 2> 715676 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
[junit4] 2> 715677 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
[junit4] 2> 715679 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 715680 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 715682 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
[junit4] 2> 715683 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
[junit4] 2> 715684 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
[junit4] 2> 715685 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
[junit4] 2> 715688 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
[junit4] 2> 715689 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
[junit4] 2> 715691 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
[junit4] 2> 715691 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
[junit4] 2> 715693 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 715693 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
[junit4] 2> 715695 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
[junit4] 2> 715696 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
[junit4] 2> 715698 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
[junit4] 2> 715698 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
[junit4] 2> 715700 T2890 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
[junit4] 2> 715700 T2890 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
[junit4] 2> 716011 T2890 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 716021 T2890 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 716030 T2890 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38483
[junit4] 2> 716036 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 716037 T2890 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 716038 T2890 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002
[junit4] 2> 716039 T2890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/'
[junit4] 2> 716072 T2890 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solr.xml
[junit4] 2> 716086 T2890 oasc.CoreContainer.<init> New CoreContainer 7359148
[junit4] 2> 716087 T2890 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/]
[junit4] 2> 716087 T2890 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 716088 T2890 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 716088 T2890 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 716088 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 716089 T2890 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 716089 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 716089 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 716090 T2890 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 716090 T2890 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 716093 T2890 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 716094 T2890 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 716094 T2890 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 716095 T2890 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46497/solr
[junit4] 2> 716095 T2890 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 716096 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 716098 T2916 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1933a3f name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 716099 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 716105 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 716107 T2918 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52451b name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 716107 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 716110 T2890 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 716115 T2890 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 716119 T2890 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 716124 T2890 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 716129 T2890 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 716137 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 716140 T2890 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38483_
[junit4] 2> 716142 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38483_
[junit4] 2> 716147 T2890 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 716150 T2890 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 716155 T2890 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38483_
[junit4] 2> 716156 T2890 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 716157 T2890 oasc.Overseer.start Overseer (id=91720770818015239-127.0.0.1:38483_-n_0000000000) starting
[junit4] 2> 716162 T2890 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 716181 T2920 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 716182 T2890 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 716183 T2920 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:38483_
[junit4] 2> 716187 T2890 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 716196 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 716201 T2919 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 716205 T2890 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 716206 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 716219 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 716222 T2923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8b35f7 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 716224 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 716227 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 716233 T2890 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 716528 T2890 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 716529 T2890 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 716536 T2890 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 716540 T2890 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:41275
[junit4] 2> 716544 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 716546 T2890 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 716547 T2890 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003
[junit4] 2> 716547 T2890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/'
[junit4] 2> 716567 T2890 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solr.xml
[junit4] 2> 716598 T2890 oasc.CoreContainer.<init> New CoreContainer 23065459
[junit4] 2> 716599 T2890 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/]
[junit4] 2> 716600 T2890 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 716600 T2890 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 716601 T2890 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 716601 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 716601 T2890 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 716602 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 716602 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 716603 T2890 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 716603 T2890 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 716607 T2890 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 716607 T2890 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 716607 T2890 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 716608 T2890 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46497/solr
[junit4] 2> 716608 T2890 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 716609 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 716612 T2934 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@172b02b name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 716613 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 716616 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 716618 T2936 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1acd247 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 716618 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 716624 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 717627 T2890 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41275_
[junit4] 2> 717628 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41275_
[junit4] 2> 717635 T2890 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 717636 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 717924 T2890 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 717925 T2890 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 717930 T2890 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 717932 T2890 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:54499
[junit4] 2> 717935 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 717935 T2890 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 717935 T2890 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004
[junit4] 2> 717936 T2890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/'
[junit4] 2> 717953 T2890 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solr.xml
[junit4] 2> 717965 T2890 oasc.CoreContainer.<init> New CoreContainer 29398386
[junit4] 2> 717966 T2890 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/]
[junit4] 2> 717966 T2890 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 717967 T2890 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 717967 T2890 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 717967 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 717968 T2890 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 717968 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 717968 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 717969 T2890 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 717969 T2890 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 717971 T2890 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 717972 T2890 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 717972 T2890 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 717973 T2890 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46497/solr
[junit4] 2> 717973 T2890 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 717974 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717976 T2948 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de81bc name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717976 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717978 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717980 T2950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e02d42 name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717980 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717987 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 718989 T2890 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54499_
[junit4] 2> 718990 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54499_
[junit4] 2> 719001 T2890 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 719003 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 719304 T2890 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 719304 T2890 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 719309 T2890 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 719311 T2890 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48090
[junit4] 2> 719313 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 719314 T2890 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 719315 T2890 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005
[junit4] 2> 719315 T2890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/'
[junit4] 2> 719334 T2890 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solr.xml
[junit4] 2> 719345 T2890 oasc.CoreContainer.<init> New CoreContainer 14930487
[junit4] 2> 719346 T2890 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/]
[junit4] 2> 719346 T2890 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 719347 T2890 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 719347 T2890 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 719347 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 719348 T2890 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 719348 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 719348 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 719349 T2890 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 719349 T2890 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 719352 T2890 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 719352 T2890 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 719352 T2890 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 719353 T2890 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46497/solr
[junit4] 2> 719353 T2890 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 719354 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719356 T2962 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a7a21 name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719356 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719358 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 719360 T2964 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15ff79e name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 719360 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 719366 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 720369 T2890 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48090_
[junit4] 2> 720370 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48090_
[junit4] 2> 720378 T2890 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 720379 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 720656 T2890 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
[junit4] 2> 720657 T2890 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 720660 T2890 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 720663 T2890 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:59603
[junit4] 2> 720665 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 720666 T2890 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 720666 T2890 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-006
[junit4] 2> 720666 T2890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-006/'
[junit4] 2> 720698 T2890 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-006/solr.xml
[junit4] 2> 720710 T2890 oasc.CoreContainer.<init> New CoreContainer 16574351
[junit4] 2> 720710 T2890 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-006/]
[junit4] 2> 720711 T2890 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 720712 T2890 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
[junit4] 2> 720712 T2890 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 720712 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 720713 T2890 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 720713 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 720713 T2890 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 720714 T2890 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 720714 T2890 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 720718 T2890 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 720718 T2890 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 720719 T2890 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 720719 T2890 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46497/solr
[junit4] 2> 720719 T2890 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 720720 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 720722 T2976 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c257be name:ZooKeeperConnection Watcher:127.0.0.1:46497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 720722 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 720725 T2890 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 720726 T2978 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@193cd2f name:ZooKeeperConnection Watcher:127.0.0.1:46497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 720727 T2890 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 720733 T2890 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 721736 T2890 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59603_
[junit4] 2> 721737 T2890 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59603_
[junit4] 2> 721746 T2890 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 721747 T2890 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 721828 T2931 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&router.name=compositeId&action=CREATE&numShards=2&wt=javabin&version=2
[junit4] 2> 721830 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4] 2> 721832 T2920 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4] 2> "operation":"createcollection",
[junit4] 2> "fromApi":"true",
[junit4] 2> "name":"solrj_collection",
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"2",
[junit4] 2> "router.field":"myOwnField",
[junit4] 2> "router.name":"compositeId"}
[junit4] 2> 721833 T2920 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
[junit4] 2> "operation":"createcollection",
[junit4] 2> "fromApi":"true",
[junit4] 2> "name":"solrj_collection",
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"2",
[junit4] 2> "router.field":"myOwnField",
[junit4] 2> "router.name":"compositeId"}
[junit4] 2> 721834 T2920 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection
[junit4] 2> 721834 T2920 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
[junit4] 2> 721838 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 721839 T2919 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 721840 T2919 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
[junit4] 2> 721840 T2919 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
[junit4] 2> 721849 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 721850 T2936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721850 T2978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721850 T2923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721850 T2918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721850 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721850 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 721938 T2920 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2] , replicationFactor : 2
[junit4] 2> 721939 T2920 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:48090_
[junit4] 2> 721941 T2920 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54499_
[junit4] 2> 721945 T2920 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:41275_
[junit4] 2> 721947 T2920 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:38483_
[junit4] 2> 722023 T2944 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
[junit4] 2> 722025 T2944 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
[junit4] 2> 722027 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722027 T2944 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 722029 T2919 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 722029 T2919 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:54499",
[junit4] 2> "core":"solrj_collection_shard1_replica2",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:54499_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"solrj_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 722032 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722034 T2930 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
[junit4] 2> 722036 T2930 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
[junit4] 2> 722038 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722040 T2919 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:41275",
[junit4] 2> "core":"solrj_collection_shard2_replica1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41275_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"solrj_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 722042 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722042 T2930 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 722053 T2957 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
[junit4] 2> 722055 T2911 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
[junit4] 2> 722056 T2957 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
[junit4] 2> 722057 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722057 T2957 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 722058 T2911 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
[junit4] 2> 722058 T2919 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:48090",
[junit4] 2> "core":"solrj_collection_shard1_replica1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:48090_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"solrj_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 722059 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722059 T2911 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 722062 T2919 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38483",
[junit4] 2> "core":"solrj_collection_shard2_replica2",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38483_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"solrj_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 722064 T2918 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 722167 T2936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 722167 T2918 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 722167 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 722167 T2923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 722167 T2978 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 722167 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 723028 T2944 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
[junit4] 2> 723029 T2944 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solr.xml
[junit4] 2> 723029 T2944 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
[junit4] 2> 723030 T2944 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 723030 T2944 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
[junit4] 2> 723032 T2944 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 723032 T2944 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/'
[junit4] 2> 723044 T2930 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
[junit4] 2> 723045 T2930 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solr.xml
[junit4] 2> 723046 T2930 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
[junit4] 2> 723047 T2930 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 723048 T2930 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
[junit4] 2> 723049 T2930 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 723049 T2930 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/'
[junit4] 2> 723058 T2957 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
[junit4] 2> 723060 T2911 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
[junit4] 2> 723060 T2957 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solr.xml
[junit4] 2> 723061 T2911 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solr.xml
[junit4] 2> 723061 T2957 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
[junit4] 2> 723062 T2911 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
[junit4] 2> 723063 T2957 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 723063 T2957 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
[junit4] 2> 723063 T2911 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 723064 T2911 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
[junit4] 2> 723064 T2957 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 723065 T2957 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/'
[junit4] 2> 723067 T2911 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 723067 T2911 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/'
[junit4] 2> 723081 T2944 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 723096 T2930 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 723098 T2944 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 723099 T2944 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 723117 T2930 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 723118 T2957 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 723119 T2930 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 723124 T2944 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
[junit4] 2> 723125 T2911 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 723130 T2930 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
[junit4] 2> 723139 T2957 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 723142 T2957 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 723147 T2911 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 723149 T2911 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 723168 T2957 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
[junit4] 2> 723176 T2911 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
[junit4] 2> 723319 T2930 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 723327 T2930 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 723329 T2930 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 723345 T2930 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723348 T2930 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723356 T2944 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 723358 T2930 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723360 T2930 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723361 T2930 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723361 T2930 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723363 T2930 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723363 T2930 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723364 T2944 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 723364 T2930 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
[junit4] 2> 723365 T2930 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 723365 T2930 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/data/
[junit4] 2> 723366 T2944 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 723366 T2930 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a3b32c
[junit4] 2> 723370 T2930 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/data
[junit4] 2> 723371 T2930 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
[junit4] 2> 723371 T2957 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 723372 T2930 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4] 2> 723374 T2930 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-003/solrj_collection_shard2_replica1/data/index
[junit4] 2> 723377 T2930 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=73.0107421875, floorSegmentMB=0.3388671875, forceMergeDeletesPctAllowed=13.023363852477885, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 723378 T2957 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 723381 T2957 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 723381 T2930 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@124feac lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a8ebd),segFN=segments_1,generation=1}
[junit4] 2> 723383 T2930 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 723394 T2911 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 723397 T2944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723400 T2930 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 723401 T2930 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 723401 T2930 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 723402 T2930 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 723402 T2911 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 723403 T2930 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 723403 T2944 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723404 T2957 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723403 T2930 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 723405 T2911 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 723405 T2930 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723406 T2930 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723407 T2930 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 723408 T2944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723408 T2930 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 723408 T2957 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723410 T2944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723410 T2930 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 723411 T2930 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 723411 T2944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723412 T2930 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 723413 T2944 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723414 T2930 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 723415 T2930 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 723415 T2944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723415 T2944 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723415 T2957 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723416 T2944 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
[junit4] 2> 723416 T2930 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 723417 T2944 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 723417 T2957 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723418 T2957 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723418 T2930 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 723418 T2944 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/data/
[junit4] 2> 723419 T2957 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723420 T2944 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a3b32c
[junit4] 2> 723421 T2957 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723421 T2957 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723421 T2911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723422 T2944 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/data
[junit4] 2> 723422 T2957 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
[junit4] 2> 723422 T2944 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/data/index/
[junit4] 2> 723422 T2930 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 723423 T2944 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4] 2> 723423 T2957 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 723425 T2944 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-004/solrj_collection_shard1_replica2/data/index
[junit4] 2> 723425 T2911 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 723425 T2957 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/data/
[junit4] 2> 723425 T2944 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=73.0107421875, floorSegmentMB=0.3388671875, forceMergeDeletesPctAllowed=13.023363852477885, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 723426 T2930 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 723426 T2957 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a3b32c
[junit4] 2> 723428 T2944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@16381dd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10b6cd0),segFN=segments_1,generation=1}
[junit4] 2> 723429 T2944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 723428 T2930 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 723429 T2957 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/data
[junit4] 2> 723431 T2957 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/data/index/
[junit4] 2> 723431 T2911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723432 T2930 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=27.37890625, floorSegmentMB=1.900390625, forceMergeDeletesPctAllowed=13.614706086156122, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 723432 T2957 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4] 2> 723433 T2911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723434 T2911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723434 T2957 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-005/solrj_collection_shard1_replica1/data/index
[junit4] 2> 723433 T2930 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@124feac lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a8ebd),segFN=segments_1,generation=1}
[junit4] 2> 723435 T2957 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=73.0107421875, floorSegmentMB=0.3388671875, forceMergeDeletesPctAllowed=13.023363852477885, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 723434 T2911 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 723436 T2957 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d19142 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@180743e),segFN=segments_1,generation=1}
[junit4] 2> 723437 T2957 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 723435 T2930 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 723437 T2944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 723438 T2911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 723437 T2930 oass.SolrIndexSearcher.<init> Opening Searcher@10ddbce[solrj_collection_shard2_replica1] main
[junit4] 2> 723438 T2911 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 723438 T2944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 723439 T2911 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
[junit4] 2> 723439 T2930 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
[junit4] 2> 723440 T2911 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 723440 T2944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 723441 T2911 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/data/
[junit4] 2> 723441 T2944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 723442 T2930 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 723442 T2930 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 723442 T2911 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@a3b32c
[junit4] 2> 723442 T2944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 723444 T2930 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 723444 T2957 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 723444 T2911 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/data
[junit4] 2> 723444 T2944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 723445 T2911 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/data/index/
[junit4] 2> 723445 T2957 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 723444 T2930 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 723446 T2957 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 723446 T2911 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4] 2> 723446 T2944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723448 T2911 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-6D0A8E25DED4B354-001/tempDir-002/solrj_collection_shard2_replica2/data/index
[junit4] 2> 723447 T2957 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 723447 T2930 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 723449 T2957 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 723449 T2911 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=73.0107421875, floorSegmentMB=0.3388671875, forceMergeDeletesPctAllowed=13.023363852477885, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 723448 T2944 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723450 T2930 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 723450 T2957 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 723451 T2930 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 723452 T2930 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 723452 T2930 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 723450 T2911 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@17803c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bd0c9b),segFN=segments_1,generation=1}
[junit4] 2> 723450 T2944 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 723453 T2911 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 723451 T2957 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723454 T2944 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 723454 T2957 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 723455 T2957 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 723455 T2944 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 723456 T2944 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 723456 T2957 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 723457 T2944 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 723458 T2984 oasc.SolrCore.registerSearcher [solrj_collection_shard2_replica1] Registered new searcher Searcher@10ddbce[solrj_collection_shard2_replica1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 723458 T2957 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 723458 T2957 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHan
[...truncated too long message...]
T3040 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 821105 T3040 oaz.ClientCnxn$SendThread.run WARN Session 0x145db8e1b220012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 821603 T3040 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2>
[junit4] 2> 822104 T3040 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 822105 T3040 oaz.ClientCnxn$SendThread.run WARN Session 0x145db8e1b220012 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 119148 T2889 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=sr_ME_#Latn, timezone=Europe/Uzhgorod
[junit4] 2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.8.0_20-ea (32-bit)/cpus=8,threads=2,free=67515848,total=246415360
[junit4] 2> NOTE: All tests run in this JVM: [ResourceLoaderTest, NoCacheHeaderTest, SortByFunctionTest, TestSolrQueryParserResource, PluginInfoTest, CursorPagingTest, DirectSolrSpellCheckerTest, FileUtilsTest, TestComplexPhraseQParserPlugin, TestZkChroot, ParsingFieldUpdateProcessorsTest, TestAnalyzedSuggestions, TestFiltering, TestCustomSort, TestDefaultSearchFieldResource, TestDynamicFieldResource, QueryResultKeyTest, TestExpandComponent, TestArbitraryIndexDir, TestUpdate, UniqFieldsUpdateProcessorFactoryTest, SpellCheckCollatorTest, TestLMDirichletSimilarityFactory, TermVectorComponentDistributedTest, DirectUpdateHandlerTest, TestIndexSearcher, TestIBSimilarityFactory, StandardRequestHandlerTest, TestCollationField, DistributedDebugComponentTest, RangeFacetTest, TestStressRecovery, AsyncMigrateRouteKeyTest, StressHdfsTest, CollectionsAPIAsyncDistributedZkTest, AtomicUpdatesTest, TestFuzzyAnalyzedSuggestions, SuggesterTSTTest, TestRequestStatusCollectionAPI, TestFieldCollectionResource, TestDocumentBuilder, TestWordDelimiterFilterFactory, TestSweetSpotSimilarityFactory, TestHashPartitioner, TestManagedSchemaFieldResource, TestComponentsName, TestManagedResource, TestRandomMergePolicy, TestSimpleQParserPlugin, HardAutoCommitTest, DOMUtilTest, SystemInfoHandlerTest, ChaosMonkeySafeLeaderTest, TestFoldingMultitermQuery, TestSolrJ, CoreAdminRequestStatusTest, TestOmitPositions, TestReload, BasicZkTest, JSONWriterTest, HdfsRecoveryZkTest, TestRecovery, TestCSVLoader, FullSolrCloudDistribCmdsTest, TestSolrXMLSerializer, TestCloudManagedSchema, AssignTest, DistanceFunctionTest, TestSchemaNameResource, TestSolr4Spatial, TestSolrXmlPersistor, TestDistributedSearch, TestCodecSupport, TestRTGBase, MinimalSchemaTest, PrimUtilsTest, TestPartialUpdateDeduplication, CoreContainerCoreInitFailuresTest, XmlUpdateRequestHandlerTest, BadIndexSchemaTest, PingRequestHandlerTest, TestSchemaVersionResource, TestQuerySenderNoQuery, RequestHandlersTest, HdfsLockFactoryTest, SolrPluginUtilsTest, TestOverriddenPrefixQueryForCustomFieldType, TestStressReorder, TestSolrCoreProperties, EnumFieldTest, HdfsBasicDistributedZkTest, ZkSolrClientTest, TestDFRSimilarityFactory, TestIndexingPerformance, DocExpirationUpdateProcessorFactoryTest, HdfsSyncSliceTest, SignatureUpdateProcessorFactoryTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.seed=6D0A8E25DED4B354 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_ME_#Latn -Dtests.timezone=Europe/Uzhgorod -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | CollectionsAPIDistributedZkTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=21 closes=20
[junit4] > at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest:
[junit4] > 1) Thread[id=3041, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-EventThread, state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > 2) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3040, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[6D0A8E25DED4B354]-SendThread(127.0.0.1:46497), state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] > at __randomizedtesting.SeedInfo.seed([6D0A8E25DED4B354]:0)
[junit4] Completed on J1 in 119.22s, 1 test, 2 failures, 2 errors <<< FAILURES!
[...truncated 594 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 398 suites, 1673 tests, 3 suite-level errors, 1 failure, 704 ignored (16 assumptions)
Total time: 58 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_20-ea-b11 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b15) -
Build # 10259 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10259/
Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
All tests passed
Build Log:
[...truncated 20421 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java
Total time: 64 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b15 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_55) - Build #
10258 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10258/
Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseG1GC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test
Error Message:
ERROR: SolrZkClient opens=13 closes=12
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=13 closes=12
at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test: 1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) 2) Thread[id=3186, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-EventThread, state=WAITING, group=TGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:
1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
2) Thread[id=3186, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-EventThread, state=WAITING, group=TGRP-BasicDistributedZk2Test]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)
REGRESSION: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 503 Server is shutting down or failed to initialize</title> </head> <body> <h2>HTTP ERROR: 503</h2> <p>Problem accessing /qrb/mq/admin/cores. Reason: <pre> Server is shutting down or failed to initialize</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html>
Stack Trace:
java.lang.AssertionError: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Server is shutting down or failed to initialize</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /qrb/mq/admin/cores. Reason:
<pre> Server is shutting down or failed to initialize</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([B0CE312826844528:3128BF3051DB2514]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:176)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:82)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11191 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/init-core-data-001
[junit4] 2> 685415 T3141 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /qrb/mq
[junit4] 2> 685419 T3141 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 685419 T3141 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 685420 T3142 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 685520 T3141 oasc.ZkTestServer.run start zk server on port:44425
[junit4] 2> 685520 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 685567 T3148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22ad90f6 name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 685568 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 685568 T3141 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 685572 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 685573 T3150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66c5f0e9 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 685574 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 685574 T3141 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 685577 T3141 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 685580 T3141 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 685581 T3141 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 685583 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 685583 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 685586 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 685587 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 685589 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 685589 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 685591 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 685591 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 685593 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 685593 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 685594 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 685595 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 685596 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 685596 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 685598 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 685598 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 685600 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 685600 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 685602 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 685602 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 685605 T3141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 685605 T3141 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 685883 T3141 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 685887 T3141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34666
[junit4] 2> 685887 T3141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 685888 T3141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 685888 T3141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002
[junit4] 2> 685888 T3141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/'
[junit4] 2> 685909 T3141 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/solr.xml
[junit4] 2> 685962 T3141 oasc.CoreContainer.<init> New CoreContainer 2026325894
[junit4] 2> 685962 T3141 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/]
[junit4] 2> 685964 T3141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 685964 T3141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 685964 T3141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 685965 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 685965 T3141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 685966 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 685966 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 685966 T3141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 685967 T3141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 685967 T3141 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 685968 T3141 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 685968 T3141 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 685969 T3141 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44425/solr
[junit4] 2> 685969 T3141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 685970 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 685971 T3161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6328ec5c name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 685972 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 685974 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 685975 T3163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dcaa32f name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 685975 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 685976 T3141 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 685978 T3141 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 685980 T3141 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 685982 T3141 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 685983 T3141 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 685985 T3141 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 685986 T3141 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34666_qrb%2Fmq
[junit4] 2> 685987 T3141 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34666_qrb%2Fmq
[junit4] 2> 685989 T3141 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 685990 T3141 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 687046 T3145 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1055ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 687050 T3141 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34666_qrb%2Fmq
[junit4] 2> 687050 T3141 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 687052 T3141 oasc.Overseer.start Overseer (id=91719318907584515-127.0.0.1:34666_qrb%2Fmq-n_0000000000) starting
[junit4] 2> 687054 T3141 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 687059 T3165 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 687060 T3141 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 687060 T3165 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:34666_qrb%2Fmq
[junit4] 2> 687062 T3141 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 687063 T3141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 687064 T3164 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 687068 T3166 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 687068 T3166 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 687069 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 687069 T3166 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 687070 T3164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 687070 T3164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34666/qrb/mq",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34666_qrb%2Fmq",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 687070 T3164 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 687070 T3164 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 687072 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 687072 T3163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 688069 T3166 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 688069 T3166 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 688070 T3166 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 688070 T3166 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 688071 T3166 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 688071 T3166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/collection1/'
[junit4] 2> 688071 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 688072 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 688072 T3166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 688135 T3166 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 688183 T3166 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 688184 T3166 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 688193 T3166 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 688519 T3166 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 688525 T3166 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 688528 T3166 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 688536 T3166 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 688539 T3166 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 688542 T3166 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 688542 T3166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 688543 T3166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 688543 T3166 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 688544 T3166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 688544 T3166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 688544 T3166 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 688544 T3166 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 688545 T3166 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-002/collection1/, dataDir=./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-001/control/data/
[junit4] 2> 688545 T3166 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@34e04c26
[junit4] 2> 688545 T3166 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-001/control/data
[junit4] 2> 688546 T3166 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-001/control/data/index/
[junit4] 2> 688546 T3166 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 688546 T3166 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-001/control/data/index
[junit4] 2> 688546 T3166 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=93.748046875, floorSegmentMB=0.89453125, forceMergeDeletesPctAllowed=24.729679776307954, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 688547 T3166 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1245c4a3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b94be11),segFN=segments_1,generation=1}
[junit4] 2> 688547 T3166 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 688550 T3166 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 688550 T3166 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 688551 T3166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 688551 T3166 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 688551 T3166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 688551 T3166 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 688552 T3166 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 688552 T3166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 688552 T3166 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 688552 T3166 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 688553 T3166 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 688553 T3166 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 688553 T3166 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 688554 T3166 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 688554 T3166 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 688554 T3166 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 688555 T3166 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 688557 T3166 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 688559 T3166 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 688560 T3166 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 688560 T3166 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=19, maxMergeAtOnceExplicit=50, maxMergedSegmentMB=81.392578125, floorSegmentMB=0.609375, forceMergeDeletesPctAllowed=7.647986325812458, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 688560 T3166 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1245c4a3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b94be11),segFN=segments_1,generation=1}
[junit4] 2> 688561 T3166 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 688561 T3166 oass.SolrIndexSearcher.<init> Opening Searcher@13ee0014[collection1] main
[junit4] 2> 688561 T3166 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 688562 T3166 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 688562 T3166 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 688563 T3166 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 688563 T3166 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 688563 T3166 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 688563 T3166 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 688564 T3166 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 688564 T3166 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 688564 T3166 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 688567 T3167 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13ee0014[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 688568 T3166 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 688569 T3170 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34666/qrb/mq collection:control_collection shard:shard1
[junit4] 2> 688569 T3141 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 688569 T3141 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 688570 T3170 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 688570 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 688589 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 688589 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 688609 T3141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 688668 T3141 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 688668 T3170 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 688708 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 688709 T3170 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 688709 T3170 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1069 name=collection1 org.apache.solr.core.SolrCore@64e8fbf6 url=http://127.0.0.1:34666/qrb/mq/collection1 node=127.0.0.1:34666_qrb%2Fmq C1069_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:34666/qrb/mq, core=collection1, node_name=127.0.0.1:34666_qrb%2Fmq}
[junit4] 2> 688709 T3170 C1069 P34666 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34666/qrb/mq/collection1/
[junit4] 2> 688709 T3170 C1069 P34666 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 688710 T3170 C1069 P34666 oasc.SyncStrategy.syncToMe http://127.0.0.1:34666/qrb/mq/collection1/ has no replicas
[junit4] 2> 688710 T3170 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34666/qrb/mq/collection1/ shard1
[junit4] 2> 688710 T3170 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 688711 T3164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 688901 T3141 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 688902 T3141 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 688904 T3141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34555
[junit4] 2> 688905 T3141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 688905 T3141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 688906 T3141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-003
[junit4] 2> 688906 T3141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-003/'
[junit4] 2> 688920 T3141 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-003/solr.xml
[junit4] 2> 688961 T3141 oasc.CoreContainer.<init> New CoreContainer 278058062
[junit4] 2> 688961 T3141 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-003/]
[junit4] 2> 688962 T3141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 688962 T3141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 688963 T3141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 688963 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 688963 T3141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 688963 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 688963 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 688964 T3141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 688964 T3141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 688964 T3141 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 688965 T3141 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 688965 T3141 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 688965 T3141 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44425/solr
[junit4] 2> 688965 T3141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 688966 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 695335 T3143 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145da3c0e870004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 695435 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 695435 T3172 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 697067 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 702067 T3143 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145da3c0e870003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 702167 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: null state: Disconnected type None
[junit4] 2> 702168 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: null state: Disconnected type None
[junit4] 2> 702168 T3163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dcaa32f name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 702168 T3163 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 703736 T3143 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145da3c0e870004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 703824 T3163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dcaa32f name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 703825 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: null state: SyncConnected type None
[junit4] 2> 703825 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: null state: SyncConnected type None
[junit4] 2> 703836 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 703836 T3172 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 705333 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 712002 T3143 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145da3c0e870004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 712102 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 712102 T3172 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 713351 T3172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fd08d0 name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715433 T3145 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 26703ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 715434 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715435 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715435 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715436 T3183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b05562 name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 715436 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715436 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 715436 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715437 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715438 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715438 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715439 T3170 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:291)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:956)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:773)
[junit4] 2> at org.apache.solr.cloud.ZkController.register(ZkController.java:736)
[junit4] 2> at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:200)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/control_collection/leaders/shard1
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
[junit4] 2> ... 11 more
[junit4] 2>
[junit4] 2> 715439 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715439 T3170 oasc.ShardLeaderElectionContext.rejoinLeaderElection There may be a better leader candidate than us - going back into recovery
[junit4] 2> 715440 T3170 oasc.ElectionContext.cancelElection canceling election /collections/control_collection/leader_elect/shard1/election/91719318907584515-core_node1-n_0000000000
[junit4] 2> 715441 T3170 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> 715441 T3184 C1069 P34666 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 715441 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 715442 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1118)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:120)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:92)
[junit4] 2> at org.apache.zookeeper.server.DataTree.setData(DataTree.java:620)
[junit4] 2> at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
[junit4] 2> at org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
[junit4] 2> at org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:994)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715442 T3172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 715441 T3184 C1069 P34666 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 715442 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1118)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:120)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:92)
[junit4] 2> at org.apache.zookeeper.server.DataTree.setData(DataTree.java:620)
[junit4] 2> at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:807)
[junit4] 2> at org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
[junit4] 2> at org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:994)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 715443 T3163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 715444 T3170 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 715457 T3170 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4] 2> 715457 T3170 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4] 2> 715457 T3170 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 715457 T3170 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node1core=collection1
[junit4] 2> 715459 T3164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 715460 T3163 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 715540 T3183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b05562 name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 715541 T3183 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 715563 T3172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 715563 T3163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 717348 T3183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b05562 name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:Expired type:None path:null path:null type:None
[junit4] 2> 717349 T3183 oascc.ConnectionManager.process Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper...
[junit4] 2> 717350 T3183 oascc.DefaultConnectionStrategy.reconnect Connection expired - starting a new one...
[junit4] 2> 717350 T3183 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717353 T3186 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b05562 name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717353 T3183 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717354 T3183 oascc.ConnectionManager$1.update Connection with ZooKeeper reestablished.
[junit4] 2> 717354 T3183 oascc.DefaultConnectionStrategy.reconnect Reconnected to ZooKeeper
[junit4] 2> 717354 T3183 oascc.ConnectionManager.process Connected:true
[junit4] 2> 717449 T3141 oasc.ZkContainer.initZooKeeper ERROR org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /solr
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:662)
[junit4] 2> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:127)
[junit4] 2> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:61)
[junit4] 2> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:190)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
[junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:325)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:430)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:403)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:481)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:361)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:305)
[junit4] 2> at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 717451 T3141 oass.SolrDispatchFilter.init ERROR Could not start Solr. Check solr/home property and the logs
[junit4] 2> 717452 T3141 oasc.SolrException.log ERROR null:org.apache.solr.common.cloud.ZooKeeperException:
[junit4] 2> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:186)
[junit4] 2> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:61)
[junit4] 2> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:190)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.doStart(FilterHolder.java:119)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:719)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1309)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1345)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1085)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:931)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:888)
[junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:340)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:325)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:174)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:65)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:430)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:403)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJetty(AbstractFullDistribZkTestBase.java:481)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:361)
[junit4] 2> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:305)
[junit4] 2> at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /solr
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.cloud.ZkController.checkChrootPath(ZkController.java:662)
[junit4] 2> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:127)
[junit4] 2> ... 63 more
[junit4] 2>
[junit4] 2> 717453 T3141 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 717696 T3141 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 717697 T3141 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 717699 T3141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46979
[junit4] 2> 717700 T3141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 717700 T3141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 717700 T3141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-004
[junit4] 2> 717700 T3141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-004/'
[junit4] 2> 717715 T3141 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-004/solr.xml
[junit4] 2> 717755 T3141 oasc.CoreContainer.<init> New CoreContainer 689216134
[junit4] 2> 717755 T3141 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solr.cloud.BasicDistributedZk2Test-B0CE312826844528-001/tempDir-004/]
[junit4] 2> 717756 T3141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 717757 T3141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 717757 T3141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 717757 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 717757 T3141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 717758 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 717758 T3141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 717758 T3141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 717758 T3141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 717759 T3141 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 717759 T3141 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 717759 T3141 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 717760 T3141 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44425/solr
[junit4] 2> 717760 T3141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 717760 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717762 T3197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ace87c name:ZooKeeperConnection Watcher:127.0.0.1:44425 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717763 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717765 T3141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 717767 T3199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d40733b name:ZooKeeperConnection Watcher:127.0.0.1:44425/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 717767 T3141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 717773 T3141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 718775 T3141 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46979_qrb%2Fmq
[junit4] 2> 718776 T3141 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46979_qrb%2Fmq
[junit4] 2> 718778 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1118)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:120)
[junit4] 2> at org.apache.zookeeper.server.WatchManager.triggerWatch(WatchManager.java:92)
[junit4] 2> at org.apache.zookeeper.server.DataTree.createNode(DataTree.java:523)
[junit4] 2> at org.apache.zookeeper.server.DataTree.processTxn(DataTree.java:791)
[junit4] 2> at org.apache.zookeeper.server.ZKDatabase.processTxn(ZKDatabase.java:329)
[junit4] 2> at org.apache.zookeeper.server.ZooKeeperServer.processTxn(ZooKeeperServer.java:994)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:116)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 718780 T3145 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionK
[...truncated too long message...]
ientCnxn.java:994)
[junit4] 2>
[junit4] 2> 751198 T3185 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 751199 T3185 oaz.ClientCnxn$SendThread.run WARN Session 0x145da3c0e870006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 751698 T3185 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2>
[junit4] 2> 752198 T3185 oazc.StaticHostProvider.next WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 752199 T3185 oaz.ClientCnxn$SendThread.run WARN Session 0x145da3c0e870006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[junit4] 2>
[junit4] 2> 752698 T3185 oaz.ClientCnxn$SendThread.run WARN Unexpected exception java.lang.InterruptedException: sleep interrupted
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2>
[junit4] 2> 67785 T3140 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ro_RO, timezone=Asia/Ulan_Bator
[junit4] 2> NOTE: Linux 3.8.0-39-generic amd64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=8,threads=2,free=226738032,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [HdfsUnloadDistributedZkTest, CustomCollectionTest, CollectionsAPIDistributedZkTest, TestQuerySenderNoQuery, TestFiltering, TestRecovery, TestDistribDocBasedVersion, TestManagedStopFilterFactory, DateMathParserTest, TestStressLucene, TestSchemaResource, CachingDirectoryFactoryTest, TestExceedMaxTermLength, RequestHandlersTest, SolrIndexConfigTest, StressHdfsTest, BJQParserTest, PluginInfoTest, QueryResultKeyTest, DistributedQueryComponentOptimizationTest, TestZkChroot, SortByFunctionTest, StatsComponentTest, TestIndexSearcher, DirectSolrSpellCheckerTest, FileUtilsTest, TestReload, SpellCheckComponentTest, BadIndexSchemaTest, TestBinaryResponseWriter, XsltUpdateRequestHandlerTest, TestCSVLoader, TestSweetSpotSimilarityFactory, DisMaxRequestHandlerTest, TestSolrQueryParserResource, DistanceFunctionTest, TestManagedSchemaFieldResource, SSLMigrationTest, ChaosMonkeySafeLeaderTest, TestLRUCache, HighlighterTest, SystemInfoHandlerTest, CurrencyFieldXmlFileTest, CoreAdminRequestStatusTest, TestRangeQuery, MultiTermTest, TestSolr4Spatial, LeaderElectionIntegrationTest, NoCacheHeaderTest, TestStressRecovery, ShardRoutingCustomTest, TestDocumentBuilder, HighlighterConfigTest, OverseerRolesTest, XmlUpdateRequestHandlerTest, TestXIncludeConfig, DeleteShardTest, AlternateDirectoryTest, TestLMDirichletSimilarityFactory, PrimUtilsTest, TestCollationFieldDocValues, TestUpdate, TestCSVResponseWriter, NumericFieldsTest, DocumentAnalysisRequestHandlerTest, DistributedDebugComponentTest, TestMultiCoreConfBootstrap, OpenExchangeRatesOrgProviderTest, TestConfigSets, TestBinaryField, BasicDistributedZk2Test]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZk2Test -Dtests.seed=B0CE312826844528 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ro_RO -Dtests.timezone=Asia/Ulan_Bator -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | BasicDistributedZk2Test (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=13 closes=12
[junit4] > at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:451)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:185)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:
[junit4] > 1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > 2) Thread[id=3186, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-EventThread, state=WAITING, group=TGRP-BasicDistributedZk2Test]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3185, name=TEST-BasicDistributedZk2Test.testDistribSearch-seed#[B0CE312826844528]-SendThread(127.0.0.1:44425), state=TIMED_WAITING, group=TGRP-BasicDistributedZk2Test]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > at __randomizedtesting.SeedInfo.seed([B0CE312826844528]:0)
[junit4] Completed on J1 in 67.82s, 1 test, 2 failures, 2 errors <<< FAILURES!
[...truncated 699 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1296: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:920: There were test failures: 398 suites, 1673 tests, 3 suite-level errors, 1 failure, 944 ignored (12 assumptions)
Total time: 48 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_55 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure