You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/25 19:40:42 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) -
Build # 9904 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9904/
Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
at __randomizedtesting.SeedInfo.seed([E6F891076BD51336:58E98928CAF8644]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
... 45 more
Build Log:
[...truncated 11628 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 30306 T143 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=17 numCloses=17
[junit4] 2> 30306 T143 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 30307 T143 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 30310 T143 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 30313 T143 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395772696955
[junit4] 2> 30316 T143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 30318 T144 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 30418 T143 oasc.ZkTestServer.run start zk server on port:52972
[junit4] 2> 30457 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111109 T147 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 80633ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 111111 T143 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 111129 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 111131 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 111132 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 111133 T145 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8929b00004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 111136 T143 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52972 52972
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=E6F891076BD51336 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=WET -Dtests.file.encoding=UTF-8
[junit4] ERROR 81.0s J0 | CloudSolrServerTest.testShutdown <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
[junit4] > at __randomizedtesting.SeedInfo.seed([E6F891076BD51336:58E98928CAF8644]:0)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:52972 within 45000 ms
[junit4] > at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
[junit4] > ... 45 more
[junit4] 2> 111334 T143 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395772777976
[junit4] 2> 111335 T143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 111335 T151 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 111435 T143 oasc.ZkTestServer.run start zk server on port:40963
[junit4] 2> 111436 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111448 T157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f03c14 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 111449 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 111450 T143 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 111477 T152 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fa8a65fa0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 111477 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111479 T159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63f864 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 111479 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 111482 T143 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 111487 T143 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 111490 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 111492 T143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 111495 T143 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 111495 T143 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 111501 T143 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 111501 T143 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 111507 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 111507 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 111508 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 111508 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 111513 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 111513 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 111514 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 111514 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 111515 T143 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 111523 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111524 T161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2d985 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 111525 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 111594 T143 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 111598 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 111600 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:47295
[junit4] 2> 111602 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 111602 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 111602 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170
[junit4] 2> 111603 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/'
[junit4] 2> 111618 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/solr.xml
[junit4] 2> 111669 T143 oasc.CoreContainer.<init> New CoreContainer 16423974
[junit4] 2> 111669 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/]
[junit4] 2> 111670 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 111670 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 111671 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 111672 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 111675 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 111675 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 111675 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 111676 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
[junit4] 2> 111685 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 111685 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111687 T172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1760a96 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 111688 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 111693 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 111695 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9b695 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 111695 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 111705 T143 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 111710 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 111714 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 111718 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 111721 T143 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 111728 T143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 111731 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47295_
[junit4] 2> 111733 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47295_
[junit4] 2> 111736 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 111799 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 111810 T143 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47295_
[junit4] 2> 111810 T143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 111814 T143 oasc.Overseer.start Overseer (id=91473364777566212-127.0.0.1:47295_-n_0000000000) starting
[junit4] 2> 111825 T143 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 111857 T176 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 111858 T143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 111859 T176 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 111865 T143 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 111867 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 111873 T175 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 111879 T177 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 111880 T177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 111883 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 111883 T177 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 111886 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 111887 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:47295",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47295_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 111887 T175 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 111893 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 111897 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 111901 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 112884 T177 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 112884 T177 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 112885 T177 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 112885 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 112887 T177 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 112887 T177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/collection1/'
[junit4] 2> 112926 T177 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 112951 T177 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 112952 T177 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 112960 T177 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 112989 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 112995 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 113000 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 113005 T177 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 113296 T177 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 113298 T177 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 113300 T177 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 113300 T177 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 113303 T177 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 113303 T177 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 113303 T177 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395772778170/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/
[junit4] 2> 113304 T177 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 113304 T177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data
[junit4] 2> 113304 T177 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index/
[junit4] 2> 113304 T177 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 113305 T177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index
[junit4] 2> 113306 T177 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@2be308 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index),segFN=segments_1,generation=1}
[junit4] 2> 113306 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 113307 T177 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 113307 T177 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 113307 T177 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 113308 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 113309 T177 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 113311 T177 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 113313 T177 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 113313 T177 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 113314 T177 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@2be308 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/control/data/index),segFN=segments_1,generation=1}
[junit4] 2> 113315 T177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 113315 T177 oass.SolrIndexSearcher.<init> Opening Searcher@1640c79[collection1] main
[junit4] 2> 113315 T177 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 113317 T177 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 113317 T177 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 113320 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 113320 T177 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 113321 T177 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 113322 T177 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 113322 T177 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 113322 T177 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 113323 T177 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 113323 T178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1640c79[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 113323 T177 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 113324 T181 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:47295 collection:control_collection shard:shard1
[junit4] 2> 113325 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 113325 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 113331 T181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 113331 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 113333 T183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14747a3 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 113333 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 113334 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 113339 T143 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 113339 T181 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 113342 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113343 T181 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 113343 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 113343 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@33e9df url=https://127.0.0.1:47295/collection1 node=127.0.0.1:47295_ C24_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:47295, core=collection1, node_name=127.0.0.1:47295_}
[junit4] 2> 113344 T181 C24 P47295 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:47295/collection1/
[junit4] 2> 113345 T181 C24 P47295 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 113346 T181 C24 P47295 oasc.SyncStrategy.syncToMe https://127.0.0.1:47295/collection1/ has no replicas
[junit4] 2> 113346 T181 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:47295/collection1/ shard1
[junit4] 2> 113349 T181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 113355 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113362 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113366 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113398 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 113399 T143 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 113402 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 113403 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:35092
[junit4] 2> 113405 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 113405 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 113406 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980
[junit4] 2> 113406 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/'
[junit4] 2> 113420 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/solr.xml
[junit4] 2> 113450 T143 oasc.CoreContainer.<init> New CoreContainer 6608415
[junit4] 2> 113450 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/]
[junit4] 2> 113451 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 113452 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 113453 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 113454 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 113456 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 113457 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 113457 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 113457 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
[junit4] 2> 113458 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 113458 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 113461 T194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d31d61 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 113461 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 113463 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 113464 T196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1353d85 name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 113464 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 113467 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 113467 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 113478 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 113513 T181 oasc.ZkController.register We are https://127.0.0.1:47295/collection1/ and leader is https://127.0.0.1:47295/collection1/
[junit4] 2> 113514 T181 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:47295
[junit4] 2> 113514 T181 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 113514 T181 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 113514 T181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 113515 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113515 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113516 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113516 T181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 113517 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 113518 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:47295",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47295_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 113521 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 113623 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 113623 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 113624 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 114483 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35092_
[junit4] 2> 114485 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35092_
[junit4] 2> 114488 T174 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 114488 T196 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 114488 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 114495 T197 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 114496 T197 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 114497 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 114497 T197 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 114497 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 114498 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 114500 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 114500 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:35092",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35092_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 114501 T175 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 114501 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 114503 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 114607 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 114607 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 114607 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 115498 T197 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 115498 T197 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 115499 T197 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 115499 T197 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 115500 T197 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 115501 T197 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/collection1/'
[junit4] 2> 115539 T197 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 115573 T197 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 115574 T197 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 115586 T197 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 115844 T197 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 115846 T197 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 115848 T197 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 115848 T197 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 115849 T197 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 115849 T197 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 115849 T197 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395772779980/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/
[junit4] 2> 115849 T197 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 115850 T197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1
[junit4] 2> 115850 T197 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index/
[junit4] 2> 115850 T197 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 115850 T197 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index
[junit4] 2> 115852 T197 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7c7f14 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index),segFN=segments_1,generation=1}
[junit4] 2> 115852 T197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 115853 T197 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 115853 T197 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 115853 T197 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 115854 T197 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 115855 T197 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 115857 T197 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 115860 T197 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 115860 T197 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 115861 T197 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7c7f14 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty1/index),segFN=segments_1,generation=1}
[junit4] 2> 115861 T197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 115861 T197 oass.SolrIndexSearcher.<init> Opening Searcher@127a21a[collection1] main
[junit4] 2> 115861 T197 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 115862 T197 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 115863 T197 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 115863 T197 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 115863 T197 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 115864 T197 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 115867 T197 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 115868 T197 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 115868 T197 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 115868 T197 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 115870 T198 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@127a21a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 115870 T197 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 115871 T201 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:35092 collection:collection1 shard:shard1
[junit4] 2> 115872 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 115872 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 115875 T201 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 115883 T201 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 115886 T201 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 115886 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 115886 T201 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@1883f39 url=https://127.0.0.1:35092/collection1 node=127.0.0.1:35092_ C25_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:35092, core=collection1, node_name=127.0.0.1:35092_}
[junit4] 2> 115887 T201 C25 P35092 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:35092/collection1/
[junit4] 2> 115887 T201 C25 P35092 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 115888 T201 C25 P35092 oasc.SyncStrategy.syncToMe https://127.0.0.1:35092/collection1/ has no replicas
[junit4] 2> 115888 T201 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:35092/collection1/ shard1
[junit4] 2> 115888 T201 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 115889 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 115895 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 115901 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 115937 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 115938 T143 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 115942 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 115943 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50957
[junit4] 2> 115945 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 115946 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 115946 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515
[junit4] 2> 115946 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/'
[junit4] 2> 115958 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/solr.xml
[junit4] 2> 115988 T143 oasc.CoreContainer.<init> New CoreContainer 8610917
[junit4] 2> 115988 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/]
[junit4] 2> 115989 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 115990 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 115991 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 115992 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 115994 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 115994 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 115995 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 115995 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
[junit4] 2> 115995 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 115996 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 115998 T212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1182ea8 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 115998 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 116000 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 116107 T214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8b11e name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 116107 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 116109 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116109 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116109 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116115 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 116147 T201 oasc.ZkController.register We are https://127.0.0.1:35092/collection1/ and leader is https://127.0.0.1:35092/collection1/
[junit4] 2> 116147 T201 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:35092
[junit4] 2> 116147 T201 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 116147 T201 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 116147 T201 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 116149 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 116149 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 116150 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 116151 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:35092",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35092_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 116154 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 116256 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116256 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116256 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 116256 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 117119 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50957_
[junit4] 2> 117121 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50957_
[junit4] 2> 117126 T196 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 117126 T214 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 117126 T174 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 117127 T183 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 117141 T215 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 117141 T215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 117144 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 117144 T215 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 117145 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 117146 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 117148 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 117149 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50957",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50957_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 117150 T175 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 117151 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 117155 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 117259 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 117260 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 117260 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 117259 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118145 T215 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 118146 T215 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 118147 T215 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 118148 T215 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 118150 T215 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 118150 T215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/collection1/'
[junit4] 2> 118182 T215 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 118221 T215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 118223 T215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 118230 T215 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 118474 T215 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 118477 T215 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 118478 T215 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 118479 T215 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 118479 T215 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 118479 T215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 118480 T215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395772782515/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/
[junit4] 2> 118480 T215 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 118480 T215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2
[junit4] 2> 118481 T215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index/
[junit4] 2> 118481 T215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 118481 T215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index
[junit4] 2> 118482 T215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@c78e3a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index),segFN=segments_1,generation=1}
[junit4] 2> 118482 T215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 118483 T215 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 118484 T215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 118485 T215 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 118487 T215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 118490 T215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 118490 T215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 118491 T215 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@c78e3a lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index),segFN=segments_1,generation=1}
[junit4] 2> 118491 T215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 118491 T215 oass.SolrIndexSearcher.<init> Opening Searcher@1cc3776[collection1] main
[junit4] 2> 118492 T215 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 118493 T215 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 118494 T215 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 118494 T215 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 118495 T215 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 118495 T215 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 118496 T215 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 118496 T215 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 118496 T215 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 118496 T215 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 118497 T216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1cc3776[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 118497 T215 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 118498 T219 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50957 collection:collection1 shard:shard2
[junit4] 2> 118498 T143 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 118498 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 118498 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 118503 T219 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118505 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118505 T219 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 118506 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@1f3f11c url=https://127.0.0.1:50957/collection1 node=127.0.0.1:50957_ C26_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50957, core=collection1, node_name=127.0.0.1:50957_}
[junit4] 2> 118506 T219 C26 P50957 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50957/collection1/
[junit4] 2> 118506 T219 C26 P50957 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 118506 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 118506 T219 C26 P50957 oasc.SyncStrategy.syncToMe https://127.0.0.1:50957/collection1/ has no replicas
[junit4] 2> 118507 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50957/collection1/ shard2
[junit4] 2> 118507 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 118509 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118511 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118514 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118550 T143 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 118550 T143 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 118556 T143 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 118558 T143 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49088
[junit4] 2> 118560 T143 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 118560 T143 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 118561 T143 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141
[junit4] 2> 118561 T143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/'
[junit4] 2> 118583 T143 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/solr.xml
[junit4] 2> 118621 T143 oasc.CoreContainer.<init> New CoreContainer 3282466
[junit4] 2> 118621 T143 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/]
[junit4] 2> 118622 T143 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 118624 T143 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 118625 T143 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 118625 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118625 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118625 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118625 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118625 T143 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 118627 T143 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 118627 T143 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 118630 T143 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 118631 T143 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 118631 T143 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 118631 T143 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40963/solr
[junit4] 2> 118632 T143 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 118632 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 118634 T230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7da08 name:ZooKeeperConnection Watcher:127.0.0.1:40963 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 118634 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 118638 T143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 118639 T232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19823ed name:ZooKeeperConnection Watcher:127.0.0.1:40963/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 118639 T143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 118649 T143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 118670 T219 oasc.ZkController.register We are https://127.0.0.1:50957/collection1/ and leader is https://127.0.0.1:50957/collection1/
[junit4] 2> 118670 T219 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50957
[junit4] 2> 118670 T219 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 118670 T219 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 118671 T219 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 118672 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118673 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118673 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118673 T219 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 118676 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 118677 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50957",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50957_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 118681 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 118784 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118785 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118784 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118784 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 118785 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 119654 T143 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49088_
[junit4] 2> 119657 T143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49088_
[junit4] 2> 119661 T196 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 119661 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 119661 T174 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 119662 T232 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 119662 T214 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 119675 T233 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 119675 T233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 119678 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 119678 T233 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 119678 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 119679 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 119682 T175 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 119683 T175 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:49088",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49088_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 119684 T175 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 119684 T175 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 119688 T174 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 119792 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 119792 T174 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 119792 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 119792 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 119792 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 120679 T233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 120680 T233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 120681 T233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 120681 T233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 120683 T233 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 120684 T233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395772785141/collection1/'
[junit4] 2> 120720 T233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 120746 T233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 120747 T233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 120755 T233 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 121005 T233 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 121007 T233 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 121009 T233 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 121009 T233 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 121010 T233 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collec
[...truncated too long message...]
faultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 129559 T253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 129561 T143 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 129561 T253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50957",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50957_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 129562 T143 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 129563 T143 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2;done=false>>]
[junit4] 2> 129563 T143 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2
[junit4] 2> 129564 T143 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index;done=false>>]
[junit4] 2> 129564 T143 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty2/index
[junit4] 2> 129566 T214 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 129568 T214 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 129570 T183 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 129592 T143 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 129643 T143 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49088
[junit4] 2> 129644 T143 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=3282466
[junit4] 2> 129644 T143 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 129645 T143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 129650 T183 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 129672 T143 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 129727 T143 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47295
[junit4] 2> 129736 T143 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 129738 T143 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40963 40963
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 129887 T143 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 130975 T196 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 131984 T232 C27 P49088 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 131991 T232 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 131992 T232 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1a7bed0
[junit4] 2> 131993 T232 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
[junit4] 2> 131993 T232 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 131994 T232 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 131994 T232 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 131997 T232 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 131997 T232 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 131997 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3;done=false>>]
[junit4] 2> 131998 T232 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3
[junit4] 2> 131998 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index;done=false>>]
[junit4] 2> 131998 T232 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395772777975/jetty3/index
[junit4] 2> 131999 T232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 131999 T232 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 132000 T232 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 132001 T232 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene46: {text=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=FSTPulsing41), range_facet_sl=PostingsFormat(name=MockSep), range_facet_l=PostingsFormat(name=FSTPulsing41), multiDefault=PostingsFormat(name=NestedPulsing), intDefault=PostingsFormat(name=FSTPulsing41), timestamp=PostingsFormat(name=FSTPulsing41), range_facet_si=PostingsFormat(name=NestedPulsing), id=PostingsFormat(name=Lucene41VarGapFixedInterval), a_t=PostingsFormat(name=FSTPulsing41)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=hu_HU, timezone=WET
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=90386648,total=185597952
[junit4] 2> NOTE: All tests run in this JVM: [SolrExampleStreamingBinaryTest, ModifiableSolrParamsTest, TestDocumentObjectBinder, LargeVolumeEmbeddedTest, ContentStreamTest, TestUpdateRequestCodec, LargeVolumeJettyTest, SolrExceptionTest, TermsResponseTest, DocumentAnalysisResponseTest, TestEmbeddedSolrServer, TestCoreAdmin, CloudSolrServerTest]
[junit4] Completed on J0 in 102.62s, 2 tests, 1 error <<< FAILURES!
[...truncated 94 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error
Total time: 64 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b10) -
Build # 9912 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9912/
Java: 64bit/jdk1.7.0_60-ea-b10 -XX:+UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58118 within 45000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58118 within 45000 ms
at __randomizedtesting.SeedInfo.seed([6E16AA3B66C6C381:EFF024231199A3BD]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58118 within 45000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
... 45 more
Build Log:
[...truncated 11605 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 28328 T84 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 28328 T84 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 28330 T84 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 28334 T84 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395856000772
[junit4] 2> 28336 T84 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 28338 T85 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 28438 T84 oasc.ZkTestServer.run start zk server on port:58118
[junit4] 2> 28468 T84 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 93709 T88 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 65221ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 93710 T84 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 93727 T86 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144ff8047320000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 93728 T86 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 93729 T86 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 93730 T86 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 93731 T84 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58118 58118
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=6E16AA3B66C6C381 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_BO -Dtests.timezone=Atlantic/Faroe -Dtests.file.encoding=UTF-8
[junit4] ERROR 65.5s J0 | CloudSolrServerTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58118 within 45000 ms
[junit4] > at __randomizedtesting.SeedInfo.seed([6E16AA3B66C6C381:EFF024231199A3BD]:0)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:58118 within 45000 ms
[junit4] > at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
[junit4] > ... 45 more
[junit4] 2> 93842 T84 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395856066279
[junit4] 2> 93843 T84 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 93843 T92 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 93943 T84 oasc.ZkTestServer.run start zk server on port:53758
[junit4] 2> 93944 T84 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 93956 T98 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@182d4c7f name:ZooKeeperConnection Watcher:127.0.0.1:53758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 93956 T84 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 93959 T84 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 93981 T84 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 93982 T100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b9389ee name:ZooKeeperConnection Watcher:127.0.0.1:53758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 93983 T84 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 93986 T84 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 93991 T84 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 93994 T84 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 94002 T84 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 94006 T84 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 94007 T84 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 94017 T84 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 94018 T84 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 94034 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 94035 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 94035 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 94035 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 94036 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 94036 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 94037 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 94037 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 94038 T84 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 94049 T84 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 94050 T101 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 94050 T101 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
[junit4] 2> at sun.nio.ch.Net.connect0(Native Method)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:465)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:457)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 95152 T101 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 95254 T84 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 95256 T84 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53758 53758
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 95422 T84 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 67101 T83 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=285), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=285)), sim=DefaultSimilarity, locale=es_BO, timezone=Atlantic/Faroe
[junit4] 2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=119970504,total=219152384
[junit4] 2> NOTE: All tests run in this JVM: [SolrExampleStreamingBinaryTest, ModifiableSolrParamsTest, DocumentAnalysisResponseTest, ClientUtilsTest, SolrPingTest, TestDocumentObjectBinder, LargeVolumeJettyTest, AnlysisResponseBaseTest, CloudSolrServerTest]
[junit4] Completed on J0 in 67.38s, 2 tests, 1 error <<< FAILURES!
[...truncated 97 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error
Total time: 52 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b10 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b10) -
Build # 9911 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9911/
Java: 64bit/jdk1.7.0_60-ea-b10 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([CDB667E8A1E7CCF3]:0)
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([CDB667E8A1E7CCF3]:0)
Build Log:
[...truncated 11371 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 739012 T4588 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 739012 T4588 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /qdw/
[junit4] 2> 739016 T4588 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395831395260
[junit4] 2> 739016 T4588 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 739017 T4589 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 739117 T4588 oasc.ZkTestServer.run start zk server on port:59894
[junit4] 2> 739118 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 739128 T4595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c38f7ad name:ZooKeeperConnection Watcher:127.0.0.1:59894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 739128 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 739128 T4588 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 739132 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 739133 T4597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73a41d1b name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 739134 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 739134 T4588 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 739136 T4588 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 739137 T4588 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 739138 T4588 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 739139 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 739140 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 739142 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 739142 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 739144 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 739145 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 739146 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 739146 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 739149 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 739149 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 739150 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 739151 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 739152 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 739152 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 739154 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 739155 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 739156 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 739156 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 739158 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 739158 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 739160 T4588 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 739161 T4588 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 739163 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 739164 T4599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7733ee56 name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 739164 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 739374 T4588 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 739377 T4588 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 739380 T4588 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60275
[junit4] 2> 739382 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 739382 T4588 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 739382 T4588 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409
[junit4] 2> 739383 T4588 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/'
[junit4] 2> 739395 T4588 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/solr.xml
[junit4] 2> 739426 T4588 oasc.CoreContainer.<init> New CoreContainer 1055075769
[junit4] 2> 739427 T4588 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/]
[junit4] 2> 739427 T4588 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 739428 T4588 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 739428 T4588 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 739428 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 739428 T4588 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 739429 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 739429 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 739429 T4588 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 739429 T4588 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 739431 T4588 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 739431 T4588 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 739432 T4588 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 739432 T4588 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59894/solr
[junit4] 2> 739432 T4588 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 739433 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 739434 T4610 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5af6945c name:ZooKeeperConnection Watcher:127.0.0.1:59894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 739434 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 739436 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 739437 T4612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47eadf35 name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 739437 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 739438 T4588 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 739440 T4588 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 739441 T4588 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 739443 T4588 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 739445 T4588 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 739446 T4588 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 739448 T4588 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60275_qdw
[junit4] 2> 739450 T4588 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60275_qdw
[junit4] 2> 739451 T4588 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 739453 T4588 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 739460 T4588 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60275_qdw
[junit4] 2> 739460 T4588 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 739462 T4588 oasc.Overseer.start Overseer (id=91477206319890436-127.0.0.1:60275_qdw-n_0000000000) starting
[junit4] 2> 739464 T4588 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 739471 T4614 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 739471 T4588 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 739471 T4614 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 739472 T4588 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 739473 T4588 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 739475 T4613 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 739478 T4615 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 739478 T4615 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 739479 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 739479 T4615 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 739480 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 739481 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:60275/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60275_qdw",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 739481 T4613 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 739481 T4613 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 739485 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 739485 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740480 T4615 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 740480 T4615 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 740481 T4615 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 740481 T4615 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 740482 T4615 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 740482 T4615 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/collection1/'
[junit4] 2> 740482 T4615 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/collection1/lib/.svn/' to classloader
[junit4] 2> 740483 T4615 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/collection1/lib/classes/' to classloader
[junit4] 2> 740483 T4615 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/collection1/lib/README' to classloader
[junit4] 2> 740508 T4615 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 740539 T4615 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 740541 T4615 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 740546 T4615 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 740816 T4615 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 740817 T4615 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 740818 T4615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 740821 T4615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 740836 T4615 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 740836 T4615 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 740836 T4615 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395831395409/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/control/data/
[junit4] 2> 740836 T4615 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41d1465c
[junit4] 2> 740837 T4615 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/control/data
[junit4] 2> 740837 T4615 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/control/data/index/
[junit4] 2> 740837 T4615 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395831395260/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 740837 T4615 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/control/data/index
[junit4] 2> 740838 T4615 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.314084743067576]
[junit4] 2> 740838 T4615 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3e63d565 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105725a4),segFN=segments_1,generation=1}
[junit4] 2> 740838 T4615 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 740842 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 740842 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 740842 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 740842 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 740842 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 740843 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 740843 T4615 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 740843 T4615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 740843 T4615 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 740844 T4615 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 740844 T4615 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 740844 T4615 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 740844 T4615 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 740845 T4615 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 740845 T4615 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 740845 T4615 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 740846 T4615 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 740848 T4615 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740849 T4615 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 740850 T4615 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 740850 T4615 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=74.349609375, floorSegmentMB=1.142578125, forceMergeDeletesPctAllowed=5.120953378894949, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18522701469733677
[junit4] 2> 740850 T4615 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3e63d565 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@105725a4),segFN=segments_1,generation=1}
[junit4] 2> 740851 T4615 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 740851 T4615 oass.SolrIndexSearcher.<init> Opening Searcher@4ada27b0[collection1] main
[junit4] 2> 740851 T4615 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 740852 T4615 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 740852 T4615 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 740852 T4615 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 740852 T4615 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 740852 T4615 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 740853 T4615 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 740853 T4615 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 740853 T4615 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 740853 T4615 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 740855 T4616 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ada27b0[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 740856 T4615 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 740865 T4619 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60275/qdw collection:control_collection shard:shard1
[junit4] 2> 740865 T4588 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 740866 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 740866 T4619 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 740869 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 740870 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 740870 T4621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c85b274 name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 740871 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 740871 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740871 T4588 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 740872 T4619 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 740872 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2104 name=collection1 org.apache.solr.core.SolrCore@12a7a3f1 url=https://127.0.0.1:60275/qdw/collection1 node=127.0.0.1:60275_qdw C2104_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:60275/qdw, core=collection1, node_name=127.0.0.1:60275_qdw}
[junit4] 2> 740872 T4619 C2104 P60275 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60275/qdw/collection1/
[junit4] 2> 740872 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 740872 T4619 C2104 P60275 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 740873 T4619 C2104 P60275 oasc.SyncStrategy.syncToMe https://127.0.0.1:60275/qdw/collection1/ has no replicas
[junit4] 2> 740873 T4619 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60275/qdw/collection1/ shard1
[junit4] 2> 740873 T4619 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 740873 T4588 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 740874 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740878 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740881 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740982 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740982 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 741029 T4619 oasc.ZkController.register We are https://127.0.0.1:60275/qdw/collection1/ and leader is https://127.0.0.1:60275/qdw/collection1/
[junit4] 2> 741029 T4619 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60275/qdw
[junit4] 2> 741029 T4619 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 741029 T4619 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 741029 T4619 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 741030 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741030 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741030 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741030 T4619 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 741031 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 741032 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:60275/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60275_qdw",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 741033 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741092 T4588 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 741092 T4588 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 741095 T4588 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 741096 T4588 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:40052
[junit4] 2> 741098 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 741098 T4588 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 741098 T4588 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118
[junit4] 2> 741099 T4588 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/'
[junit4] 2> 741111 T4588 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/solr.xml
[junit4] 2> 741137 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 741137 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 741146 T4588 oasc.CoreContainer.<init> New CoreContainer 58481025
[junit4] 2> 741146 T4588 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/]
[junit4] 2> 741147 T4588 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 741147 T4588 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 741148 T4588 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 741148 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 741148 T4588 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 741148 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 741148 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 741149 T4588 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 741149 T4588 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 741151 T4588 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 741151 T4588 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 741151 T4588 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 741152 T4588 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59894/solr
[junit4] 2> 741152 T4588 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 741152 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741155 T4632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10ad14a8 name:ZooKeeperConnection Watcher:127.0.0.1:59894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741155 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741156 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741157 T4634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@170e8785 name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741158 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741161 T4588 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 742163 T4588 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40052_qdw
[junit4] 2> 742164 T4588 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40052_qdw
[junit4] 2> 742166 T4621 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 742166 T4634 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 742166 T4612 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 742171 T4635 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 742171 T4635 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 742172 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742172 T4635 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 742172 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742172 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742173 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 742173 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:40052/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40052_qdw",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 742173 T4613 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 742173 T4613 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 742179 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742281 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 742281 T4634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 742281 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743172 T4635 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 743172 T4635 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 743173 T4635 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 743173 T4635 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 743174 T4635 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 743174 T4635 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/collection1/'
[junit4] 2> 743174 T4635 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/collection1/lib/.svn/' to classloader
[junit4] 2> 743175 T4635 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/collection1/lib/classes/' to classloader
[junit4] 2> 743175 T4635 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/collection1/lib/README' to classloader
[junit4] 2> 743212 T4635 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 743242 T4635 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 743243 T4635 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 743249 T4635 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 743497 T4635 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 743499 T4635 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 743500 T4635 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 743503 T4635 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 743527 T4635 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 743528 T4635 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 743528 T4635 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395831397118/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty1/
[junit4] 2> 743528 T4635 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41d1465c
[junit4] 2> 743529 T4635 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty1
[junit4] 2> 743529 T4635 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty1/index/
[junit4] 2> 743529 T4635 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 743529 T4635 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty1/index
[junit4] 2> 743530 T4635 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.314084743067576]
[junit4] 2> 743530 T4635 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@683d8bb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@23e772d0),segFN=segments_1,generation=1}
[junit4] 2> 743530 T4635 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 743535 T4635 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 743535 T4635 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 743535 T4635 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 743535 T4635 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 743536 T4635 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 743536 T4635 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 743536 T4635 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 743537 T4635 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 743537 T4635 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 743537 T4635 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 743538 T4635 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 743538 T4635 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 743538 T4635 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 743538 T4635 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 743539 T4635 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 743539 T4635 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 743540 T4635 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 743542 T4635 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 743544 T4635 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 743544 T4635 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 743544 T4635 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=74.349609375, floorSegmentMB=1.142578125, forceMergeDeletesPctAllowed=5.120953378894949, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18522701469733677
[junit4] 2> 743545 T4635 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@683d8bb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@23e772d0),segFN=segments_1,generation=1}
[junit4] 2> 743545 T4635 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 743545 T4635 oass.SolrIndexSearcher.<init> Opening Searcher@114c0ac[collection1] main
[junit4] 2> 743545 T4635 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 743546 T4635 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 743546 T4635 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 743547 T4635 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 743547 T4635 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 743547 T4635 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 743547 T4635 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 743548 T4635 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 743548 T4635 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 743548 T4635 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 743550 T4636 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@114c0ac[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 743552 T4635 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 743553 T4639 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:40052/qdw collection:collection1 shard:shard1
[junit4] 2> 743553 T4588 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 743553 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 743556 T4639 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 743561 T4639 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 743562 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743562 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743563 T4639 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 743563 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743563 T4639 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2105 name=collection1 org.apache.solr.core.SolrCore@2863cd79 url=https://127.0.0.1:40052/qdw/collection1 node=127.0.0.1:40052_qdw C2105_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:40052/qdw, core=collection1, node_name=127.0.0.1:40052_qdw}
[junit4] 2> 743563 T4639 C2105 P40052 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:40052/qdw/collection1/
[junit4] 2> 743563 T4639 C2105 P40052 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 743563 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743563 T4639 C2105 P40052 oasc.SyncStrategy.syncToMe https://127.0.0.1:40052/qdw/collection1/ has no replicas
[junit4] 2> 743564 T4639 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:40052/qdw/collection1/ shard1
[junit4] 2> 743564 T4639 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 743565 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743567 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743569 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743671 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743671 T4634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743671 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743717 T4639 oasc.ZkController.register We are https://127.0.0.1:40052/qdw/collection1/ and leader is https://127.0.0.1:40052/qdw/collection1/
[junit4] 2> 743718 T4639 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:40052/qdw
[junit4] 2> 743718 T4639 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 743718 T4639 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 743718 T4639 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 743719 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743719 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743719 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743719 T4639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743720 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743721 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:40052/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40052_qdw",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 743723 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743768 T4588 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 743769 T4588 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 743771 T4588 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 743772 T4588 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58364
[junit4] 2> 743774 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 743775 T4588 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 743775 T4588 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799
[junit4] 2> 743775 T4588 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/'
[junit4] 2> 743787 T4588 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/solr.xml
[junit4] 2> 743820 T4588 oasc.CoreContainer.<init> New CoreContainer 864195439
[junit4] 2> 743820 T4588 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/]
[junit4] 2> 743821 T4588 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 743821 T4588 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 743821 T4588 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 743821 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 743821 T4588 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 743822 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 743822 T4588 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 743822 T4588 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 743822 T4588 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 743824 T4634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743824 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743824 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743824 T4588 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 743825 T4588 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 743825 T4588 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 743826 T4588 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59894/solr
[junit4] 2> 743826 T4588 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 743827 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 743828 T4650 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13710722 name:ZooKeeperConnection Watcher:127.0.0.1:59894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 743828 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 743829 T4588 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 743830 T4652 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@155efc72 name:ZooKeeperConnection Watcher:127.0.0.1:59894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 743830 T4588 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 743835 T4588 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 744836 T4588 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58364_qdw
[junit4] 2> 744838 T4588 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58364_qdw
[junit4] 2> 744839 T4634 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 744839 T4652 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 744839 T4612 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 744839 T4621 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 744845 T4653 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 744845 T4653 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 744846 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744846 T4653 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 744846 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744847 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744848 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744849 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:58364/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58364_qdw",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 744849 T4613 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 744849 T4613 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 744856 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744957 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 744957 T4634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 744957 T4652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 744957 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 745847 T4653 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 745847 T4653 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 745848 T4653 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 745848 T4653 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 745849 T4653 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 745849 T4653 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/collection1/'
[junit4] 2> 745849 T4653 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/collection1/lib/.svn/' to classloader
[junit4] 2> 745849 T4653 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/collection1/lib/classes/' to classloader
[junit4] 2> 745850 T4653 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/collection1/lib/README' to classloader
[junit4] 2> 745900 T4653 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 745939 T4653 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 745940 T4653 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 745946 T4653 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 746210 T4653 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 746212 T4653 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 746213 T4653 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 746216 T4653 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 746233 T4653 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 746233 T4653 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 746233 T4653 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395831399799/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty2/
[junit4] 2> 746233 T4653 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41d1465c
[junit4] 2> 746234 T4653 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty2
[junit4] 2> 746234 T4653 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty2/index/
[junit4] 2> 746235 T4653 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 746235 T4653 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395831395260/jetty2/index
[junit4] 2> 746235 T4653 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.314084743067576]
[junit4] 2> 746236 T4653 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@34dd709f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@60fa3b38),segFN=segments_1,generation=1}
[junit4] 2> 746237 T4653 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 746243 T4653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 746243 T4653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 746244 T4653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 746244 T4653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 746244 T4653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 746244 T4653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 746245 T4653 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 746245 T4653 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 746245 T4653 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 746246 T4653 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 746247 T4653 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 746247 T4653 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 746247 T4653 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 746248 T4653 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 746248 T4653 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 746249 T4653 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 746249 T4653 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 746252 T4653 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 746254 T4653 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 746254 T4653 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 746255 T4653 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=74.349609375, floorSegmentMB=1.142578125, forceMergeDeletesPctAllowed=5.120953378894949, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.18522701469733677
[junit4] 2> 746255 T4653 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@34dd709f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@60fa3b38),segFN=segments_1,generation=1}
[junit4] 2> 746255 T4653 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 746256 T4653 oass.SolrIndexSearcher.<init> Opening Searcher@72059540[collection1] main
[junit4] 2> 746256 T4653 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 746257 T4653 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 746257 T4653 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 746257 T4653 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 746257 T4653 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 746257 T4653 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 746258 T4653 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 746258 T4653 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 746258 T4653 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 746258 T4653 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 746261 T4654 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72059540[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 746263 T4653 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 746264 T4657 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58364/qdw collection:collection1 shard:shard1
[junit4] 2> 746264 T4588 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 746264 T4588 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 746266 T4588 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746267 T4657 oasc.ZkController.register We are https://127.0.0.1:58364/qdw/collection1/ and leader is https://127.0.0.1:40052/qdw/collection1/
[junit4] 2> 746267 T4657 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58364/qdw
[junit4] 2> 746268 T4657 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 746268 T4657 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C2106 name=collection1 org.apache.solr.core.SolrCore@5d5ec64 url=https://127.0.0.1:58364/qdw/collection1 node=127.0.0.1:58364_qdw C2106_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:58364/qdw, core=collection1, node_name=127.0.0.1:58364_qdw}
[junit4] 2> 746268 T4659 C2106 P58364 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 746268 T4657 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746270 T4659 C2106 P58364 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 746270 T4659 C2106 P58364 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 746271 T4659 C2106 P58364 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 746272 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746272 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746272 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746273 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746273 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"https://127.0.0.1:58364/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58364_qdw",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 746277 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746330 T4628 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 746331 T4628 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746332 T4628 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> ASYNC NEW_CORE C2107 name=collection1 org.apache.solr.core.SolrCore@12a7a3f1 url=https://127.0.0.1:60275/qdw/collection1 node=127.0.0.1:60275_qdw C2107_STATE=coll:control_collection core:collection1 props:{state=active, base_url=https://127.0.0.1:60275/qdw, core=collection1, node_name=127.0.0.1:60275_qdw, leader=true}
[junit4] 2> 746340 T4608 C2107 P60275 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463635308793823232)]} 0 2
[junit4] 2> 746344 T4607 C2107 P60275 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463635308799066112)]} 0 1
[junit4] 2> 746381 T4621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746381 T4652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746381 T4634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746381 T4612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> C2106_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:58364/qdw, core=collection1, node_name=127.0.0.1:58364_qdw}
[junit4] 2> 746460 T4647 C2106 P58364 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={distrib.from=https://127.0.0.1:40052/qdw/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[2-0 (1463635308856737792)]} 0 2
[junit4] 2> ASYNC NEW_CORE C2108 name=collection1 org.apache.solr.core.SolrCore@2863cd79 url=https://127.0.0.1:40052/qdw/collection1 node=127.0.0.1:40052_qdw C2108_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:40052/qdw, core=collection1, node_name=127.0.0.1:40052_qdw, leader=true}
[junit4] 2> 746460 T4630 C2108 P40052 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin} {add=[2-0 (1463635308856737792)]} 0 62
[junit4] 2> 746467 T4645 C2106 P58364 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={distrib.from=https://127.0.0.1:40052/qdw/collection1/&version=2&update.distrib=FROMLEADER&wt=javabin} {add=[1-0 (1463635308866174976)]} 0 1
[junit4] 2> 746468 T4627 C2108 P40052 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin} {add=[1-0 (1463635308866174976)]} 0 61
[junit4] 2> 746468 T4588 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58364
[junit4] 2> 746469 T4588 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=864195439
[junit4] 2> 746470 T4588 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
[junit4] 2> 746471 T4659 C2106 P58364 oasc.SolrException.log ERROR Error while trying to recover. core=collection1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:40052/qdw
[junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
[junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:611)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:370)
[junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
[junit4] 2> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:40052/qdw
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:565)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer$1.call(HttpSolrServer.java:244)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer$1.call(HttpSolrServer.java:240)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> Caused by: java.net.SocketException: Socket closed
[junit4] 2> at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:152)
[junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:122)
[junit4] 2> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
[junit4] 2> at sun.security.ssl.InputRecord.read(InputRecord.java:480)
[junit4] 2> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
[junit4] 2> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
[junit4] 2> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
[junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
[junit4] 2> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
[junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
[junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
[junit4] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
[junit4] 2> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
[junit4] 2> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
[junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
[junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
[junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
[junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
[junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:451)
[junit4] 2> ... 6 more
[junit4] 2>
[junit4] 2> 746471 T4659 C2106 P58364 oasc.RecoveryStrategy.doRecovery ERROR Recovery failed - trying again... (0) core=collection1
[junit4] 2> 746472 T4659 C2106 P58364 oasc.SolrException.log ERROR Recovery failed - interrupted. core=collection1
[junit4] 2> 746472 T4659 C2106 P58364 oasc.SolrException.log ERROR Recovery failed - I give up. core=collection1
[junit4] 2> 746472 T4659 C2106 P58364 oasc.ZkController.publish publishing core=collection1 state=recovery_failed collection=collection1
[junit4] 2> 746472 T4659 C2106 P58364 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 746472 T4607 C2107 P60275 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463635308934332416)]} 0 0
[junit4] 2> 746473 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746474 T4659 C2106 P58364 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
[junit4] 2> 746474 T4613 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746475 T4659 C2106 P58364 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4] 2> 746474 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746475 T4588 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 746475 T4588 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 746475 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746476 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovery_failed",
[junit4] 2> "base_url":"https://127.0.0.1:58364/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58364_qdw",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 746478 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746478 T4588 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5d5ec64
[junit4] 2> 746481 T4605 C2107 P60275 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463635308942721024)]} 0 1
[junit4] 2> 746481 T4613 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:58364/qdw",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58364_qdw",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 746484 T4612 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746484 T4663 oasu.StreamingSolrServers$1.handleError ERROR error org.apache.solr.common.SolrException: Can not find: /qdw/collection1/update
[junit4] 2>
[junit4] 2>
[junit4] 2>
[junit4] 2> request: https://127.0.0.1:58364/qdw/collection1/update?update.distrib=FROMLEADER&distrib.from=https%3A%2F%2F127.0.0.1%3A40052%2Fqdw%2Fcollection1%2F&wt=javabin&version=2
[junit4] 2> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> ASYNC NEW_CORE C2109 name=collection1 org.apache.solr.core.SolrCore@2863cd79 url=https://127.0.0.1:40052/qdw/collection1 node=127.0.0.1:40052_qdw C2109_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:40052/qdw, core=collection1, node_name=127.0.0.1:40052_qdw, leader=true}
[junit4] 2> 746495 T4630 C2109 P40052 oasup.DistributedUpdateProcessor.doFinish WARN Error sending update org.apache.solr.common.SolrException: Can not find: /qdw/collection1/update
[junit4] 2>
[junit4] 2>
[junit4] 2>
[junit4] 2> request: https://127.0.0.1:58364/qdw/collection1/update?update.distrib=FROMLEADER&distrib.from=https%3A%2F%2F127.0.0.1%3A40052%2Fqdw%2Fcollection1%2F&wt=javabin&version=2
[junit4] 2> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 746496 T4663 oasup.DistributedUpdateProcessor$1.run try and ask https://127.0.0.1:58364/qdw to recover
[junit4] 2> 746496 T4630 C2109 P40052 oasup.LogUpdateProcessor.finish [collection1] webapp=/qdw path=/update params={version=2&wt=javabin} {add=[2-1 (1463635308939575296)]} 0 19
[junit4] 2> 746497 T4588 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=2,adds=2,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 746497 T4588 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 746497 T4588 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 746497 T4588 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> ASYNC NEW_CORE C2110 name=collection1 org.apache.solr.core.SolrCore@5d5ec64 url=https://127.0.0.1:58364/qdw/collection1 node=127.0.0.1:58364_qdw C2110_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=https://127.0.0.1:58364/qdw, core=collection1, node_name=127.0.0.1:58364_qdw}
[junit4] 2> 746500 T4588 C2110 P58364 oasc
[...truncated too long message...]
QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 14) Thread[id=4680, name=searcherExecutor-2062-thread-1, state=WAITING, group=TGRP-RecoveryZkTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 15) Thread[id=4668, name=qtp522691791-4668 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 16) Thread[id=4639, name=coreZkRegister-2042-thread-1, state=WAITING, group=TGRP-RecoveryZkTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 17) Thread[id=4633, name=TEST-RecoveryZkTest.testDistribSearch-seed#[CDB667E8A1E7CCF3]-SendThread(127.0.0.1:59894), state=TIMED_WAITING, group=TGRP-RecoveryZkTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2> 18) Thread[id=4624, name=qtp1550174314-4624 Selector1, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 19) Thread[id=4669, name=qtp522691791-4669 Acceptor0 SslSelectChannelConnector@127.0.0.1:58364, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2> NOTE: test params are: codec=Lucene46: {id=MockFixedIntBlock(blockSize=1830), a_si=MockVariableIntBlock(baseBlockSize=125), rnd_b=MockFixedIntBlock(blockSize=1830), _version_=PostingsFormat(name=FSTPulsing41), a_t=PostingsFormat(name=FSTPulsing41)}, docValues:{}, sim=DefaultSimilarity, locale=hr_HR, timezone=Pacific/Port_Moresby
[junit4] 2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=25,free=215069208,total=334983168
[junit4] 2> NOTE: All tests run in this JVM: [TestNoOpRegenerator, SolrInfoMBeanTest, TestFreeTextSuggestions, TestComplexPhraseQParserPlugin, StatelessScriptUpdateProcessorFactoryTest, SimpleFacetsTest, TestSolrXml, TestFileDictionaryLookup, TestDistribDocBasedVersion, CachingDirectoryFactoryTest, TestFieldTypeResource, UnloadDistributedZkTest, BasicDistributedZkTest, DistributedQueryComponentOptimizationTest, TestStressVersions, TermsComponentTest, SuggesterTest, TestLRUCache, DocValuesTest, TestLuceneMatchVersion, UpdateRequestProcessorFactoryTest, TestCollationFieldDocValues, CSVRequestHandlerTest, BinaryUpdateRequestHandlerTest, JsonLoaderTest, MoreLikeThisHandlerTest, TestGroupingSearch, RangeFacetTest, CoreMergeIndexesAdminHandlerTest, ShardRoutingTest, DistributedQueryComponentCustomSortTest, TestSearcherReuse, TestSurroundQueryParser, IndexSchemaTest, FunctionTest, LukeRequestHandlerTest, ExpressionTest, LoggingHandlerTest, TestFieldSortValues, DocumentBuilderTest, DOMUtilTest, UpdateParamsTest, MBeansHandlerTest, EchoParamsTest, DistributedTermsComponentTest, ResourceLoaderTest, SyncSliceTest, TestLMDirichletSimilarityFactory, TestReplicationHandler, TestManagedResourceStorage, FieldMutatingUpdateProcessorTest, TestRemoteStreaming, TriLevelCompositeIdRoutingTest, TestPostingsSolrHighlighter, TestFunctionQuery, TestSuggestSpellingConverter, ReturnFieldsTest, SpellPossibilityIteratorTest, DistributedQueryElevationComponentTest, StressHdfsTest, TestDocumentBuilder, SampleTest, SliceStateUpdateTest, SliceStateTest, HdfsBasicDistributedZkTest, TestJmxMonitoredMap, PeerSyncTest, LeaderElectionTest, TestFaceting, ClusterStateTest, SynonymTokenizerTest, SpatialFilterTest, ClusterStateUpdateTest, SOLR749Test, TestAddFieldRealTimeGet, TestPerFieldSimilarity, TestDocSet, SchemaVersionSpecificBehaviorTest, BadComponentTest, TestSolrDeletionPolicy1, CacheHeaderTest, SearchHandlerTest, OverseerTest, PolyFieldTest, CircularListTest, TestRealTimeGet, OverseerRolesTest, SolrTestCaseJ4Test, TestCharFilters, ZkCLITest, DeleteInactiveReplicaTest, QueryParsingTest, AliasIntegrationTest, TestSolrXMLSerializer, RecoveryZkTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=CDB667E8A1E7CCF3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hr_HR -Dtests.timezone=Pacific/Port_Moresby -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([CDB667E8A1E7CCF3]:0)
[junit4] Completed on J0 in 7223.32s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 393 suites, 1651 tests, 1 suite-level error, 1 error, 49 ignored (28 assumptions)
Total time: 165 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b10 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build #
9910 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9910/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([FA4A880D969CF294:7BAC0615E1C392A8]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:256)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 11621 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 42717 T147 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=6 numCloses=6
[junit4] 2> 42718 T147 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 42719 T147 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /asj/h
[junit4] 2> 42721 T147 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 42726 T147 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395820582019
[junit4] 2> 42730 T147 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 42733 T148 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 42832 T147 oasc.ZkTestServer.run start zk server on port:39266
[junit4] 2> 42867 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69316 T151 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 26426ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 69338 T149 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fd63d4b50000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 69341 T149 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 69344 T154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1114305 name:ZooKeeperConnection Watcher:127.0.0.1:39266 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69345 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69347 T147 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 69376 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69379 T156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ff834 name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69379 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69382 T147 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 69386 T147 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 69389 T147 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 69391 T147 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 69395 T147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 69395 T147 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 69437 T147 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 69438 T147 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 69443 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 69443 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 69443 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 69444 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 69444 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 69444 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 69444 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 69445 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 69445 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 69449 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69451 T158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@242d75 name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69451 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69543 T147 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 69694 T147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 69697 T147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50566
[junit4] 2> 69699 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 69700 T147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 69700 T147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746
[junit4] 2> 69700 T147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746/'
[junit4] 2> 69712 T147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746/solr.xml
[junit4] 2> 69779 T147 oasc.CoreContainer.<init> New CoreContainer 17709234
[junit4] 2> 69780 T147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746/]
[junit4] 2> 69781 T147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 69782 T147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 69782 T147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 69782 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 69783 T147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 69783 T147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 69784 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 69784 T147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 69785 T147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 69789 T147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 69789 T147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 69789 T147 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 69790 T147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39266/solr
[junit4] 2> 69797 T147 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 69798 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69802 T169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1128e87 name:ZooKeeperConnection Watcher:127.0.0.1:39266 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69802 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69808 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69810 T171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@166de77 name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69810 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69817 T147 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 69823 T147 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 69834 T147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 69839 T147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 69846 T147 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 69858 T147 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 69861 T147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50566_asj%2Fh
[junit4] 2> 69864 T147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50566_asj%2Fh
[junit4] 2> 69929 T147 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 69932 T147 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 69945 T147 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50566_asj%2Fh
[junit4] 2> 69946 T147 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 69951 T147 oasc.Overseer.start Overseer (id=91476497666473990-127.0.0.1:50566_asj%2Fh-n_0000000000) starting
[junit4] 2> 69972 T147 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 70043 T173 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 70043 T147 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 70044 T173 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 70053 T147 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 70057 T147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 70068 T172 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 70077 T174 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 70077 T174 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 70081 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 70081 T174 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 70085 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 70086 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50566/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50566_asj%2Fh",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 70086 T172 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 70092 T172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 70101 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 70111 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71082 T174 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 71082 T174 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 71083 T174 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 71083 T174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 71085 T174 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 71085 T174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746/collection1/'
[junit4] 2> 71132 T174 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 71181 T174 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 71184 T174 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 71195 T174 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 71524 T174 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 71527 T174 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 71528 T174 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 71528 T174 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 71529 T174 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 71529 T174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 71529 T174 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1395820608746/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/
[junit4] 2> 71529 T174 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 71530 T174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data
[junit4] 2> 71530 T174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/index/
[junit4] 2> 71530 T174 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 71530 T174 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/index
[junit4] 2> 71532 T174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@a98388 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/index),segFN=segments_1,generation=1}
[junit4] 2> 71532 T174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 71533 T174 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 71533 T174 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 71533 T174 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 71533 T174 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 71534 T174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 71534 T174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 71534 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 71534 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 71536 T174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 71541 T174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 71541 T174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 71542 T174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@a98388 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/control/data/index),segFN=segments_1,generation=1}
[junit4] 2> 71542 T174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 71543 T174 oass.SolrIndexSearcher.<init> Opening Searcher@1c6bce9[collection1] main
[junit4] 2> 71543 T174 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 71544 T174 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 71544 T174 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 71547 T174 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 71547 T174 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 71547 T174 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 71548 T174 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 71548 T174 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 71548 T174 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 71548 T174 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 71549 T175 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c6bce9[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 71549 T174 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 71550 T178 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50566/asj/h collection:control_collection shard:shard1
[junit4] 2> 71551 T147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 71551 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 71557 T178 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 71562 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 71564 T180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d6ea name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 71565 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 71565 T178 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 71566 T147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 71568 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71569 T178 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 71569 T178 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 71570 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C43 name=collection1 org.apache.solr.core.SolrCore@52f59 url=https://127.0.0.1:50566/asj/h/collection1 node=127.0.0.1:50566_asj%2Fh C43_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50566/asj/h, core=collection1, node_name=127.0.0.1:50566_asj%2Fh}
[junit4] 2> 71570 T178 C43 P50566 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50566/asj/h/collection1/
[junit4] 2> 71571 T178 C43 P50566 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 71571 T178 C43 P50566 oasc.SyncStrategy.syncToMe https://127.0.0.1:50566/asj/h/collection1/ has no replicas
[junit4] 2> 71571 T178 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50566/asj/h/collection1/ shard1
[junit4] 2> 71573 T178 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 71681 T147 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 71682 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71683 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71684 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71693 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 71697 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71736 T147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 71737 T147 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 71743 T147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 71746 T147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37321
[junit4] 2> 71749 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 71750 T147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 71750 T147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974
[junit4] 2> 71751 T147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974/'
[junit4] 2> 71767 T147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974/solr.xml
[junit4] 2> 71800 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71800 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71820 T147 oasc.CoreContainer.<init> New CoreContainer 20483608
[junit4] 2> 71820 T147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974/]
[junit4] 2> 71821 T147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 71822 T147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 71822 T147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 71822 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 71823 T147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 71823 T147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 71823 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 71824 T147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 71824 T147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 71828 T147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 71828 T147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 71829 T147 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 71829 T147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39266/solr
[junit4] 2> 71830 T147 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 71830 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 71833 T191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d40c2 name:ZooKeeperConnection Watcher:127.0.0.1:39266 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 71833 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 71836 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 71837 T193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13c354f name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 71837 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 71845 T178 oasc.ZkController.register We are https://127.0.0.1:50566/asj/h/collection1/ and leader is https://127.0.0.1:50566/asj/h/collection1/
[junit4] 2> 71845 T178 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50566/asj/h
[junit4] 2> 71845 T178 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 71846 T178 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 71846 T178 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 71847 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71847 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71847 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71848 T178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 71849 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 71851 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50566/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50566_asj%2Fh",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 71854 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 71856 T147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 71961 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71961 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71961 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 72860 T147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37321_asj%2Fh
[junit4] 2> 72861 T147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37321_asj%2Fh
[junit4] 2> 72864 T171 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 72864 T180 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 72864 T193 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 72873 T194 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 72873 T194 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 72874 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72874 T194 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 72874 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72874 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72875 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 72876 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37321/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37321_asj%2Fh",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 72877 T172 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 72877 T172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 72880 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72984 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 72984 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 72984 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 73874 T194 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 73875 T194 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 73876 T194 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 73876 T194 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 73878 T194 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 73878 T194 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974/collection1/'
[junit4] 2> 73942 T194 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 73983 T194 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 73986 T194 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 73999 T194 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 74325 T194 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 74327 T194 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 74329 T194 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 74329 T194 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 74329 T194 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 74330 T194 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 74330 T194 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1395820610974/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/
[junit4] 2> 74330 T194 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 74330 T194 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1
[junit4] 2> 74331 T194 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/index/
[junit4] 2> 74331 T194 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 74331 T194 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/index
[junit4] 2> 74332 T194 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d27722 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/index),segFN=segments_1,generation=1}
[junit4] 2> 74332 T194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 74333 T194 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 74333 T194 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 74334 T194 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 74334 T194 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 74334 T194 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 74334 T194 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 74334 T194 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 74335 T194 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 74336 T194 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74339 T194 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 74339 T194 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 74340 T194 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d27722 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty1/index),segFN=segments_1,generation=1}
[junit4] 2> 74340 T194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 74341 T194 oass.SolrIndexSearcher.<init> Opening Searcher@85be0[collection1] main
[junit4] 2> 74341 T194 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 74342 T194 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 74342 T194 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 74343 T194 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 74343 T194 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 74343 T194 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 74344 T194 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 74344 T194 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 74345 T194 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 74345 T194 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 74346 T195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@85be0[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 74346 T194 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 74347 T198 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:37321/asj/h collection:collection1 shard:shard2
[junit4] 2> 74347 T147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 74348 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 74348 T198 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 74358 T198 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 74360 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74360 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74360 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74362 T198 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 74362 T198 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C44 name=collection1 org.apache.solr.core.SolrCore@bcc3e9 url=https://127.0.0.1:37321/asj/h/collection1 node=127.0.0.1:37321_asj%2Fh C44_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:37321/asj/h, core=collection1, node_name=127.0.0.1:37321_asj%2Fh}
[junit4] 2> 74362 T198 C44 P37321 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:37321/asj/h/collection1/
[junit4] 2> 74362 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74362 T198 C44 P37321 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 74363 T198 C44 P37321 oasc.SyncStrategy.syncToMe https://127.0.0.1:37321/asj/h/collection1/ has no replicas
[junit4] 2> 74363 T198 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:37321/asj/h/collection1/ shard2
[junit4] 2> 74363 T198 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 74367 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74391 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74398 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74418 T147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 74418 T147 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 74423 T147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 74427 T147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:55446
[junit4] 2> 74430 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 74430 T147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 74431 T147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642
[junit4] 2> 74431 T147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642/'
[junit4] 2> 74451 T147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642/solr.xml
[junit4] 2> 74493 T147 oasc.CoreContainer.<init> New CoreContainer 4821315
[junit4] 2> 74494 T147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642/]
[junit4] 2> 74494 T147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 74495 T147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 74495 T147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 74495 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 74495 T147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 74496 T147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 74496 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 74496 T147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 74497 T147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 74500 T147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 74500 T147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 74500 T147 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 74500 T147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39266/solr
[junit4] 2> 74501 T147 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 74501 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74501 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74501 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74502 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 74504 T209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18778c6 name:ZooKeeperConnection Watcher:127.0.0.1:39266 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 74504 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 74507 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 74509 T211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1387da1 name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 74509 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 74519 T147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 74542 T198 oasc.ZkController.register We are https://127.0.0.1:37321/asj/h/collection1/ and leader is https://127.0.0.1:37321/asj/h/collection1/
[junit4] 2> 74543 T198 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:37321/asj/h
[junit4] 2> 74543 T198 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 74543 T198 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 74544 T198 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 74547 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74548 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74548 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74549 T198 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74552 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74554 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:37321/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37321_asj%2Fh",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 74559 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74665 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74665 T211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74665 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74665 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 75523 T147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55446_asj%2Fh
[junit4] 2> 75525 T147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55446_asj%2Fh
[junit4] 2> 75528 T211 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 75528 T193 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 75528 T171 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 75528 T180 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 75538 T212 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 75538 T212 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 75540 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 75540 T212 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 75540 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 75540 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 75542 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 75543 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:55446/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:55446_asj%2Fh",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 75543 T172 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 75543 T172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 75549 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 75653 T211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 75653 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 75653 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 75653 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 76540 T212 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 76540 T212 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 76541 T212 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 76541 T212 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 76543 T212 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 76543 T212 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642/collection1/'
[junit4] 2> 76576 T212 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 76611 T212 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 76612 T212 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 76621 T212 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 76940 T212 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 76942 T212 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 76944 T212 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 76944 T212 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 76944 T212 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 76945 T212 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 76945 T212 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1395820613642/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/
[junit4] 2> 76945 T212 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 76945 T212 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2
[junit4] 2> 76946 T212 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/index/
[junit4] 2> 76946 T212 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 76946 T212 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/index
[junit4] 2> 76947 T212 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@da6066 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/index),segFN=segments_1,generation=1}
[junit4] 2> 76947 T212 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 76948 T212 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 76949 T212 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 76949 T212 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 76949 T212 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 76949 T212 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 76949 T212 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 76950 T212 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 76950 T212 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 76952 T212 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 76954 T212 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 76954 T212 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 76955 T212 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@da6066 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty2/index),segFN=segments_1,generation=1}
[junit4] 2> 76955 T212 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 76955 T212 oass.SolrIndexSearcher.<init> Opening Searcher@1e98963[collection1] main
[junit4] 2> 76956 T212 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 76957 T212 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 76957 T212 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 76958 T212 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 76958 T212 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 76959 T212 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 76959 T212 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 76960 T212 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 76960 T212 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 76960 T212 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 76961 T213 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e98963[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 76961 T212 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 76962 T216 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:55446/asj/h collection:collection1 shard:shard1
[junit4] 2> 76962 T147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4] 2> 76963 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 76963 T216 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 76970 T216 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 76972 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76972 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76972 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76972 T216 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 76973 T216 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C45 name=collection1 org.apache.solr.core.SolrCore@19752e1 url=https://127.0.0.1:55446/asj/h/collection1 node=127.0.0.1:55446_asj%2Fh C45_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:55446/asj/h, core=collection1, node_name=127.0.0.1:55446_asj%2Fh}
[junit4] 2> 76973 T216 C45 P55446 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:55446/asj/h/collection1/
[junit4] 2> 76973 T216 C45 P55446 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 76973 T216 C45 P55446 oasc.SyncStrategy.syncToMe https://127.0.0.1:55446/asj/h/collection1/ has no replicas
[junit4] 2> 76973 T216 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:55446/asj/h/collection1/ shard1
[junit4] 2> 76974 T216 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 76976 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 76982 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76985 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76989 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 77036 T147 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 77037 T147 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 77041 T147 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 77044 T147 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38364
[junit4] 2> 77046 T147 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 77047 T147 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 77047 T147 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257
[junit4] 2> 77047 T147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257/'
[junit4] 2> 77058 T147 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257/solr.xml
[junit4] 2> 77093 T211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77093 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77093 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77094 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77108 T147 oasc.CoreContainer.<init> New CoreContainer 6721172
[junit4] 2> 77109 T147 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257/]
[junit4] 2> 77110 T147 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 77110 T147 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 77110 T147 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 77110 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 77111 T147 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 77111 T147 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 77111 T147 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 77111 T147 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 77112 T147 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 77120 T147 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 77120 T147 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 77120 T147 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 77121 T147 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39266/solr
[junit4] 2> 77121 T147 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 77122 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 77124 T227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16d56a5 name:ZooKeeperConnection Watcher:127.0.0.1:39266 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 77124 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 77127 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 77128 T229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f2fae4 name:ZooKeeperConnection Watcher:127.0.0.1:39266/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 77129 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 77136 T216 oasc.ZkController.register We are https://127.0.0.1:55446/asj/h/collection1/ and leader is https://127.0.0.1:55446/asj/h/collection1/
[junit4] 2> 77137 T216 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:55446/asj/h
[junit4] 2> 77137 T216 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 77137 T216 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 77137 T216 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 77139 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 77139 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 77140 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 77140 T216 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 77141 T147 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 77141 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 77142 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:55446/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:55446_asj%2Fh",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 77149 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 77255 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77255 T211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77255 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77255 T229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77255 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 78151 T147 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38364_asj%2Fh
[junit4] 2> 78152 T147 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38364_asj%2Fh
[junit4] 2> 78155 T229 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 78155 T171 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 78155 T193 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 78155 T211 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 78155 T180 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 78163 T230 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 78163 T230 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 78164 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 78164 T230 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 78164 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 78165 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 78166 T172 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 78167 T172 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38364/asj/h",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38364_asj%2Fh",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 78167 T172 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 78167 T172 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 78170 T171 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 78272 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 78273 T211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 78273 T171 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 78273 T229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 78273 T193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 79165 T230 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 79165 T230 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 79166 T230 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 79166 T230 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 79167 T230 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 79167 T230 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257/collection1/'
[junit4] 2> 79200 T230 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 79236 T230 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 79237 T230 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 79245 T230 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 79544 T230 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 79547 T230 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 79550 T230 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 79550 T230 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 79551 T230 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 79551 T230 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 79551 T230 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1395820616257/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/
[junit4] 2> 79552 T230 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 79552 T230 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3
[junit4] 2> 79552 T230 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index/
[junit4] 2> 79553 T230 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 79553 T230 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index
[junit4] 2> 79554 T230 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@15798c9 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index),segFN=segments_1,generation=1}
[junit4] 2> 79555 T230 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 79556 T230 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 79556 T230 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 79556 T230 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 79556 T230 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 79556 T230 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 79557 T230 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 79557 T230 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 79557 T230 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
[junit4] 2> 79559 T230 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 79561 T230 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 79561 T230 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 79562 T230 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@15798c9 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index),segFN=segments_1,generation=1}
[junit4] 2> 79562 T230 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 79562 T230 oass.SolrIndexSearcher.<init> Opening Searcher@1b31a82[collection1] main
[junit4] 2> 79563 T230 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 79565 T230 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 79565 T230 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 79566 T230 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 79566 T230 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 79566 T230 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 79567 T230 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 79567 T230 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 79568 T230 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 79568 T230 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 79569 T231 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b31a82[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 79569 T230 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 79570 T234 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38364/asj/h collection:collection1 shard:shard2
[junit4] 2> 79570 T147 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-so
[...truncated too long message...]
87445 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 87445 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 87446 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 87446 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 87446 T147 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 87448 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 87449 T260 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a20db3 name:ZooKeeperConnection Watcher:127.0.0.1:41238/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 87449 T147 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 87453 T147 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 87454 T261 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 87454 T261 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
[junit4] 2> at sun.nio.ch.Net.connect0(Native Method)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:465)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:457)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 88039 T171 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 88129 T193 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 88223 T211 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 88556 T261 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 88658 T147 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 88660 T147 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41238 41238
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 88769 T147 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> ASYNC NEW_CORE C58 name=collection1 org.apache.solr.core.SolrCore@3c934b url=https://127.0.0.1:38364/asj/h/collection1 node=127.0.0.1:38364_asj%2Fh C58_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:38364/asj/h, core=collection1, node_name=127.0.0.1:38364_asj%2Fh}
[junit4] 2> 89137 T229 C58 P38364 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 89141 T229 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 89141 T229 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@3c934b
[junit4] 2> 89142 T229 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=4}
[junit4] 2> 89142 T229 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 89142 T229 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 89142 T229 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 89143 T229 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 89144 T229 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 89144 T229 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3;done=false>>]
[junit4] 2> 89144 T229 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3
[junit4] 2> 89144 T229 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index;done=false>>]
[junit4] 2> 89144 T229 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1395820582019/jetty3/index
[junit4] 2> 89144 T229 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 89145 T229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89145 T229 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 89145 T229 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=15), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=15)), sim=DefaultSimilarity, locale=en_IN, timezone=Etc/GMT-14
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=25868576,total=65142784
[junit4] 2> NOTE: All tests run in this JVM: [SolrPingTest, LargeVolumeEmbeddedTest, DocumentAnalysisResponseTest, TestDocumentObjectBinder, SolrExampleEmbeddedTest, ContentStreamTest, LargeVolumeJettyTest, TestCoreAdmin, ModifiableSolrParamsTest, SolrExampleStreamingBinaryTest, ClientUtilsTest, TestEmbeddedSolrServer, CloudSolrServerTest]
[junit4] Completed on J0 in 47.10s, 2 tests, 1 failure <<< FAILURES!
[...truncated 76 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 failure
Total time: 59 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build #
9909 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9909/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59183 within 45000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59183 within 45000 ms
at __randomizedtesting.SeedInfo.seed([36C4E8B30829AAE1:B72266AB7F76CADD]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59183 within 45000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
... 45 more
Build Log:
[...truncated 11648 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 7312 T30 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 7313 T30 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ehw/
[junit4] 2> 7315 T30 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 7329 T30 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395812776743
[junit4] 2> 7333 T30 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 7336 T31 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 7435 T30 oasc.ZkTestServer.run start zk server on port:59183
[junit4] 2> 7485 T30 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 96462 T34 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 88957ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 96463 T30 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 96481 T32 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fcecbb690000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 96484 T32 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 96488 T32 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 96491 T30 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59183 59183
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=36C4E8B30829AAE1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl -Dtests.timezone=Etc/GMT-14 -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 94.4s J0 | CloudSolrServerTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59183 within 45000 ms
[junit4] > at __randomizedtesting.SeedInfo.seed([36C4E8B30829AAE1:B72266AB7F76CADD]:0)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59183 within 45000 ms
[junit4] > at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
[junit4] > ... 45 more
[junit4] 2> 101761 T30 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395812871175
[junit4] 2> 101763 T30 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 101763 T38 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 101863 T30 oasc.ZkTestServer.run start zk server on port:37438
[junit4] 2> 101865 T30 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 101932 T44 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131ef93 name:ZooKeeperConnection Watcher:127.0.0.1:37438 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 101932 T30 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 101934 T30 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 101959 T30 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 101961 T46 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70f34a name:ZooKeeperConnection Watcher:127.0.0.1:37438/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 101962 T30 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 101965 T30 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 101970 T30 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 101974 T30 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 101977 T30 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 101981 T30 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 101989 T30 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 101995 T30 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 101996 T30 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 102000 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 102000 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 102001 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 102001 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 102002 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 102002 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 102003 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 102003 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 102004 T30 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 102013 T30 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 102015 T47 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 102015 T47 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
[junit4] 2> at sun.nio.ch.Net.connect0(Native Method)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:465)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:457)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 103118 T47 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 103221 T30 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 103224 T30 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:37438 37438
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 103282 T30 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 95977 T29 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=nl, timezone=Etc/GMT-14
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=32860616,total=64880640
[junit4] 2> NOTE: All tests run in this JVM: [TestDocumentObjectBinder, TermsResponseTest, AnlysisResponseBaseTest, SolrParamTest, TestUpdateRequestCodec, TestRetryUtil, ClientUtilsTest, ModifiableSolrParamsTest, CloudSolrServerTest]
[junit4] Completed on J0 in 96.54s, 2 tests, 1 error <<< FAILURES!
[...truncated 106 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error
Total time: 57 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60-ea-b10) -
Build # 9908 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9908/
Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:56982 within 45000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:56982 within 45000 ms
at __randomizedtesting.SeedInfo.seed([F5D5A1676D0BDB97:74332F7F1A54BBAB]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:56982 within 45000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
... 45 more
Build Log:
[...truncated 11602 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 21317 T95 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 21317 T95 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 21319 T95 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 21322 T95 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395801260326
[junit4] 2> 21324 T95 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 21326 T96 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 21426 T95 oasc.ZkTestServer.run start zk server on port:56982
[junit4] 2> 21461 T95 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 114513 T99 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 93035ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 114514 T95 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 114522 T97 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fc3d01580000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 114536 T97 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 114536 T97 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 114537 T97 oazs.NIOServerCnxnFactory.run WARN Ignoring unexpected runtime exception java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 114538 T95 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56982 56982
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=F5D5A1676D0BDB97 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_MT -Dtests.timezone=Asia/Kamchatka -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 93.5s J0 | CloudSolrServerTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:56982 within 45000 ms
[junit4] > at __randomizedtesting.SeedInfo.seed([F5D5A1676D0BDB97:74332F7F1A54BBAB]:0)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:56982 within 45000 ms
[junit4] > at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
[junit4] > ... 45 more
[junit4] 2> 114782 T95 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395801353787
[junit4] 2> 114783 T95 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 114783 T103 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 114883 T95 oasc.ZkTestServer.run start zk server on port:40054
[junit4] 2> 114884 T95 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 114964 T109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dcdbf7 name:ZooKeeperConnection Watcher:127.0.0.1:40054 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 114964 T95 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 114966 T95 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 114988 T95 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 114991 T111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b27a08 name:ZooKeeperConnection Watcher:127.0.0.1:40054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 114991 T95 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 114996 T95 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 115000 T95 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 115002 T95 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 115005 T95 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 115009 T95 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 115019 T95 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 115026 T95 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 115027 T95 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 115037 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 115037 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 115037 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 115038 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 115038 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 115038 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 115039 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 115039 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 115039 T95 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 115048 T95 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 115059 T112 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 115059 T112 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
[junit4] 2> at sun.nio.ch.Net.connect0(Native Method)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:465)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:457)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 116351 T112 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 116454 T95 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 116455 T95 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40054 40054
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 116514 T95 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 95204 T94 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_MT, timezone=Asia/Kamchatka
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=38861720,total=192413696
[junit4] 2> NOTE: All tests run in this JVM: [TestDocumentObjectBinder, AnlysisResponseBaseTest, SolrExampleXMLTest, SolrExampleEmbeddedTest, ContentStreamTest, ModifiableSolrParamsTest, TestUpdateRequestCodec, LargeVolumeJettyTest, LargeVolumeBinaryJettyTest, CloudSolrServerTest]
[junit4] Completed on J0 in 95.74s, 2 tests, 1 error <<< FAILURES!
[...truncated 97 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error
Total time: 57 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b10 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0) - Build # 9907 -
Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9907/
Java: 32bit/jdk1.8.0 -server -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59004 within 45000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59004 within 45000 ms
at __randomizedtesting.SeedInfo.seed([10D89F0D312B7E32:913E111546741E0E]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:860)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59004 within 45000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
... 45 more
Build Log:
[...truncated 11648 lines...]
[junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4] 2> 28231 T130 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 28231 T130 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rea/
[junit4] 2> 28233 T130 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
[junit4] 2> 28237 T130 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395793252856
[junit4] 2> 28239 T130 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 28241 T131 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 28341 T130 oasc.ZkTestServer.run start zk server on port:59004
[junit4] 2> 28377 T130 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 87896 T134 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 59499ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 87897 T130 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 87915 T132 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144fbc2d23b0000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 87926 T130 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59004 59004
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=10D89F0D312B7E32 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_CR -Dtests.timezone=Asia/Dubai -Dtests.file.encoding=UTF-8
[junit4] ERROR 59.8s J0 | CloudSolrServerTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59004 within 45000 ms
[junit4] > at __randomizedtesting.SeedInfo.seed([10D89F0D312B7E32:913E111546741E0E]:0)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:91)
[junit4] > at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:85)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:77)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:201)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:95)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:59004 within 45000 ms
[junit4] > at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
[junit4] > ... 45 more
[junit4] 2> 88062 T130 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1395793312681
[junit4] 2> 88062 T130 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 88063 T138 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 88163 T130 oasc.ZkTestServer.run start zk server on port:54733
[junit4] 2> 88164 T130 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 88219 T144 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd603f name:ZooKeeperConnection Watcher:127.0.0.1:54733 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 88219 T130 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 88221 T130 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 88245 T130 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 88247 T146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e946f0 name:ZooKeeperConnection Watcher:127.0.0.1:54733/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 88247 T130 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 88251 T130 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 88255 T130 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 88258 T130 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 88263 T130 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 88269 T130 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 88270 T130 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 88277 T130 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 88278 T130 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 88281 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
[junit4] 2> 88282 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4] 2> 88282 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4] 2> 88283 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4] 2> 88283 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
[junit4] 2> 88284 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4] 2> 88284 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4] 2> 88284 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4] 2> 88285 T130 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4] 2> 88294 T130 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 88298 T147 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 88299 T147 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
[junit4] 2> at sun.nio.ch.Net.connect0(Native Method)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:435)
[junit4] 2> at sun.nio.ch.Net.connect(Net.java:427)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:643)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] 2>
[junit4] 2> 89401 T147 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
[junit4] 2> 89504 T130 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
[junit4] 2> 89506 T130 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54733 54733
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 89592 T130 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 61367 T129 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=es_CR, timezone=Asia/Dubai
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=166895408,total=214958080
[junit4] 2> NOTE: All tests run in this JVM: [DocumentAnalysisResponseTest, SolrExampleStreamingBinaryTest, TestXMLEscaping, TestCoreAdmin, TestDocumentObjectBinder, NoOpResponseParserTest, TestUpdateRequestCodec, SolrExceptionTest, TestJavaBinCodec, LargeVolumeEmbeddedTest, JettyWebappTest, AnlysisResponseBaseTest, CloudSolrServerTest]
[junit4] Completed on J0 in 62.40s, 2 tests, 1 error <<< FAILURES!
[...truncated 94 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 284 tests, 1 error
Total time: 56 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60-ea-b10) -
Build # 9906 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9906/
Java: 64bit/jdk1.7.0_60-ea-b10 -XX:+UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([2A4390E395B3481F:2E4B1F108716A73E]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:156)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:667)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:431)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:428)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:385)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:372)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:127)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:122)
... 45 more
Build Log:
[...truncated 10150 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> 61870 T509 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 61870 T509 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1395785362263
[junit4] 2> 61871 T509 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 61875 T509 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4] 2> 61876 T509 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 61876 T510 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 61976 T509 oasc.ZkTestServer.run start zk server on port:53640
[junit4] 2> 61978 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62343 T516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a3709d7 name:ZooKeeperConnection Watcher:127.0.0.1:53640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62344 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62345 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62351 T518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30f9c5d5 name:ZooKeeperConnection Watcher:127.0.0.1:53640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62352 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62354 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62356 T520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f72cca8 name:ZooKeeperConnection Watcher:127.0.0.1:53640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62356 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62357 T509 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 62360 T509 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 62364 T509 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 62367 T509 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 62369 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62373 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 62378 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62379 T522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c71d0bd name:ZooKeeperConnection Watcher:127.0.0.1:53640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62379 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62380 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 62382 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 62388 T509 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53640_solr
[junit4] 2> 62388 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 62390 T509 oasc.Overseer.start Overseer (id=91474189501792259-127.0.0.1:53640_solr-n_0000000000) starting
[junit4] 2> 62391 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 62395 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 62397 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 62399 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 62403 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 62409 T524 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 62410 T524 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 62411 T523 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 62412 T523 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 62412 T523 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 62413 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 62417 T523 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core2",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 62418 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 62419 T523 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 62421 T523 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 62422 T523 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "shard":"s1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core3",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 62424 T522 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 62424 T516 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 62515 T509 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53640 53640
[junit4] 2> 62586 T509 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 62596 T509 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4] 2> 62597 T509 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 62597 T525 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 62697 T509 oasc.ZkTestServer.run start zk server on port:41760
[junit4] 2> 62698 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62713 T531 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72b4b408 name:ZooKeeperConnection Watcher:127.0.0.1:41760 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62713 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62730 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62746 T533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50ffdfc5 name:ZooKeeperConnection Watcher:127.0.0.1:41760 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62746 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62746 T509 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 62759 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62761 T535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34e605d9 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62761 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62761 T509 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 62764 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62765 T537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50bcc833 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62765 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62768 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 62770 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 62774 T509 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41760_solr
[junit4] 2> 62774 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 62776 T509 oasc.Overseer.start Overseer (id=91474189549305859-127.0.0.1:41760_solr-n_0000000000) starting
[junit4] 2> 62778 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 62785 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 62788 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 62791 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 62802 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 62806 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 62811 T539 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 62811 T509 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 62812 T538 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 62812 T539 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 62814 T509 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 62816 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62819 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62821 T541 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47f9bd26 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62821 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62822 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62825 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4] 2> 62827 T535 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 62827 T541 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4] 2> 62828 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62830 T543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a7d434d name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62830 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62832 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62834 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 62838 T535 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 62838 T543 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 62838 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62838 T541 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 62840 T545 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@248eba46 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62840 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62842 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62844 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 62848 T535 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 62848 T541 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 62848 T543 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 62848 T545 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 62849 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62850 T547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e25e0d8 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62850 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62851 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62854 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4] 2> 62857 T545 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 62857 T543 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 62857 T535 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 62857 T547 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 62857 T541 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 62858 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62859 T549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c46465f name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62860 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62861 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62863 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4] 2> 62866 T543 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62866 T541 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62867 T535 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62867 T545 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62868 T547 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62880 T549 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 62880 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62882 T551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32ae8db4 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62882 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62884 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62886 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4] 2> 62889 T543 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62889 T541 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62890 T547 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62890 T535 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62890 T545 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62893 T549 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62893 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62894 T551 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4] 2> 62895 T553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a47c8ea name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62895 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62897 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62899 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4] 2> 62902 T545 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62902 T549 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62902 T543 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62903 T547 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62903 T535 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62903 T541 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62903 T553 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62903 T551 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4] 2> 62904 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62905 T555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56540233 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62906 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62907 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62910 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4] 2> 62914 T547 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62914 T545 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62916 T549 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62916 T551 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62915 T553 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62915 T543 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62917 T555 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62917 T541 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62917 T535 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4] 2> 62918 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62919 T557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e024652 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62920 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62921 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62924 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4] 2> 62927 T545 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62927 T551 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62927 T547 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62927 T543 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62928 T535 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62928 T541 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62928 T549 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62928 T555 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62928 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62929 T553 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62929 T557 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4] 2> 62932 T559 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54680b15 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62935 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62936 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62941 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
[junit4] 2> 62944 T535 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62944 T543 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62946 T547 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62946 T541 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62947 T545 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62947 T553 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62947 T551 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62948 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62950 T561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e9de2af name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62950 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62951 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62953 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
[junit4] 2> 62947 T557 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62950 T559 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62948 T555 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62947 T549 oascc.ZkStateReader$3.process Updating live nodes... (10)
[junit4] 2> 62957 T555 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T551 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T541 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T557 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T549 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T545 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62959 T561 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62959 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62959 T543 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T535 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T559 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62958 T553 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62961 T563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19015800 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62958 T547 oascc.ZkStateReader$3.process Updating live nodes... (11)
[junit4] 2> 62961 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62962 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62964 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
[junit4] 2> 62968 T545 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62968 T549 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62968 T557 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62969 T543 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62969 T559 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62969 T555 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62969 T553 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62970 T547 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62970 T551 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62969 T541 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62970 T561 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62970 T535 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62971 T563 oascc.ZkStateReader$3.process Updating live nodes... (12)
[junit4] 2> 62974 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62974 T565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20a01f5b name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62975 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62976 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62979 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
[junit4] 2> 62983 T551 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62983 T535 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62983 T555 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62983 T549 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T559 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62983 T547 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T553 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T565 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T545 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62983 T543 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T561 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T541 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62984 T557 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62985 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62985 T563 oascc.ZkStateReader$3.process Updating live nodes... (13)
[junit4] 2> 62987 T567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f770091 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 62988 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 62989 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 62991 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
[junit4] 2> 62995 T547 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62995 T549 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62995 T541 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62995 T535 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62996 T555 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62996 T561 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62996 T551 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62996 T567 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62997 T553 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62997 T557 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62997 T559 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62998 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 62998 T543 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62998 T565 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62998 T563 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 62998 T545 oascc.ZkStateReader$3.process Updating live nodes... (14)
[junit4] 2> 63000 T569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@318f05ef name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63000 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63001 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63003 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
[junit4] 2> 63008 T547 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63008 T549 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63009 T561 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63009 T543 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63009 T551 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63009 T535 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63009 T569 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63010 T567 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63010 T541 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63010 T555 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63010 T553 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63011 T559 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63011 T557 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63011 T563 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63011 T545 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63011 T565 oascc.ZkStateReader$3.process Updating live nodes... (15)
[junit4] 2> 63012 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63014 T571 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58e14ffc name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63014 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63015 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63017 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
[junit4] 2> 63021 T535 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T549 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T561 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T553 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T565 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T571 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T551 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T543 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T555 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T563 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T557 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T545 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T547 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63022 T559 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63025 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63024 T569 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63024 T567 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63023 T541 oascc.ZkStateReader$3.process Updating live nodes... (16)
[junit4] 2> 63026 T573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e8cc1fe name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63027 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63028 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63030 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
[junit4] 2> 63034 T561 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63034 T547 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T535 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T541 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T551 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T549 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T573 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63035 T563 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63036 T571 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63036 T553 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63037 T555 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63037 T567 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63038 T559 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63038 T557 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63040 T545 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63040 T543 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63040 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63042 T565 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63042 T575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30bc0e10 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63042 T569 oascc.ZkStateReader$3.process Updating live nodes... (17)
[junit4] 2> 63042 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63044 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63047 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
[junit4] 2> 63052 T535 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63052 T555 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63053 T561 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63053 T567 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63053 T559 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63054 T547 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63054 T557 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63055 T545 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63055 T563 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63055 T553 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63056 T551 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63057 T549 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63057 T541 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63055 T565 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63057 T543 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63057 T573 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63058 T571 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63058 T575 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63060 T569 oascc.ZkStateReader$3.process Updating live nodes... (18)
[junit4] 2> 63071 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63071 T577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12e12b4 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63072 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63073 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63075 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
[junit4] 2> 63079 T549 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63080 T561 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63081 T555 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T547 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T543 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T559 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63083 T573 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63080 T545 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63080 T535 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63079 T567 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63079 T541 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63079 T565 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63084 T571 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63084 T575 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63084 T577 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63083 T557 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63083 T569 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T551 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T553 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63082 T563 oascc.ZkStateReader$3.process Updating live nodes... (19)
[junit4] 2> 63092 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63095 T579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65fa805 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63095 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63101 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63104 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
[junit4] 2> 63108 T535 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63108 T545 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63109 T557 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63109 T553 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63110 T569 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63108 T565 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63108 T577 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63110 T573 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T567 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T563 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T575 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T571 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63110 T559 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63110 T551 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63109 T543 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63109 T555 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T541 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T561 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T547 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63111 T549 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63117 T579 oascc.ZkStateReader$3.process Updating live nodes... (20)
[junit4] 2> 63125 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63127 T581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50d04b3e name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63127 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63128 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63131 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
[junit4] 2> 63143 T535 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T547 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T553 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63143 T555 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T545 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T577 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T579 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T563 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T561 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63146 T567 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T541 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T551 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T565 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63144 T569 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63146 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63146 T549 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63146 T581 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63146 T571 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63146 T575 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T573 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T559 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T543 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63145 T557 oascc.ZkStateReader$3.process Updating live nodes... (21)
[junit4] 2> 63149 T583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fe4801d name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63149 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63151 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63153 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
[junit4] 2> 63158 T563 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63158 T561 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63158 T559 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63158 T555 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T541 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63163 T545 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63163 T573 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T547 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T565 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T543 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T557 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T569 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63162 T553 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63161 T577 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63161 T579 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63161 T551 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63161 T571 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63159 T581 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63159 T567 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63159 T549 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63159 T535 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63165 T585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ebd9110 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63163 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63163 T575 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63163 T583 oascc.ZkStateReader$3.process Updating live nodes... (22)
[junit4] 2> 63167 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63169 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63172 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
[junit4] 2> 63178 T553 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63178 T571 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63179 T575 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63179 T547 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63178 T545 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63178 T569 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T559 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T549 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T583 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T573 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63181 T557 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63181 T555 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63178 T581 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63181 T541 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63181 T567 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T535 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T565 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T579 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T563 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63180 T577 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63179 T561 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63179 T551 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63179 T543 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63184 T585 oascc.ZkStateReader$3.process Updating live nodes... (23)
[junit4] 2> 63187 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63187 T587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21b535ed name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63187 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63189 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63192 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
[junit4] 2> 63197 T571 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63198 T535 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63198 T573 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63198 T547 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63198 T579 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T565 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T551 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63200 T545 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63200 T577 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63200 T563 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63201 T555 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63201 T553 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63202 T557 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63202 T583 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63203 T549 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T581 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T569 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63203 T561 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63203 T585 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63202 T559 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63202 T587 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63201 T543 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63200 T567 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T541 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63199 T575 oascc.ZkStateReader$3.process Updating live nodes... (24)
[junit4] 2> 63207 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63208 T589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c354da1 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63208 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63209 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63212 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
[junit4] 2> 63217 T567 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63218 T543 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63218 T555 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63218 T585 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63217 T569 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63218 T541 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63218 T583 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63219 T587 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63219 T557 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T545 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T565 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T553 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T547 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T559 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T573 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T563 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T577 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63222 T561 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63222 T535 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T589 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63220 T579 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63222 T551 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63221 T581 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63221 T575 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63223 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63221 T549 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63223 T571 oascc.ZkStateReader$3.process Updating live nodes... (25)
[junit4] 2> 63224 T591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33b0d0c5 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63224 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63225 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63227 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
[junit4] 2> 63232 T587 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63232 T563 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63232 T577 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63232 T549 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T555 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T569 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T565 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T551 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T561 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T573 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T575 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T571 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T589 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T553 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T591 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T535 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T543 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T567 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63234 T545 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T583 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63233 T557 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63236 T585 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63236 T541 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63236 T547 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63235 T579 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63235 T581 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63235 T559 oascc.ZkStateReader$3.process Updating live nodes... (26)
[junit4] 2> 63238 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63241 T593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4197d0de name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63241 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63242 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63245 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
[junit4] 2> 63250 T579 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63250 T545 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63251 T567 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63251 T569 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T535 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T581 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T565 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T573 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T541 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63252 T547 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63253 T563 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63253 T553 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63253 T589 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63254 T561 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63253 T559 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63254 T585 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63254 T583 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63253 T557 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63255 T591 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63255 T549 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63255 T551 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63256 T575 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63256 T577 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63257 T593 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63258 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63259 T595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fa0d548 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63259 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63259 T543 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63259 T555 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63259 T571 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63259 T587 oascc.ZkStateReader$3.process Updating live nodes... (27)
[junit4] 2> 63260 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63263 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
[junit4] 2> 63268 T585 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T555 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T543 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T559 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T587 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T583 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T563 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T575 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T553 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T561 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63271 T569 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T589 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63269 T591 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63272 T567 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63272 T541 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63271 T593 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63271 T579 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T557 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T549 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63273 T565 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63273 T571 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T577 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63270 T551 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63273 T573 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63273 T535 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63273 T545 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63272 T595 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63272 T581 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63274 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63274 T547 oascc.ZkStateReader$3.process Updating live nodes... (28)
[junit4] 2> 63276 T597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35926cb6 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63276 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63277 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63279 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
[junit4] 2> 63285 T559 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63285 T589 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T563 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T583 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T585 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T597 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T581 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T553 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T569 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T587 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T579 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T561 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T577 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T595 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T549 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T545 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T565 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T547 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T593 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T535 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T567 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T571 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T541 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63291 T573 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63287 T557 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63287 T555 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63287 T543 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63286 T591 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T551 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63292 T575 oascc.ZkStateReader$3.process Updating live nodes... (29)
[junit4] 2> 63307 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63309 T599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e5a75b5 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63309 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63310 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63316 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
[junit4] 2> 63406 T551 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63406 T577 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63406 T595 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63406 T553 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63408 T573 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63408 T589 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63408 T569 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63408 T591 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T547 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T597 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T543 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T563 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T585 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T587 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T571 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T575 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63411 T579 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63411 T593 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T557 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T559 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T565 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63409 T581 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63411 T545 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63411 T561 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63411 T535 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T549 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T555 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T541 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T583 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T599 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63410 T567 oascc.ZkStateReader$3.process Updating live nodes... (30)
[junit4] 2> 63427 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63427 T601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a711356 name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63428 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63429 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63431 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
[junit4] 2> 63436 T567 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63436 T569 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63436 T535 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63437 T593 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63437 T545 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63438 T561 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63438 T549 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63439 T591 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63439 T577 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63439 T547 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T595 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T551 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T579 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T573 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63441 T571 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63441 T541 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T555 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63441 T565 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63441 T581 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63440 T553 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63443 T557 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63444 T563 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63444 T589 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63443 T597 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63444 T575 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63444 T585 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63444 T583 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63443 T559 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63445 T599 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63445 T587 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63445 T543 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63445 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63453 T601 oascc.ZkStateReader$3.process Updating live nodes... (31)
[junit4] 2> 63458 T603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c6174c name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63459 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63460 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63463 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
[junit4] 2> 63473 T545 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63473 T569 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63473 T567 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63473 T565 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63474 T593 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63474 T541 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63474 T573 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63475 T571 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63474 T547 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63474 T579 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63475 T587 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63475 T583 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63475 T581 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63476 T557 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63476 T535 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63476 T559 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63476 T553 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63476 T591 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63477 T585 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63477 T543 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63477 T555 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T589 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T599 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63477 T603 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63477 T597 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T575 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63479 T601 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T577 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T595 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T549 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63478 T563 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63479 T561 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63479 T551 oascc.ZkStateReader$3.process Updating live nodes... (32)
[junit4] 2> 63480 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 63482 T605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38b6cf3d name:ZooKeeperConnection Watcher:127.0.0.1:41760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 63482 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 63483 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 63485 T509 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
[junit4] 2> 63491 T547 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T541 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T535 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T579 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T569 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T573 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T605 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T577 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T571 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63491 T581 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T599 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T589 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T595 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T563 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T549 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63494 T587 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63493 T561 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T601 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T575 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T565 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T551 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T567 oascc.ZkStateReader$3.process Updating live nodes... (33)
[junit4] 2> 63492 T545 oascc.ZkStateReader$3.process Updating
[...truncated too long message...]
atcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 112229 T1062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 112229 T1056 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 112276 T1062 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4] 2> 112279 T509 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42927 42927
[junit4] 2> 112530 T509 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 112537 T509 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4] 2> 112537 T509 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 112537 T1063 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 112637 T509 oasc.ZkTestServer.run start zk server on port:41204
[junit4] 2> 112638 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 112642 T1069 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43f88247 name:ZooKeeperConnection Watcher:127.0.0.1:41204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 112643 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 112643 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 112645 T1071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9cc6d29 name:ZooKeeperConnection Watcher:127.0.0.1:41204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 112645 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 112669 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 112671 T1073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@397d29ec name:ZooKeeperConnection Watcher:127.0.0.1:41204 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 112671 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 112671 T509 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 112673 T509 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 112674 T509 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 112675 T509 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 112677 T509 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 112679 T509 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 112681 T1075 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f891438 name:ZooKeeperConnection Watcher:127.0.0.1:41204/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 112681 T509 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 112682 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 112683 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 112686 T509 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41204_solr
[junit4] 2> 112686 T509 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 112687 T509 oasc.Overseer.start Overseer (id=91474192824729603-127.0.0.1:41204_solr-n_0000000000) starting
[junit4] 2> 112688 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 112690 T509 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 112693 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 112694 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 112697 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 112699 T509 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 112703 T1077 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 112704 T1077 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 112704 T1076 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 112704 T509 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 112706 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 112706 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"recovering"}
[junit4] 2> 112706 T1076 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 112708 T1075 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 112709 T1069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 112755 T509 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 112757 T1075 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 112758 T1076 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 112759 T1076 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "base_url":"http://127.0.0.1/solr",
[junit4] 2> "node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"core1",
[junit4] 2> "roles":"",
[junit4] 2> "state":"active"}
[junit4] 2> 112759 T1076 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
[junit4] 2> 112759 T1076 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 112761 T1075 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 112864 T1069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 112911 T509 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41204 41204
[junit4] 2> 113079 T509 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 113081 T509 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 113081 T509 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 116081 T509 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=bg_BG, timezone=America/Argentina/Buenos_Aires
[junit4] 2> NOTE: Linux 3.8.0-37-generic amd64/Oracle Corporation 1.7.0_60-ea (64-bit)/cpus=8,threads=1,free=115328232,total=284164096
[junit4] 2> NOTE: All tests run in this JVM: [TestManagedSchemaFieldResource, SearchHandlerTest, TestSolrXmlPersistor, TestManagedStopFilterFactory, SolrIndexSplitterTest, AliasIntegrationTest, ReturnFieldsTest, DistributedQueryElevationComponentTest, TestDefaultSearchFieldResource, DistributedTermsComponentTest, OverseerTest]
[junit4] Completed on J0 in 54.25s, 9 tests, 1 error <<< FAILURES!
[...truncated 1091 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 393 suites, 1651 tests, 1 error, 49 ignored (28 assumptions)
Total time: 49 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_60-ea-b10 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure