You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/08/04 15:23:07 UTC
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 597 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/597/
4 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620:C1AA825D2585F901]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131)
... 47 more
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup: 1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620]:0)
FAILED: org.apache.solr.cloud.MultiThreadedOCPTest.testDistribSearch
Error Message:
We have a failed SPLITSHARD task
Stack Trace:
java.lang.AssertionError: We have a failed SPLITSHARD task
at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620:444483B6407F761C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.MultiThreadedOCPTest.testTaskExclusivity(MultiThreadedOCPTest.java:125)
at org.apache.solr.cloud.MultiThreadedOCPTest.doTest(MultiThreadedOCPTest.java:71)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11461 lines...]
[junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/init-core-data-001
[junit4] 2> 737244 T3948 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 737245 T3948 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 737253 T3948 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 737254 T3948 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 737255 T3949 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 737355 T3948 oasc.ZkTestServer.run start zk server on port:29819
[junit4] 2> 737357 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 737361 T3955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44cd09b3 name:ZooKeeperConnection Watcher:127.0.0.1:29819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 737361 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 737362 T3948 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 737366 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 737368 T3957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cf9b4d7 name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 737368 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 737368 T3948 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 737371 T3948 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 737374 T3948 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 737376 T3948 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 737378 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 737379 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 737383 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 737384 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 737387 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 737387 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 737390 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 737391 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 737393 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 737394 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 737397 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 737397 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 737400 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 737401 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 737403 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 737404 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 737407 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 737407 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 737410 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 737410 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 737413 T3948 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 737414 T3948 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 737417 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 737419 T3959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13592620 name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 737419 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 738627 T3948 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 738631 T3948 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 738632 T3948 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:23125
[junit4] 2> 738636 T3948 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 738636 T3948 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 738637 T3948 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002
[junit4] 2> 738637 T3948 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/'
[junit4] 2> 738667 T3948 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/solr.xml
[junit4] 2> 738719 T3948 oasc.CoreContainer.<init> New CoreContainer 1577451753
[junit4] 2> 738720 T3948 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/]
[junit4] 2> 738721 T3948 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 738721 T3948 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 738721 T3948 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 738721 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 738722 T3948 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 738722 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 738722 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 738722 T3948 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 738723 T3948 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 738725 T3948 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 738726 T3948 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 738727 T3948 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 738727 T3948 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 738727 T3948 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29819/solr
[junit4] 2> 738728 T3948 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 738728 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 738730 T3970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c63bae6 name:ZooKeeperConnection Watcher:127.0.0.1:29819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 738731 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 738733 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 738734 T3972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3800875b name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 738734 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 738736 T3948 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 738739 T3948 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 738742 T3948 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 738744 T3948 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 738747 T3948 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 738750 T3948 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 738751 T3948 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23125_
[junit4] 2> 738752 T3948 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23125_
[junit4] 2> 738755 T3948 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 738756 T3948 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 738759 T3948 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:23125_
[junit4] 2> 738760 T3948 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 738761 T3948 oasc.Overseer.start Overseer (id=92219256168120324-127.0.0.1:23125_-n_0000000000) starting
[junit4] 2> 738764 T3948 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 738771 T3974 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 738772 T3948 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 738774 T3948 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 738776 T3948 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 738778 T3973 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 738785 T3975 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 738785 T3975 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 738786 T3975 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 738786 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 738788 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 738789 T3973 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:23125",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:23125_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 738789 T3973 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 738789 T3973 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 738791 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 738792 T3976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 739786 T3975 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 739787 T3975 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 739788 T3975 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 739788 T3975 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 739789 T3975 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 739790 T3975 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/collection1/'
[junit4] 2> 739792 T3975 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 739793 T3975 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 739793 T3975 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 739859 T3975 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 739905 T3975 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 739907 T3975 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 739919 T3975 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 740295 T3975 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 740309 T3975 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 740312 T3975 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 740334 T3975 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 740339 T3975 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 740344 T3975 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 740345 T3975 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 740346 T3975 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 740346 T3975 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 740347 T3975 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 740347 T3975 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 740348 T3975 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 740348 T3975 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 740348 T3975 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-001/control/data/
[junit4] 2> 740349 T3975 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@348ec43e
[junit4] 2> 740349 T3975 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-001/control/data
[junit4] 2> 740350 T3975 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-001/control/data/index/
[junit4] 2> 740350 T3975 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 740350 T3975 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-001/control/data/index
[junit4] 2> 740351 T3975 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.505048293379147]
[junit4] 2> 740352 T3975 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2f6e78a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f9bc4ca; maxCacheMB=0.380859375 maxMergeSizeMB=0.6298828125)),segFN=segments_1,generation=1}
[junit4] 2> 740352 T3975 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 740357 T3975 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 740358 T3975 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 740358 T3975 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 740359 T3975 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 740359 T3975 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 740359 T3975 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 740360 T3975 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 740360 T3975 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 740360 T3975 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 740361 T3975 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 740361 T3975 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 740362 T3975 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 740362 T3975 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 740363 T3975 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 740364 T3975 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 740364 T3975 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 740364 T3975 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 740365 T3975 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 740365 T3975 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 740365 T3975 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 740366 T3975 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 740366 T3975 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 740384 T3975 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740386 T3975 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740388 T3975 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740389 T3975 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740391 T3975 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 740397 T3975 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 740397 T3975 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 740398 T3975 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=34.0, floorSegmentMB=1.515625, forceMergeDeletesPctAllowed=5.151218810294412, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 740399 T3975 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2f6e78a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f9bc4ca; maxCacheMB=0.380859375 maxMergeSizeMB=0.6298828125)),segFN=segments_1,generation=1}
[junit4] 2> 740399 T3975 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 740400 T3975 oass.SolrIndexSearcher.<init> Opening Searcher@5aaa3797[collection1] main
[junit4] 2> 740400 T3975 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 740401 T3975 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 740401 T3975 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 740402 T3975 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 740402 T3975 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 740402 T3975 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 740403 T3975 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 740404 T3975 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 740404 T3975 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 740407 T3975 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 740407 T3975 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 740407 T3975 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 740410 T3977 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5aaa3797[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 740421 T3975 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 740422 T3980 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:23125 collection:control_collection shard:shard1
[junit4] 2> 740423 T3948 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 740423 T3948 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 740423 T3980 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 740429 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 740430 T3982 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@99512c2 name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 740430 T3980 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 740430 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 740433 T3948 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 740433 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740433 T3980 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 740434 T3980 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C180 name=collection1 org.apache.solr.core.SolrCore@21efe33a url=https://127.0.0.1:23125/collection1 node=127.0.0.1:23125_ C180_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:23125, core=collection1, node_name=127.0.0.1:23125_}
[junit4] 2> 740434 T3980 C180 P23125 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:23125/collection1/
[junit4] 2> 740434 T3980 C180 P23125 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 740435 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 740435 T3980 C180 P23125 oasc.SyncStrategy.syncToMe https://127.0.0.1:23125/collection1/ has no replicas
[junit4] 2> 740435 T3948 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 740435 T3980 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:23125/collection1/ shard1
[junit4] 2> 740436 T3980 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 740438 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740438 T3976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740439 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740442 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 740445 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740548 T3976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740548 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740595 T3980 oasc.ZkController.register We are https://127.0.0.1:23125/collection1/ and leader is https://127.0.0.1:23125/collection1/
[junit4] 2> 740596 T3980 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:23125
[junit4] 2> 740596 T3980 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 740596 T3980 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 740596 T3980 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 740599 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740599 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740599 T3980 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 740599 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740600 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 740601 T3973 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:23125",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:23125_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 740604 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 740706 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 740706 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 741795 T3948 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 741796 T3948 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 741803 T3948 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 741804 T3948 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31615
[junit4] 2> 741808 T3948 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 741809 T3948 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 741809 T3948 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003
[junit4] 2> 741810 T3948 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/'
[junit4] 2> 741847 T3948 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/solr.xml
[junit4] 2> 741900 T3948 oasc.CoreContainer.<init> New CoreContainer 1965792688
[junit4] 2> 741901 T3948 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/]
[junit4] 2> 741902 T3948 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 741902 T3948 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 741902 T3948 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 741903 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 741903 T3948 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 741903 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 741903 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 741904 T3948 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 741904 T3948 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 741906 T3948 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 741908 T3948 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 741908 T3948 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 741908 T3948 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 741908 T3948 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29819/solr
[junit4] 2> 741909 T3948 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 741910 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741911 T3996 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5afb91d3 name:ZooKeeperConnection Watcher:127.0.0.1:29819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741912 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741914 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741915 T3998 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1516121e name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741916 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741922 T3948 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 742925 T3948 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31615_
[junit4] 2> 742926 T3948 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31615_
[junit4] 2> 742940 T4000 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 742940 T4000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 742942 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742942 T4000 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 742942 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742942 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742944 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 742945 T3973 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:31615",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:31615_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 742945 T3973 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 742945 T3973 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 742948 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 742949 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 742949 T3999 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 742949 T3984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 743942 T4000 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 743943 T4000 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 743944 T4000 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 743944 T4000 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 743945 T4000 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 743945 T4000 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/'
[junit4] 2> 743948 T4000 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 743948 T4000 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 743949 T4000 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 744014 T4000 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 744060 T4000 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 744062 T4000 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 744074 T4000 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 744450 T4000 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 744464 T4000 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 744467 T4000 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 744489 T4000 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 744494 T4000 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 744498 T4000 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 744499 T4000 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 744500 T4000 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 744501 T4000 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 744502 T4000 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 744502 T4000 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 744502 T4000 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 744503 T4000 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 744503 T4000 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/data/
[junit4] 2> 744503 T4000 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@348ec43e
[junit4] 2> 744504 T4000 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/data
[junit4] 2> 744505 T4000 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/data/index/
[junit4] 2> 744505 T4000 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 744505 T4000 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-003/collection1/data/index
[junit4] 2> 744506 T4000 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.505048293379147]
[junit4] 2> 744508 T4000 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@33c8e772 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a0e9d9e; maxCacheMB=0.380859375 maxMergeSizeMB=0.6298828125)),segFN=segments_1,generation=1}
[junit4] 2> 744508 T4000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 744515 T4000 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 744515 T4000 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 744516 T4000 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 744516 T4000 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 744516 T4000 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 744517 T4000 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 744517 T4000 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 744518 T4000 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 744518 T4000 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 744519 T4000 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 744520 T4000 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 744520 T4000 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 744521 T4000 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 744522 T4000 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 744522 T4000 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 744523 T4000 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 744523 T4000 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 744524 T4000 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 744524 T4000 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 744525 T4000 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 744525 T4000 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 744526 T4000 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 744545 T4000 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 744547 T4000 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 744549 T4000 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 744551 T4000 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 744553 T4000 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 744559 T4000 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 744559 T4000 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 744560 T4000 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=34.0, floorSegmentMB=1.515625, forceMergeDeletesPctAllowed=5.151218810294412, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 744561 T4000 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@33c8e772 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7a0e9d9e; maxCacheMB=0.380859375 maxMergeSizeMB=0.6298828125)),segFN=segments_1,generation=1}
[junit4] 2> 744561 T4000 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 744562 T4000 oass.SolrIndexSearcher.<init> Opening Searcher@aeb4bd5[collection1] main
[junit4] 2> 744562 T4000 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 744563 T4000 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 744564 T4000 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 744564 T4000 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 744564 T4000 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 744565 T4000 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 744566 T4000 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 744566 T4000 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 744567 T4000 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 744567 T4000 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 744572 T4001 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@aeb4bd5[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 744577 T4000 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 744578 T4004 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:31615 collection:collection1 shard:shard2
[junit4] 2> 744579 T3948 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 744579 T3948 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 744579 T4004 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 744586 T4004 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 744588 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744588 T4004 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 744588 T4004 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C181 name=collection1 org.apache.solr.core.SolrCore@60d3a8d url=https://127.0.0.1:31615/collection1 node=127.0.0.1:31615_ C181_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:31615, core=collection1, node_name=127.0.0.1:31615_}
[junit4] 2> 744589 T4004 C181 P31615 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:31615/collection1/
[junit4] 2> 744589 T4004 C181 P31615 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 744589 T4004 C181 P31615 oasc.SyncStrategy.syncToMe https://127.0.0.1:31615/collection1/ has no replicas
[junit4] 2> 744589 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744590 T4004 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:31615/collection1/ shard2
[junit4] 2> 744590 T4004 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 744593 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744594 T3984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744594 T3999 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744594 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744597 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744600 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744703 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744703 T3999 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744703 T3984 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744749 T4004 oasc.ZkController.register We are https://127.0.0.1:31615/collection1/ and leader is https://127.0.0.1:31615/collection1/
[junit4] 2> 744750 T4004 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:31615
[junit4] 2> 744750 T4004 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 744750 T4004 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 744750 T4004 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 744753 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744753 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744753 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744753 T4004 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744755 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744756 T3973 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:31615",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:31615_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 744758 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744861 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744861 T3999 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744861 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 745750 T3948 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 745751 T3948 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 745756 T3948 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 745757 T3948 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:20429
[junit4] 2> 745759 T3948 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 745760 T3948 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 745760 T3948 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004
[junit4] 2> 745760 T3948 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/'
[junit4] 2> 745795 T3948 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/solr.xml
[junit4] 2> 745852 T3948 oasc.CoreContainer.<init> New CoreContainer 996180865
[junit4] 2> 745852 T3948 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/]
[junit4] 2> 745853 T3948 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 745854 T3948 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 745854 T3948 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 745854 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 745855 T3948 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 745855 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 745855 T3948 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 745855 T3948 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 745856 T3948 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 745858 T3948 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 745859 T3948 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 745860 T3948 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 745860 T3948 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 745860 T3948 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:29819/solr
[junit4] 2> 745861 T3948 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 745870 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 745872 T4015 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34575471 name:ZooKeeperConnection Watcher:127.0.0.1:29819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 745872 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 745875 T3948 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 745876 T4017 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ab4d80 name:ZooKeeperConnection Watcher:127.0.0.1:29819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 745876 T3948 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 745883 T3948 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 746887 T3948 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20429_
[junit4] 2> 746888 T3948 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20429_
[junit4] 2> 746900 T4019 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 746901 T4019 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 746902 T3985 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746902 T4019 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 746902 T3984 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746902 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746903 T3973 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 746904 T3973 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:20429",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:20429_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 746904 T3973 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 746905 T3973 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 746907 T3976 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746908 T4018 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746908 T3999 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746908 T3976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 746908 T3983 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 747902 T4019 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 747903 T4019 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 747904 T4019 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 747904 T4019 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 747905 T4019 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 747905 T4019 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/'
[junit4] 2> 747907 T4019 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 747908 T4019 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 747909 T4019 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 747976 T4019 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 748022 T4019 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 748024 T4019 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 748036 T4019 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 748415 T4019 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 748429 T4019 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 748432 T4019 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 748455 T4019 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 748460 T4019 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 748465 T4019 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 748466 T4019 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 748467 T4019 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 748468 T4019 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 748469 T4019 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 748469 T4019 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 748469 T4019 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 748470 T4019 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 748470 T4019 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/data/
[junit4] 2> 748470 T4019 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@348ec43e
[junit4] 2> 748472 T4019 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/collection1/data
[junit4] 2> 748472 T4019 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-C5A20DAE37201620-001/tempDir-004/
[...truncated too long message...]
ystemInfoHandlerTest, TestUniqueKeyFieldResource, BadCopyFieldTest, TestLazyCores, ChaosMonkeySafeLeaderTest, TestFieldSortValues, TestElisionMultitermQuery, TestQueryTypes, HdfsRecoveryZkTest, TestOmitPositions, CoreContainerCoreInitFailuresTest, AutoCommitTest, ScriptEngineTest, TestCollapseQParserPlugin, TestQueryUtils, MultiThreadedOCPTest, TestDocumentBuilder, SortByFunctionTest, TestHighFrequencyDictionaryFactory, DeleteShardTest, BasicDistributedZk2Test, DistributedQueryComponentOptimizationTest, CurrencyFieldXmlFileTest, RankQueryTest, SampleTest, TestSolrQueryParser, TestFunctionQuery, TestCloudSchemaless, DirectUpdateHandlerOptimizeTest, DOMUtilTest, TestBlendedInfixSuggestions, DistributedQueryComponentCustomSortTest, TestAtomicUpdateErrorCases, HdfsLockFactoryTest, DateMathParserTest, TestQuerySenderNoQuery, ResourceLoaderTest, DocValuesTest, TestRestManager, SchemaVersionSpecificBehaviorTest, TestCSVResponseWriter, ZkSolrClientTest, TestSweetSpotSimilarityFactory, TestMergePolicyConfig, ConvertedLegacyTest, OpenCloseCoreStressTest, TestPostingsSolrHighlighter, SolrTestCaseJ4Test, DistributedTermsComponentTest, TestDistributedMissingSort, TestCharFilters, TestMultiCoreConfBootstrap, TestPhraseSuggestions, StatsComponentTest, AnalysisAfterCoreReloadTest, UpdateRequestProcessorFactoryTest, TestManagedSchemaFieldResource, TestNonDefinedSimilarityFactory, TestIndexSearcher, PolyFieldTest, SolrIndexConfigTest, CollectionsAPIAsyncDistributedZkTest, TestRandomMergePolicy, TestDistribDocBasedVersion, TestPerFieldSimilarity, DistanceFunctionTest, TestValueSourceCache, HdfsDirectoryTest, TestSolrQueryParserResource, TestSimpleQParserPlugin, TestCollectionAPI, CopyFieldTest, LukeRequestHandlerTest, TestZkChroot, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SearchHandlerTest, UniqFieldsUpdateProcessorFactoryTest, CursorPagingTest, TestPseudoReturnFields, DistributedSuggestComponentTest, TestRecovery, CircularListTest, TestExpandComponent, TestNRTOpen, TestCloudManagedSchema, FullSolrCloudDistribCmdsTest, QueryFacetTest, SimplePostToolTest, TestOrdValues, TestGroupingSearch, TestPHPSerializedResponseWriter, QueryElevationComponentTest, ShardSplitTest, SuggesterFSTTest, TestRTGBase, TestSolrQueryParserDefaultOperatorResource, DistributedExpandComponentTest, FieldAnalysisRequestHandlerTest, TestRecoveryHdfs, FileBasedSpellCheckerTest, RecoveryZkTest, TestRandomDVFaceting, ShardRoutingCustomTest, TestDistributedGrouping, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestRealTimeGet, TestStressReorder, TestJoin, TestStressVersions, HardAutoCommitTest, TestCoreContainer, SolrCoreTest, SolrCmdDistributorTest, TestBadConfig, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, SolrIndexSplitterTest, TestExtendedDismaxParser, CoreAdminHandlerTest, SolrRequestParserTest, SuggesterTSTTest, TestStressLucene, SuggesterWFSTTest, NoCacheHeaderTest, TestUpdate, TestWordDelimiterFilterFactory, TestRemoteStreaming, DebugComponentTest, StandardRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, RequestHandlersTest, TestIndexingPerformance, IndexSchemaTest, TestCollationField, ReturnFieldsTest, QueryParsingTest, CSVRequestHandlerTest, HighlighterConfigTest, UpdateParamsTest, TestSolrIndexConfig, TestSolrDeletionPolicy2, NumericFieldsTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, TestSolrCoreProperties, TestFastWriter, TestJmxMonitoredMap, URLClassifyProcessorTest, TestSolrXMLSerializer, PreAnalyzedFieldTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, ClusterStateTest, TestLRUCache, SliceStateTest, UUIDFieldTest, TestCursorMarkWithoutUniqueKey, AbstractAnalyticsStatsTest, NoFacetTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, FieldFacetTest, RangeFacetTest, FunctionTest, AliasIntegrationTest, AsyncMigrateRouteKeyTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, OverseerRolesTest, RollingRestartTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, StressHdfsTest, CachingDirectoryFactoryTest, QueryResultKeyTest, TestConfigSets, TestImplicitCoreProperties, TestNonNRTOpen, TestReloadAndDeleteDocs, TestShardHandlerFactory, TestSolrXml, TestSolrXmlPersistence, TestSolrXmlPersistor, TestReplicationHandlerBackup]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestReplicationHandlerBackup -Dtests.seed=C5A20DAE37201620 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_SY -Dtests.timezone=Etc/Greenwich -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J1 | TestReplicationHandlerBackup (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
[junit4] > 1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] > at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=35207, name=Thread-8580, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] > at __randomizedtesting.SeedInfo.seed([C5A20DAE37201620]:0)
[junit4] Completed on J1 in 30.32s, 1 test, 2 errors <<< FAILURES!
[...truncated 144 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 422 suites, 1756 tests, 2 suite-level errors, 1 error, 1 failure, 249 ignored (9 assumptions)
Total time: 198 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #592
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 662075422 bytes
Compression is 0.0%
Took 1 min 48 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 600 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/600/
1 tests failed.
REGRESSION: org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings
Error Message:
startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=12,endOffset=9
Stack Trace:
java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=12,endOffset=9
at __randomizedtesting.SeedInfo.seed([A12AB084348D4A48:CB710F956DC36ABB]:0)
at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:704)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:615)
at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:925)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 6509 lines...]
[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
[junit4] 2> TEST FAIL: useCharFilter=false text='\ud834\uddbd\ud834\uddb6\ud834\udd2f\ud834\udd56 \ue09d\ue77e[\uda4b\udd19\u05c2W\u2444 </script> fybws \u298e\ud9a6\udf66\u2d61\ucbaf\u03da\ue191*\u04b7\u92d3\uf11aH\u0df6\u03f9 &a'
[junit4] 2> Exception from random analyzer:
[junit4] 2> charfilters=
[junit4] 2> org.apache.lucene.analysis.charfilter.MappingCharFilter(org.apache.lucene.analysis.charfilter.NormalizeCharMap@1d2a5506, java.io.StringReader@46340bdf)
[junit4] 2> org.apache.lucene.analysis.fa.PersianCharFilter(org.apache.lucene.analysis.charfilter.MappingCharFilter@4755b3c2)
[junit4] 2> tokenizer=
[junit4] 2> org.apache.lucene.analysis.path.PathHierarchyTokenizer()
[junit4] 2> filters=
[junit4] 2> org.apache.lucene.analysis.miscellaneous.ASCIIFoldingFilter(ValidatingTokenFilter@7ff6483b term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word)
[junit4] 2> org.apache.lucene.analysis.ar.ArabicNormalizationFilter(ValidatingTokenFilter@655e513f term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word)
[junit4] 2> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter(ValidatingTokenFilter@343234d4 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, [-123, -53, -19, 86, 30, -43, -10, -41, -17, -104, -21, 47, 39, -69, -9, 90, 36, 114, 75, -69, 77, -121, 40, 49, -37, -23, -30, 51, -106, -66, -61, 104, -127, -38, 111, -88, -43, -30, -128, -71, -107, -2, -9, -11, -115, -93, 2, 102, -119, 54, -82, 73, -58], 49, [igagfch, fzece, wdmbpkqopi, mchxdcl, snt, yipj, knrqo, n])
[junit4] 2> org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@73516627 term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word, 41)
[junit4] 2> offsetsAreCorrect=false
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestRandomChains -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=A12AB084348D4A48 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh_CN -Dtests.timezone=Australia/Melbourne -Dtests.file.encoding=US-ASCII
[junit4] ERROR 8.01s J1 | TestRandomChains.testRandomChainsWithLargeStrings <<<
[junit4] > Throwable #1: java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, startOffset=12,endOffset=9
[junit4] > at __randomizedtesting.SeedInfo.seed([A12AB084348D4A48:CB710F956DC36ABB]:0)
[junit4] > at org.apache.lucene.analysis.tokenattributes.PackedTokenAttributeImpl.setOffset(PackedTokenAttributeImpl.java:107)
[junit4] > at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:345)
[junit4] > at org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:704)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:615)
[junit4] > at org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:513)
[junit4] > at org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:925)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/analysis/common/test/J1/./temp/lucene.analysis.core.TestRandomChains-A12AB084348D4A48-001
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {dummy=DFR I(n)B3(800.0)}, locale=zh_CN, timezone=Australia/Melbourne
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=1,free=78087424,total=288358400
[junit4] 2> NOTE: All tests run in this JVM: [TestRollingCharBuffer, TestHindiAnalyzer, TestTurkishLowerCaseFilterFactory, TestCJKBigramFilter, EdgeNGramTokenizerTest, TestPersianAnalyzer, TestRussianAnalyzer, TestCzechStemmer, TestGreekStemmer, TestGalicianAnalyzer, TestTypeTokenFilter, TestKeepFilterFactory, TestScandinavianFoldingFilterFactory, TestCharTokenizers, TestArmenianAnalyzer, TestGalicianStemFilterFactory, TestSwedishLightStemFilter, DateRecognizerSinkTokenizerTest, TestBrazilianStemmer, TestLimitTokenPositionFilterFactory, TestEnglishAnalyzer, TestCharArrayIterator, TestSpanishLightStemFilterFactory, TestCharArrayMap, CommonGramsFilterTest, TestSnowball, TestDoubleEscape, TestBulgarianStemFilterFactory, TestHunspellStemFilterFactory, TestCJKWidthFilter, TestNorwegianMinimalStemFilter, TestLatvianStemFilterFactory, TestStemmer, TestDictionaryCompoundWordTokenFilterFactory, TestCodepointCountFilter, TestPortugueseAnalyzer, TokenOffsetPayloadTokenFilterTest, TestPortugueseMinimalStemFilterFactory, TestSpaces, TestEscaped, TestSpanishAnalyzer, ShingleFilterTest, TestGreekStemFilterFactory, QueryAutoStopWordAnalyzerTest, TestCompoundWordTokenFilter, TestZeroAffix2, TestHungarianLightStemFilterFactory, TestStandardAnalyzer, TestKeywordMarkerFilter, TestKStemmer, TestUAX29URLEmailTokenizer, TestTurkishAnalyzer, TestCommonGramsQueryFilterFactory, TestKeywordRepeatFilter, TestCapitalizationFilter, TestHindiFilters, TestAnalysisSPILoader, TestFrenchLightStemFilter, TestStemmerOverrideFilterFactory, TestPersianCharFilter, TestWordnetSynonymParser, TestSpanishLightStemFilter, TestFinnishLightStemFilterFactory, TestGermanStemFilterFactory, TestCollationDocValuesField, TestTruncateTokenFilter, TestFactories, TestRandomChains]
[junit4] Completed on J1 in 55.45s, 2 tests, 1 error <<< FAILURES!
[...truncated 407 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:481: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:454: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:544: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:2028: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/analysis/build.xml:106: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/analysis/build.xml:38: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/module-build.xml:60: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 268 suites, 1358 tests, 1 error, 1 ignored
Total time: 81 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #592
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 1669841 bytes
Compression is 0.0%
Took 0.6 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 599 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/599/
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
ERROR: SolrIndexSearcher opens=16 closes=15
Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=16 closes=15
at __randomizedtesting.SeedInfo.seed([A07445A293ED20EE]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:436)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 1) Thread[id=197, name=qtp1182665876-197, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:374) at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:103) at org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:228) at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:89) at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:766) at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1662) at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) 2) Thread[id=205, name=searcherExecutor-87-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
1) Thread[id=197, name=qtp1182665876-197, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.blockUntilFinished(ConcurrentUpdateSolrServer.java:374)
at org.apache.solr.update.StreamingSolrServers.blockUntilFinished(StreamingSolrServers.java:103)
at org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:228)
at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:89)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:766)
at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1662)
at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
2) Thread[id=205, name=searcherExecutor-87-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([A07445A293ED20EE]:0)
FAILED: org.apache.solr.cloud.MultiThreadedOCPTest.testDistribSearch
Error Message:
We have a failed SPLITSHARD task
Stack Trace:
java.lang.AssertionError: We have a failed SPLITSHARD task
at __randomizedtesting.SeedInfo.seed([A07445A293ED20EE:2192CBBAE4B240D2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.MultiThreadedOCPTest.testTaskExclusivity(MultiThreadedOCPTest.java:125)
at org.apache.solr.cloud.MultiThreadedOCPTest.doTest(MultiThreadedOCPTest.java:71)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 13326 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/init-core-data-001
[junit4] 2> 64454 T154 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pb_/
[junit4] 2> 64489 T154 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 64490 T154 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 64491 T155 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 64590 T154 oasc.ZkTestServer.run start zk server on port:40090
[junit4] 2> 64591 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 64595 T161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@517f37f5 name:ZooKeeperConnection Watcher:127.0.0.1:40090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 64595 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 64596 T154 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 64600 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 64601 T163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@289f65df name:ZooKeeperConnection Watcher:127.0.0.1:40090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 64602 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 64602 T154 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 64606 T154 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 64608 T154 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 64611 T154 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 64614 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 64615 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 64619 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 64620 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 64624 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 64624 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 64627 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 64628 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 64631 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 64632 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 64635 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 64636 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 64639 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 64640 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 64643 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 64643 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 64646 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 64647 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 64650 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 64651 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 64654 T154 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 64654 T154 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 67611 T154 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 67616 T154 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:53547
[junit4] 2> 67617 T154 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 67617 T154 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 67617 T154 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002
[junit4] 2> 67618 T154 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/'
[junit4] 2> 67678 T154 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/solr.xml
[junit4] 2> 67792 T154 oasc.CoreContainer.<init> New CoreContainer 1286591841
[junit4] 2> 67792 T154 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/]
[junit4] 2> 67794 T154 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 67795 T154 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 67795 T154 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 67795 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 67796 T154 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 67796 T154 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 67796 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 67797 T154 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 67797 T154 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 67798 T154 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 67798 T154 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 67799 T154 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 67799 T154 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 67800 T154 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40090/solr
[junit4] 2> 67800 T154 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 67801 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 67803 T174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c2e46 name:ZooKeeperConnection Watcher:127.0.0.1:40090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 67803 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 67806 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 67808 T176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52aea950 name:ZooKeeperConnection Watcher:127.0.0.1:40090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 67808 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 67809 T154 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 67812 T154 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 67815 T154 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 67818 T154 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 67820 T154 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 67823 T154 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 67825 T154 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53547_pb_
[junit4] 2> 67826 T154 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53547_pb_
[junit4] 2> 67828 T154 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 67830 T154 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 67834 T154 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53547_pb_
[junit4] 2> 67834 T154 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 67836 T154 oasc.Overseer.start Overseer (id=92237202507694083-127.0.0.1:53547_pb_-n_0000000000) starting
[junit4] 2> 67839 T154 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 67847 T178 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 67847 T154 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 67849 T154 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 67851 T154 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 67853 T177 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 67875 T179 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 67875 T179 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 67877 T179 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 67877 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 67883 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 67883 T177 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:53547/pb_",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53547_pb_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 67884 T177 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 67914 T177 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 67917 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 67918 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 68878 T179 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 68878 T179 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 68879 T179 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 68879 T179 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 68880 T179 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 68881 T179 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/collection1/'
[junit4] 2> 68882 T179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 68883 T179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 68883 T179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 68937 T179 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 68980 T179 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 68981 T179 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 69001 T179 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 69414 T179 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 69417 T179 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 69420 T179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 69426 T179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 69471 T179 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 69471 T179 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 69542 T179 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/
[junit4] 2> 69543 T179 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@54b83713
[junit4] 2> 69545 T179 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data
[junit4] 2> 69547 T179 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/index/
[junit4] 2> 69548 T179 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 69549 T179 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/index
[junit4] 2> 69550 T179 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=0.4306640625, floorSegmentMB=1.0390625, forceMergeDeletesPctAllowed=3.007953945489976, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 69582 T179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 69583 T179 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 69590 T179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 69591 T179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 69591 T179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 69591 T179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 69592 T179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 69592 T179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 69592 T179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 69593 T179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 69593 T179 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 69594 T179 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 69594 T179 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 69595 T179 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 69596 T179 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 69596 T179 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 69597 T179 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 69597 T179 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 69597 T179 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 69598 T179 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 69598 T179 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 69599 T179 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 69599 T179 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 69600 T179 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 69619 T179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 69622 T179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 69624 T179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 69626 T179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 69628 T179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 69637 T179 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 69638 T179 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 69639 T179 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=56.107421875, floorSegmentMB=0.4912109375, forceMergeDeletesPctAllowed=26.23316267623028, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.28139084101318934
[junit4] 2> 69641 T179 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 69642 T179 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 69642 T179 oass.SolrIndexSearcher.<init> Opening Searcher@36e6512c[collection1] main
[junit4] 2> 69643 T179 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 69644 T179 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 69644 T179 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 69656 T179 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 69657 T179 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 69657 T179 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 69658 T179 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 69659 T179 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 69660 T179 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 69663 T179 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 69664 T179 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 69664 T179 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 69668 T181 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36e6512c[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 69674 T179 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 69675 T184 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53547/pb_ collection:control_collection shard:shard1
[junit4] 2> 69676 T154 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 69676 T154 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 69714 T184 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 69721 T184 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 69723 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69724 T184 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 69724 T184 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 69725 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C49 name=collection1 org.apache.solr.core.SolrCore@3856a77c url=http://127.0.0.1:53547/pb_/collection1 node=127.0.0.1:53547_pb_ C49_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:53547/pb_, core=collection1, node_name=127.0.0.1:53547_pb_}
[junit4] 2> 69732 T184 C49 P53547 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53547/pb_/collection1/
[junit4] 2> 69732 T184 C49 P53547 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 69733 T184 C49 P53547 oasc.SyncStrategy.syncToMe http://127.0.0.1:53547/pb_/collection1/ has no replicas
[junit4] 2> 69733 T184 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53547/pb_/collection1/ shard1
[junit4] 2> 69734 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69735 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 69747 T184 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 69753 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69755 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 69758 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69760 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 69761 T186 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ae41e92 name:ZooKeeperConnection Watcher:127.0.0.1:40090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 69762 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 69763 T154 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 69769 T154 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 69861 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 69861 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 69906 T184 oasc.ZkController.register We are http://127.0.0.1:53547/pb_/collection1/ and leader is http://127.0.0.1:53547/pb_/collection1/
[junit4] 2> 69907 T184 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53547/pb_
[junit4] 2> 69907 T184 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 69907 T184 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 69907 T184 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 69910 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69910 T188 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69910 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 69910 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 69912 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 69913 T177 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:53547/pb_",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53547_pb_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 69916 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 70019 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 70019 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 71199 T154 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 71200 T154 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 71202 T154 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:45461
[junit4] 2> 71203 T154 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 71203 T154 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 71203 T154 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003
[junit4] 2> 71204 T154 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/'
[junit4] 2> 71236 T154 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/solr.xml
[junit4] 2> 71299 T154 oasc.CoreContainer.<init> New CoreContainer 95952472
[junit4] 2> 71300 T154 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/]
[junit4] 2> 71301 T154 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 71301 T154 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 71302 T154 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 71302 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 71302 T154 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 71302 T154 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 71303 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 71303 T154 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 71303 T154 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 71304 T154 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 71304 T154 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 71304 T154 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 71304 T154 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 71305 T154 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40090/solr
[junit4] 2> 71305 T154 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 71306 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 71308 T200 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ad02656 name:ZooKeeperConnection Watcher:127.0.0.1:40090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 71308 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 71310 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 71312 T202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d8b69db name:ZooKeeperConnection Watcher:127.0.0.1:40090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 71312 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 71319 T154 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 72323 T154 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45461_pb_
[junit4] 2> 72324 T154 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45461_pb_
[junit4] 2> 72342 T204 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 72342 T204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 72343 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72343 T188 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72343 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72343 T204 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 72345 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 72346 T177 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45461/pb_",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:45461_pb_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 72346 T177 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 72346 T177 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 72349 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 72349 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 72350 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 72349 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 73344 T204 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 73345 T204 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 73346 T204 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 73346 T204 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 73347 T204 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 73348 T204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/collection1/'
[junit4] 2> 73349 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 73350 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 73351 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 73425 T204 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 73479 T204 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 73481 T204 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 73511 T204 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 73935 T204 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 73939 T204 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 73942 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 73948 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 73994 T204 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 73995 T204 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 73995 T204 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-003/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/
[junit4] 2> 73995 T204 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@54b83713
[junit4] 2> 73997 T204 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1
[junit4] 2> 73998 T204 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/index/
[junit4] 2> 73999 T204 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 74001 T204 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/index
[junit4] 2> 74002 T204 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=0.4306640625, floorSegmentMB=1.0390625, forceMergeDeletesPctAllowed=3.007953945489976, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 74037 T204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 74038 T204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 74048 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 74048 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 74049 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 74049 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 74049 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 74050 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 74050 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 74051 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 74051 T204 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 74052 T204 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 74052 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 74053 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 74054 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 74055 T204 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 74056 T204 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 74056 T204 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 74056 T204 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 74057 T204 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 74057 T204 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 74058 T204 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 74059 T204 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 74059 T204 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 74081 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74084 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74086 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74088 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74090 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 74095 T204 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 74095 T204 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 74096 T204 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=56.107421875, floorSegmentMB=0.4912109375, forceMergeDeletesPctAllowed=26.23316267623028, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.28139084101318934
[junit4] 2> 74099 T204 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 74099 T204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 74100 T204 oass.SolrIndexSearcher.<init> Opening Searcher@2fe00fed[collection1] main
[junit4] 2> 74100 T204 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 74101 T204 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 74102 T204 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 74102 T204 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 74103 T204 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 74103 T204 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 74104 T204 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 74105 T204 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 74105 T204 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 74105 T204 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 74111 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fe00fed[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 74118 T204 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 74119 T208 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45461/pb_ collection:collection1 shard:shard2
[junit4] 2> 74120 T154 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 74120 T154 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 74120 T208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 74127 T208 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 74130 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74130 T208 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 74131 T208 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C50 name=collection1 org.apache.solr.core.SolrCore@47c1facb url=http://127.0.0.1:45461/pb_/collection1 node=127.0.0.1:45461_pb_ C50_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:45461/pb_, core=collection1, node_name=127.0.0.1:45461_pb_}
[junit4] 2> 74131 T208 C50 P45461 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45461/pb_/collection1/
[junit4] 2> 74131 T208 C50 P45461 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 74132 T208 C50 P45461 oasc.SyncStrategy.syncToMe http://127.0.0.1:45461/pb_/collection1/ has no replicas
[junit4] 2> 74132 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74132 T208 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45461/pb_/collection1/ shard2
[junit4] 2> 74132 T208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 74135 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74136 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74137 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74136 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74142 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74145 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74248 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74248 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74248 T180 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74293 T208 oasc.ZkController.register We are http://127.0.0.1:45461/pb_/collection1/ and leader is http://127.0.0.1:45461/pb_/collection1/
[junit4] 2> 74294 T208 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45461/pb_
[junit4] 2> 74294 T208 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 74294 T208 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 74294 T208 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 74297 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74297 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74297 T208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74297 T188 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74299 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 74300 T177 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:45461/pb_",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:45461_pb_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 74303 T188 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 74406 T188 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74406 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 74406 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 75674 T154 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 75675 T154 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 75677 T154 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:41438
[junit4] 2> 75678 T154 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 75678 T154 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 75678 T154 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004
[junit4] 2> 75678 T154 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/'
[junit4] 2> 75711 T154 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/solr.xml
[junit4] 2> 75774 T154 oasc.CoreContainer.<init> New CoreContainer 1040609761
[junit4] 2> 75774 T154 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/]
[junit4] 2> 75775 T154 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 75776 T154 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 75776 T154 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 75776 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 75776 T154 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 75777 T154 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 75777 T154 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 75777 T154 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 75777 T154 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 75778 T154 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 75778 T154 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 75778 T154 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 75779 T154 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 75779 T154 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40090/solr
[junit4] 2> 75779 T154 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 75780 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 75782 T219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@191e8a97 name:ZooKeeperConnection Watcher:127.0.0.1:40090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 75782 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 75785 T154 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 75795 T221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@562cf72d name:ZooKeeperConnection Watcher:127.0.0.1:40090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 75795 T154 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 75802 T154 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 76806 T154 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41438_pb_
[junit4] 2> 76807 T154 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41438_pb_
[junit4] 2> 76819 T223 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 76819 T223 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 76820 T188 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76820 T180 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76820 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76820 T223 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 76822 T177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 76823 T177 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:41438/pb_",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41438_pb_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 76823 T177 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 76823 T177 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 76826 T189 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 76827 T189 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 76827 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 76827 T187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 76827 T222 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 77821 T223 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 77822 T223 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 77823 T223 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 77823 T223 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 77824 T223 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 77824 T223 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/collection1/'
[junit4] 2> 77827 T223 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 77828 T223 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 77828 T223 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 77902 T223 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 77956 T223 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 77958 T223 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 77983 T223 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 78440 T223 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 78444 T223 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 78446 T223 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 78452 T223 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 78496 T223 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 78496 T223 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 78497 T223 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-004/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2/
[junit4] 2> 78497 T223 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@54b83713
[junit4] 2> 78499 T223 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2
[junit4] 2> 78500 T223 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2/index/
[junit4] 2> 78501 T223 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 78502 T223 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2/index
[junit4] 2> 78503 T223 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=0.4306640625, floorSegmentMB=1.0390625, forceMergeDeletesPctAllowed=3.007953945489976, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 78515 T223 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-A07445A293ED20EE-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 78515 T223 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 78525 T223 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 78525 T223 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 78526 T223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 78527 T223 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 78527 T223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 78527 T223 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 78528 T223 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 78528 T223 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 78529 T223 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 78530 T223 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 78530 T223 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 78531 T223 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 78532 T223 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 78533 T223 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 78533 T223 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 78534 T223 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 78534 T223 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 78535 T223 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 78535 T223 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 78535 T223 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 78536
[...truncated too long message...]
it4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 4118055 T16920 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 4118057 T15349 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/m_gef/be,null}
[junit4] 2> 4120433 T15349 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 4120435 T15349 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39587 39587
[junit4] 2> 4120617 T15350 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39587 39587
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MultiThreadedOCPTest -Dtests.method=testDistribSearch -Dtests.seed=A07445A293ED20EE -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh_TW -Dtests.timezone=Europe/Stockholm -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 57.0s J0 | MultiThreadedOCPTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: We have a failed SPLITSHARD task
[junit4] > at __randomizedtesting.SeedInfo.seed([A07445A293ED20EE:2192CBBAE4B240D2]:0)
[junit4] > at org.apache.solr.cloud.MultiThreadedOCPTest.testTaskExclusivity(MultiThreadedOCPTest.java:125)
[junit4] > at org.apache.solr.cloud.MultiThreadedOCPTest.doTest(MultiThreadedOCPTest.java:71)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 4120628 T15349 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.cloud.MultiThreadedOCPTest-A07445A293ED20EE-001
[junit4] 2> 56998 T15348 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=zh_TW, timezone=Europe/Stockholm
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=1,free=210295832,total=467140608
[junit4] 2> NOTE: All tests run in this JVM: [DirectSolrSpellCheckerTest, HdfsCollectionsAPIDistributedZkTest, TermsComponentTest, EchoParamsTest, TestDefaultSearchFieldResource, ZkControllerTest, TestAnalyzedSuggestions, TestStressVersions, TestFieldTypeCollectionResource, ChaosMonkeyNothingIsSafeTest, SuggesterWFSTTest, TestCollationFieldDocValues, RAMDirectoryFactoryTest, CoreAdminRequestStatusTest, SolrCoreCheckLockOnStartupTest, OverseerTest, PrimUtilsTest, DeleteInactiveReplicaTest, TestHighlightDedupGrouping, PrimitiveFieldTypeTest, HdfsSyncSliceTest, ChangedSchemaMergeTest, CursorMarkTest, TestManagedSynonymFilterFactory, ClusterStateUpdateTest, XsltUpdateRequestHandlerTest, TestDefaultSimilarityFactory, AnalyticsMergeStrategyTest, TestBinaryResponseWriter, PeerSyncTest, TestComponentsName, UnloadDistributedZkTest, TestFoldingMultitermQuery, PluginInfoTest, TestCodecSupport, TestStressRecovery, TestInfoStreamLogging, PingRequestHandlerTest, XmlUpdateRequestHandlerTest, TestDFRSimilarityFactory, DirectUpdateHandlerTest, SpellCheckCollatorTest, SolrCmdDistributorTest, TestFastOutputStream, DocumentBuilderTest, ReplicationFactorTest, MBeansHandlerTest, TestSolrDeletionPolicy1, TestUtils, TestSolrIndexConfig, ZkNodePropsTest, TestSolrDeletionPolicy2, AssignTest, AbstractAnalyticsStatsTest, TestExceedMaxTermLength, TestSolr4Spatial, TestFastLRUCache, SuggestComponentTest, BadIndexSchemaTest, TestSolrXmlPersistence, SimpleFacetsTest, OverseerCollectionProcessorTest, TestMissingGroups, ReturnFieldsTest, MoreLikeThisHandlerTest, FastVectorHighlighterTest, NoCacheHeaderTest, TestStressReorder, TestPartialUpdateDeduplication, JsonLoaderTest, ShardRoutingTest, TestFastWriter, HdfsBasicDistributedZk2Test, TestSchemaVersionResource, DocExpirationUpdateProcessorFactoryTest, TestFuzzyAnalyzedSuggestions, TestFieldResource, TestSchemaResource, SolrIndexSplitterTest, SuggesterTSTTest, TestAnalyzeInfixSuggestions, TimeZoneUtilsTest, TestLuceneMatchVersion, ExternalFileFieldSortTest, DisMaxRequestHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestComplexPhraseQParserPlugin, TestManagedResource, HighlighterMaxOffsetTest, TestReplicationHandlerBackup, TestLeaderElectionZkExpiry, TestJmxIntegration, TestCopyFieldCollectionResource, SolrXmlInZkTest, ShowFileRequestHandlerTest, HighlighterConfigTest, HdfsWriteToMultipleCollectionsTest, UpdateParamsTest, RegexBoostProcessorTest, TestSolrXml, TestAddFieldRealTimeGet, OpenExchangeRatesOrgProviderTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, TestRandomFaceting, LeaderElectionTest, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, TestRealTimeGet, TestReload, DistributedTermsComponentTest, TestRangeQuery, SpellCheckComponentTest, TestGroupingSearch, QueryElevationComponentTest, TestSort, BasicFunctionalityTest, HighlighterTest, CurrencyFieldOpenExchangeTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, CoreAdminHandlerTest, SuggesterTest, TestTrie, PolyFieldTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, DocValuesMultiTest, DistanceFunctionTest, SolrInfoMBeanTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, LukeRequestHandlerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, TestOmitPositions, FileBasedSpellCheckerTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, TestCSVResponseWriter, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, SOLR749Test, TestQuerySenderListener, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, SampleTest, TestDocSet, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestConfig, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, ResourceLoaderTest, ScriptEngineTest, URLClassifyProcessorTest, PreAnalyzedFieldTest, DateFieldTest, SpellingQueryConverterTest, DOMUtilTest, ClusterStateTest, TestSolrJ, TestDocumentBuilder, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, TestRTGBase, CursorPagingTest, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestTolerantSearch, NoFacetTest, ExpressionTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, FieldFacetTest, RangeFacetTest, FunctionTest, CustomCollectionTest, DistribDocExpirationUpdateProcessorTest, MultiThreadedOCPTest]
[junit4] Completed on J0 in 57.28s, 1 test, 1 failure <<< FAILURES!
[...truncated 234 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:481: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:454: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 422 suites, 1761 tests, 2 suite-level errors, 1 failure, 51 ignored (9 assumptions)
Total time: 200 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #592
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 719889489 bytes
Compression is 0.0%
Took 2 min 5 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 598 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/598/
6 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MultiThreadedOCPTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MultiThreadedOCPTest
Error Message:
Captured an uncaught exception in thread: Thread[id=328, name=qtp149694054-328, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=328, name=qtp149694054-328, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MultiThreadedOCPTest
Error Message:
Captured an uncaught exception in thread: Thread[id=232, name=qtp1627420012-232, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=232, name=qtp1627420012-232, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup: 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
FAILED: org.apache.solr.cloud.MultiThreadedOCPTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
Build Log:
[...truncated 11528 lines...]
[junit4] Suite: org.apache.solr.handler.TestReplicationHandlerBackup
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/init-core-data-001
[junit4] 2> 1720889 T3189 oas.SolrTestCaseJ4.setUp ###Starting doTestBackup
[junit4] 2> 1720907 T3189 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1720910 T3189 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60236
[junit4] 2> 1720911 T3189 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1720911 T3189 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1720911 T3189 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001
[junit4] 2> 1720912 T3189 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/'
[junit4] 2> 1720956 T3189 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/solr.xml
[junit4] 2> 1721034 T3189 oasc.CoreContainer.<init> New CoreContainer 1708505120
[junit4] 2> 1721034 T3189 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/]
[junit4] 2> 1721035 T3189 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1721036 T3189 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1721036 T3189 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1721036 T3189 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1721037 T3189 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1721037 T3189 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1721037 T3189 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1721038 T3189 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1721038 T3189 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1721039 T3189 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1721039 T3189 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1721039 T3189 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1721040 T3189 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1721045 T3199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/'
[junit4] 2> 1721088 T3199 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1721126 T3199 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1721127 T3199 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1721134 T3199 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1721150 T3199 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1721151 T3199 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/
[junit4] 2> 1721152 T3199 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1721152 T3199 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/
[junit4] 2> 1721152 T3199 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 1721154 T3199 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data
[junit4] 2> 1721155 T3199 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index/
[junit4] 2> 1721155 T3199 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
[junit4] 2> 1721156 T3199 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index
[junit4] 2> 1721156 T3199 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=71.3056640625, floorSegmentMB=0.4873046875, forceMergeDeletesPctAllowed=15.706223013408033, segmentsPerTier=32.0, maxCFSSegmentSizeMB=0.7490234375, noCFSRatio=0.0
[junit4] 2> 1721157 T3199 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@8eba254 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71968fe0),segFN=segments_1,generation=1}
[junit4] 2> 1721157 T3199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1721159 T3199 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1721159 T3199 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1721159 T3199 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1721160 T3199 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1721160 T3199 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1721160 T3199 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1721160 T3199 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1721161 T3199 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1721161 T3199 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1721180 T3199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1721182 T3199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1721183 T3199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1721185 T3199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1721187 T3199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1721189 T3199 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1721190 T3199 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1721190 T3199 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6265002443195115]
[junit4] 2> 1721191 T3199 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@8eba254 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71968fe0),segFN=segments_1,generation=1}
[junit4] 2> 1721191 T3199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1721191 T3199 oass.SolrIndexSearcher.<init> Opening Searcher@59f64efa[collection1] main
[junit4] 2> 1721192 T3199 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/conf
[junit4] 2> 1721192 T3199 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/conf}
[junit4] 2> 1721192 T3199 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/conf
[junit4] 2> 1721193 T3199 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
[junit4] 2> 1721194 T3199 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/conf
[junit4] 2> 1721194 T3199 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1721194 T3199 oash.ReplicationHandler.inform Replication enabled for following config files: schema-replication2.xml:schema.xml
[junit4] 2> 1721195 T3199 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4] 2> 1721195 T3200 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@59f64efa[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 1721195 T3199 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1721196 T3189 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1721196 T3189 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> ASYNC NEW_CORE C1234 name=collection1 org.apache.solr.core.SolrCore@7d1d22be
[junit4] 2> 1721198 T3192 C1234 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4] 2> 1721199 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 1
[junit4] 2> 1721202 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 0
[junit4] 2> 1721205 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 0
[junit4] 2> 1721207 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 0
[junit4] 2> 1721209 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 0
[junit4] 2> 1721211 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 0
[junit4] 2> 1721213 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 0
[junit4] 2> 1721215 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 0
[junit4] 2> 1721217 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 0
[junit4] 2> 1721219 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 0
[junit4] 2> 1721221 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 0
[junit4] 2> 1721223 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 0
[junit4] 2> 1721225 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 0
[junit4] 2> 1721227 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 0
[junit4] 2> 1721229 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 0
[junit4] 2> 1721231 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 0
[junit4] 2> 1721233 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 0
[junit4] 2> 1721235 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 0
[junit4] 2> 1721237 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 0
[junit4] 2> 1721239 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 0
[junit4] 2> 1721241 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 0
[junit4] 2> 1721243 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 0
[junit4] 2> 1721245 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 0
[junit4] 2> 1721247 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 0
[junit4] 2> 1721249 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 0
[junit4] 2> 1721251 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 0
[junit4] 2> 1721253 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 0
[junit4] 2> 1721255 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26]} 0 0
[junit4] 2> 1721257 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27]} 0 0
[junit4] 2> 1721259 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[28]} 0 0
[junit4] 2> 1721261 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[29]} 0 0
[junit4] 2> 1721263 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[30]} 0 0
[junit4] 2> 1721265 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[31]} 0 0
[junit4] 2> 1721267 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[32]} 0 0
[junit4] 2> 1721269 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[33]} 0 0
[junit4] 2> 1721271 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[34]} 0 0
[junit4] 2> 1721273 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[35]} 0 0
[junit4] 2> 1721275 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36]} 0 0
[junit4] 2> 1721277 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[37]} 0 0
[junit4] 2> 1721279 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[38]} 0 0
[junit4] 2> 1721281 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39]} 0 0
[junit4] 2> 1721283 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[40]} 0 0
[junit4] 2> 1721285 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41]} 0 0
[junit4] 2> 1721287 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[42]} 0 0
[junit4] 2> 1721289 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[43]} 0 0
[junit4] 2> 1721291 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44]} 0 0
[junit4] 2> 1721293 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45]} 0 0
[junit4] 2> 1721295 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[46]} 0 0
[junit4] 2> 1721297 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47]} 0 0
[junit4] 2> 1721299 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48]} 0 0
[junit4] 2> 1721301 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[49]} 0 0
[junit4] 2> 1721303 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[50]} 0 0
[junit4] 2> 1721305 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[51]} 0 0
[junit4] 2> 1721307 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[52]} 0 0
[junit4] 2> 1721309 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[53]} 0 0
[junit4] 2> 1721311 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54]} 0 0
[junit4] 2> 1721313 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[55]} 0 0
[junit4] 2> 1721315 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56]} 0 0
[junit4] 2> 1721317 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[57]} 0 0
[junit4] 2> 1721319 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[58]} 0 0
[junit4] 2> 1721321 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[59]} 0 0
[junit4] 2> 1721323 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[60]} 0 0
[junit4] 2> 1721325 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[61]} 0 0
[junit4] 2> 1721327 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[62]} 0 0
[junit4] 2> 1721329 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63]} 0 0
[junit4] 2> 1721331 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64]} 0 0
[junit4] 2> 1721333 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[65]} 0 0
[junit4] 2> 1721335 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66]} 0 0
[junit4] 2> 1721337 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[67]} 0 0
[junit4] 2> 1721339 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[68]} 0 0
[junit4] 2> 1721341 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[69]} 0 0
[junit4] 2> 1721343 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[70]} 0 0
[junit4] 2> 1721345 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[71]} 0 0
[junit4] 2> 1721347 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[72]} 0 0
[junit4] 2> 1721349 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[73]} 0 0
[junit4] 2> 1721351 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[74]} 0 0
[junit4] 2> 1721353 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[75]} 0 0
[junit4] 2> 1721355 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[76]} 0 0
[junit4] 2> 1721357 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[77]} 0 0
[junit4] 2> 1721359 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[78]} 0 0
[junit4] 2> 1721361 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[79]} 0 0
[junit4] 2> 1721363 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[80]} 0 0
[junit4] 2> 1721365 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[81]} 0 0
[junit4] 2> 1721367 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[82]} 0 0
[junit4] 2> 1721369 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[83]} 0 0
[junit4] 2> 1721371 T3192 C1234 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1721373 T3192 C1234 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@8eba254 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71968fe0),segFN=segments_1,generation=1}
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@8eba254 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@71968fe0),segFN=segments_2,generation=2}
[junit4] 2> 1721374 T3192 C1234 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 1721376 T3192 C1234 oass.SolrIndexSearcher.<init> Opening Searcher@7effa51f[collection1] main
[junit4] 2> 1721377 T3192 C1234 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4] 2> 1721377 T3200 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7effa51f[collection1] main{UninvertingDirectoryReader(Uninverting(_0(5.0):c84))}
[junit4] 2> 1721377 T3192 C1234 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={waitSearcher=true&softCommit=false&commit=true&wt=javabin&version=2} {commit=} 0 6
[junit4] 2> 1721382 T3194 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1
[junit4] 2> 1721395 T3193 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={name=vdookyxjtgmxqws&command=backup} status=0 QTime=14
[junit4] 2> 1721395 T3204 oash.SnapShooter.createSnapshot Creating backup snapshot...
[junit4] 2> 1721408 T3204 oash.SnapShooter.createSnapshot Done creating backup snapshot: vdookyxjtgmxqws
[junit4] 2> 1721584 T3193 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0
[junit4] 2> 1722593 T3193 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1
[junit4] 2> 1722594 T3194 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={name=ribyuvugqsowycnhiq&command=backup} status=0 QTime=2
[junit4] 2> 1722594 T3206 oash.SnapShooter.createSnapshot Creating backup snapshot...
[junit4] 2> 1722602 T3206 oash.SnapShooter.createSnapshot Done creating backup snapshot: ribyuvugqsowycnhiq
[junit4] 2> 1722795 T3194 C1234 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0
[junit4] 2> ASYNC NEW_CORE C1235 name=collection1 org.apache.solr.core.SolrCore@7d1d22be
[junit4] 2> 1722809 T3193 C1235 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0
[junit4] 2> 1722814 T3194 C1235 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={name=vdookyxjtgmxqws&command=deletebackup} status=0 QTime=6
[junit4] 2> 1722814 T3208 oash.SnapShooter.deleteNamedSnapshot Deleting snapshot: vdookyxjtgmxqws
[junit4] 2> 1723011 T3194 C1235 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0
[junit4] 2> 1723013 T3194 C1235 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1
[junit4] 2> 1723013 T3189 oas.SolrTestCaseJ4.tearDown ###Ending doTestBackup
[junit4] 2> 1723013 T3210 oash.SnapShooter.deleteNamedSnapshot Deleting snapshot: ribyuvugqsowycnhiq
[junit4] 2> 1723013 T3193 C1235 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={name=ribyuvugqsowycnhiq&command=deletebackup} status=0 QTime=0
[junit4] 2> 1723014 T3189 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1708505120
[junit4] 2> 1723015 T3189 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4] 2> 1723016 T3193 C1235 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
[junit4] 2> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
[junit4] 2> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
[junit4] 2> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] 2> at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
[junit4] 2> at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
[junit4] 2> at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
[junit4] 2> at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
[junit4] 2> at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:769)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:428)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:368)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
[junit4] 2> at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
[junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
[junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
[junit4] 2> at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
[junit4] 2> at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1723017 T3193 C1235 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
[junit4] 2> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
[junit4] 2> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
[junit4] 2> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
[junit4] 2> at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
[junit4] 2> at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
[junit4] 2> at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
[junit4] 2> at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
[junit4] 2> at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:769)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:428)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:368)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
[junit4] 2> at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
[junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
[junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
[junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
[junit4] 2> at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
[junit4] 2> at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 1723019 T3193 C1235 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7d1d22be
[junit4] 2> 1723019 T3193 C1235 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=84,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4] 2> 1723020 T3193 C1235 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 1723020 T3193 C1235 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 1723020 T3193 C1235 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 1723021 T3193 C1235 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 1723021 T3193 C1235 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 1723021 T3193 C1235 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index;done=false>>]
[junit4] 2> 1723022 T3193 C1235 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data/index
[junit4] 2> 1723022 T3193 C1235 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data;done=false>>]
[junit4] 2> 1723022 T3193 C1235 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./temp/solr.handler.TestReplicationHandlerBackup-936281279AE7F682-001/solr-instance-001/collection1/data
[junit4] 2> 1728026 T3189 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 7147 T3188 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> 27233 T3188 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
[junit4] 2> 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4] 2> at java.net.Socket.connect(Socket.java:528)
[junit4] 2> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] 2> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] 2> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] 2> at java.net.URL.openStream(URL.java:1037)
[junit4] 2> at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] 2> 27234 T3188 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] 2> 30240 T3188 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4] 2> at java.net.Socket.connect(Socket.java:528)
[junit4] 2> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] 2> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] 2> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] 2> at java.net.URL.openStream(URL.java:1037)
[junit4] 2> at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=hu, timezone=Asia/Kashgar
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=2,free=219235216,total=311427072
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudInspectUtil, SpellPossibilityIteratorTest, TestJmxIntegration, FastVectorHighlighterTest, XsltUpdateRequestHandlerTest, MigrateRouteKeyTest, TestDefaultSimilarityFactory, TestRandomDVFaceting, TermsComponentTest, BadIndexSchemaTest, SolrXmlInZkTest, BlockCacheTest, PeerSyncTest, HdfsBasicDistributedZk2Test, TestManagedSynonymFilterFactory, ZkControllerTest, ExternalFileFieldSortTest, SpellCheckComponentTest, ParsingFieldUpdateProcessorsTest, SuggestComponentTest, ChaosMonkeyNothingIsSafeTest, TestRangeQuery, TestIntervalFaceting, DistributedDebugComponentTest, TestSchemaSimilarityResource, TestDFRSimilarityFactory, TestClassNameShortening, TestArbitraryIndexDir, TestDynamicFieldResource, TestFieldResource, TestDocBasedVersionConstraints, SolrCoreCheckLockOnStartupTest, SolrCmdDistributorTest, TestCopyFieldCollectionResource, DirectSolrSpellCheckerTest, ZkNodePropsTest, ClusterStateUpdateTest, PreAnalyzedFieldTest, TestSolrJ, HighlighterConfigTest, TestReplicationHandler, IndexSchemaTest, TestSearcherReuse, TestCodecSupport, BasicFunctionalityTest, ZkCLITest, StatelessScriptUpdateProcessorFactoryTest, PreAnalyzedUpdateProcessorTest, TestReloadAndDeleteDocs, EchoParamsTest, OverseerCollectionProcessorTest, TestSolrXmlPersistence, HighlighterTest, AnalysisErrorHandlingTest, LeaderElectionTest, TestSchemaVersionResource, TestSolrCoreProperties, TestSolrDeletionPolicy1, WordBreakSolrSpellCheckerTest, FileUtilsTest, TestTolerantSearch, ResponseLogComponentTest, TestStressRecovery, FunctionTest, TestHighlightDedupGrouping, TestSolrDeletionPolicy2, QueryEqualityTest, TestReplicationHandlerBackup]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestReplicationHandlerBackup -Dtests.seed=936281279AE7F682 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=hu -Dtests.timezone=Asia/Kashgar -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J0 | TestReplicationHandlerBackup (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
[junit4] > 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] > at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3209, name=Thread-1278, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:314)
[junit4] > at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
[junit4] Completed on J0 in 30.28s, 1 test, 2 errors <<< FAILURES!
[...truncated 818 lines...]
[junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
[junit4] 2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/init-core-data-001
[junit4] 2> 3859 T11 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 3962 T11 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 4017 T11 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 4026 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 4035 T12 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 4336 T11 oasc.ZkTestServer.run start zk server on port:61636
[junit4] 2> 4613 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 4790 T18 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f9bc97 name:ZooKeeperConnection Watcher:127.0.0.1:61636 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 4792 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 4799 T11 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 4926 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 4929 T20 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a429aa2 name:ZooKeeperConnection Watcher:127.0.0.1:61636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 4930 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 4955 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 4960 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 4965 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 4970 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 4976 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 5046 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 5053 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 5054 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 5059 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 5061 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 5066 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 5067 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 5072 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 5074 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 5078 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 5080 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 5084 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 5086 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 5091 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 5092 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 5097 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 5099 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 5103 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 5105 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 5109 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 5111 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 5129 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 5132 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@302c28cc name:ZooKeeperConnection Watcher:127.0.0.1:61636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 5133 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 8789 T11 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 9359 T11 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 9407 T11 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:32014
[junit4] 2> 10281 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 10437 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 10438 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002
[junit4] 2> 10439 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/'
[junit4] 2> 11442 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/solr.xml
[junit4] 2> 11751 T11 oasc.CoreContainer.<init> New CoreContainer 788197590
[junit4] 2> 11759 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/]
[junit4] 2> 11911 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 11912 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 11912 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 11913 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 11914 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 11914 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 11915 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 11915 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 11916 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 12066 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 12091 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 12098 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 12108 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 12109 T11 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:61636/solr
[junit4] 2> 12153 T11 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 12154 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 12159 T34 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77eea508 name:ZooKeeperConnection Watcher:127.0.0.1:61636 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 12160 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 12178 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 12180 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67b8e751 name:ZooKeeperConnection Watcher:127.0.0.1:61636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 12181 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 12204 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 12220 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 12230 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 12235 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 12241 T11 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 12284 T11 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 12287 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32014_
[junit4] 2> 12295 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32014_
[junit4] 2> 12306 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 12310 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 12332 T11 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:32014_
[junit4] 2> 12333 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 12337 T11 oasc.Overseer.start Overseer (id=92228166816432132-127.0.0.1:32014_-n_0000000000) starting
[junit4] 2> 12363 T11 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 12469 T38 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 12469 T11 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 12473 T11 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 12475 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 12562 T37 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 12906 T39 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 12906 T39 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 12919 T39 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 12928 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12939 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 12941 T37 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:32014",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:32014_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 12941 T37 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 13003 T37 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 13008 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 13030 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 13921 T39 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 13921 T39 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 13923 T39 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 13924 T39 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 13926 T39 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 13927 T39 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/collection1/'
[junit4] 2> 13930 T39 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 13932 T39 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 13933 T39 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 14282 T39 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 14706 T39 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 14770 T39 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 14973 T39 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 17235 T39 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 17290 T39 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 17293 T39 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 18101 T39 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 18126 T39 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 18132 T39 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 18138 T39 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 18139 T39 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 18139 T39 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 18141 T39 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 18142 T39 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 18146 T39 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 18265 T39 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 18302 T39 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-001/control/data/
[junit4] 2> 18311 T39 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d2c041a
[junit4] 2> 18527 T39 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-001/control/data
[junit4] 2> 18527 T39 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-001/control/data/index/
[junit4] 2> 18528 T39 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 18560 T39 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.MultiThreadedOCPTest-936281279AE7F682-001/tempDir-001/control/data/index
[junit4] 2> 18563 T39 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=20.2890625, floorSegmentMB=1.1396484375, forceMergeDeletesPctAllowed=22.888894824493903, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3218127037386259
[junit4] 2> 18763 T39 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3ffd1bed lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19292ff8),segFN=segments_1,generation=1}
[junit4] 2> 18767 T39 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 19708 T39 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 19877 T39 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 19897 T39 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 19897 T39 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 19898 T39 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 19898 T39 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 20069 T39 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 20070 T39 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 20070 T39 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 20105 T39 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 20106 T39 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 20107 T39 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 20109 T39 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 20149 T39 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 20180 T39 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 20180 T39 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 20191 T39 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 20219 T39 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 20220 T39 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 20225 T39 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 20265 T39 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 20303 T39 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 20474 T39 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 20502 T39 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 20504 T39 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 20506 T39 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 20508 T39 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 20626 T39 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 20627 T39 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 20629 T39 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=21.779296875, floorSegmentMB=1.29296875, forceMergeDeletesPctAllowed=25.258018587109554, segmentsPerTier=12.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6165322401803253
[junit4] 2> 20645 T39 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@3ffd1bed lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19292ff8),segFN=segments_1,generation=1}
[junit4] 2> 20645 T39 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 20741 T39 oass.SolrIndexSearcher.<init> Opening Searcher@57944a64[collection1] main
[junit4] 2> 20790 T39 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 20792 T39 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 20792 T39 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 20808 T39 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 20809 T39 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 20935 T39 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 20936 T39 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 20937 T39 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 20938 T39 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 20941 T39 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 20942 T39 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 20942 T39 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 21177 T41 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57944a64[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 21192 T39 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 21199 T44 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:32014 collection:control_collection shard:shard1
[junit4] 2> 21200 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 21201 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 21234 T44 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 21241 T44 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 21244 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21244 T44 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 21245 T44 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 21246 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 21258 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21259 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> ASYNC NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@2154cbb6 url=https://127.0.0.1:32014/collection1 node=127.0.0.1:32014_ C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:32014, core=collection1, node_name=127.0.0.1:32014_}
[junit4] 2> 21259 T44 C0 P32014 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:32014/collection1/
[junit4] 2> 21266 T44 C0 P32014 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 21266 T44 C0 P32014 oasc.SyncStrategy.syncToMe https://127.0.0.1:32014/collection1/ has no replicas
[junit4] 2> 21267 T44 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:32014/collection1/ shard1
[junit4] 2> 21278 T44 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 21284 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21286 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 21290 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21336 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 21337 T46 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@356ebe7 name:ZooKeeperConnection Watcher:127.0.0.1:61636/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 21338 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 21339 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 21345 T11 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 21393 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 21393 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 21439 T44 oasc.ZkController.register We are https://127.0.0.1:32014/collection1/ and leader is https://127.0.0.1:32014/collection1/
[junit4] 2> 21446 T44 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:32014
[junit4] 2> 21446 T44 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 21447 T44 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 21447 T44 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 21449 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21450 T49 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21450 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 21450 T48 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 21452 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 21453 T37 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4]
[...truncated too long message...]
t]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 83) Thread[id=336, name=OverseerThreadFactory-4-thread-5, state=WAITING, group=Overseer collection creation process.]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] 2> at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 84) Thread[id=200, name=parallelCoreAdminExecutor-32-thread-1, state=WAITING, group=TGRP-MultiThreadedOCPTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 85) Thread[id=286, name=searcherExecutor-74-thread-1, state=WAITING, group=TGRP-MultiThreadedOCPTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 86) Thread[id=158, name=OverseerThreadFactory-4-thread-1, state=WAITING, group=Overseer collection creation process.]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4] 2> at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=cs, timezone=Atlantic/Faroe
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=90,free=143068944,total=310378496
[junit4] 2> NOTE: All tests run in this JVM: [MultiThreadedOCPTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MultiThreadedOCPTest -Dtests.seed=936281279AE7F682 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=cs -Dtests.timezone=Atlantic/Faroe -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J1 | MultiThreadedOCPTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=328, name=qtp149694054-328, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: unable to create new native thread
[junit4] > at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
[junit4] > at java.lang.Thread.start0(Native Method)
[junit4] > at java.lang.Thread.start(Thread.java:714)
[junit4] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
[junit4] > at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
[junit4] > at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
[junit4] > at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
[junit4] > at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=232, name=qtp1627420012-232, state=RUNNABLE, group=TGRP-MultiThreadedOCPTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: unable to create new native thread
[junit4] > at __randomizedtesting.SeedInfo.seed([936281279AE7F682]:0)
[junit4] > at java.lang.Thread.start0(Native Method)
[junit4] > at java.lang.Thread.start(Thread.java:714)
[junit4] > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
[junit4] > at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
[junit4] > at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
[junit4] > at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
[junit4] > at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] Completed on J1 in 7226.10s, 1 test, 4 errors <<< FAILURES!
[...truncated 323 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1298: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:922: There were test failures: 422 suites, 1757 tests, 5 suite-level errors, 1 error, 1275 ignored (3 assumptions)
Total time: 251 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #592
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 184747043 bytes
Compression is 0.0%
Took 33 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure