You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/01/17 06:15:58 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11623 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11623/
Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Should have had a good message here

Stack Trace:
java.lang.AssertionError: Should have had a good message here
	at __randomizedtesting.SeedInfo.seed([409FEEBDB49636E3:C17960A5C3C956DF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:138)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 8752 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/init-core-data-001
   [junit4]   2> 135483 T1460 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 135484 T1460 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 135487 T1460 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 135487 T1460 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 135488 T1461 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 135588 T1460 oasc.ZkTestServer.run start zk server on port:39666
   [junit4]   2> 135588 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135589 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135592 T1468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d243d9 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135593 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135593 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135593 T1460 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 135596 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135596 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135598 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c1241 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135598 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135598 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135598 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 135600 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 135601 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 135602 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 135604 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 135604 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 135606 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 135606 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 135607 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135607 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135609 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 135609 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 135610 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 135611 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 135612 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 135612 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 135613 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 135614 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 135615 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 135615 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 135616 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 135617 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 135618 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 135618 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 135619 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 135619 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 135872 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 135874 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@1f9b78e{/,null,AVAILABLE}
   [junit4]   2> 135880 T1460 oejs.AbstractConnector.doStart Started ServerConnector@f250e4{HTTP/1.1}{127.0.0.1:36085}
   [junit4]   2> 135880 T1460 oejs.Server.doStart Started @137156ms
   [junit4]   2> 135880 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/'
   [junit4]   2> 135892 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/solr.xml
   [junit4]   2> 135905 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135906 T1460 oasc.CoreContainer.<init> New CoreContainer 21422377
   [junit4]   2> 135906 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/]
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 135909 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 135910 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 135910 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 135910 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 135910 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 135911 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 135911 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135912 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135913 T1485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bb112 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135913 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135914 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135916 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135917 T1488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1790f4c name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135917 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135919 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 135921 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 135922 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 135924 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 135925 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 135927 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 135928 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36085_
   [junit4]   2> 135928 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36085_
   [junit4]   2> 135930 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 135931 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 135932 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 135933 T1460 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36085_
   [junit4]   2> 135933 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 135934 T1460 oasc.Overseer.start Overseer (id=93157454132019203-127.0.0.1:36085_-n_0000000000) starting
   [junit4]   2> 135937 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 135943 T1460 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 135944 T1490 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 135945 T1460 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 135946 T1460 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 135948 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 135950 T1489 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 135951 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135958 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/conf/
   [junit4]   2> 136076 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 136077 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 136077 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 136077 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136078 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136079 T1495 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c6501 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136079 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136080 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136081 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 136083 T1460 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 136084 T1460 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 136084 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136084 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136085 T1498 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161dc0f name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136086 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136086 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136088 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 136090 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 136090 T1489 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 136093 T1489 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 136094 T1495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 136095 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 136333 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 136334 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 136336 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7f3edf{/,null,AVAILABLE}
   [junit4]   2> 136337 T1460 oejs.AbstractConnector.doStart Started ServerConnector@15db3d1{HTTP/1.1}{127.0.0.1:52980}
   [junit4]   2> 136337 T1460 oejs.Server.doStart Started @137613ms
   [junit4]   2> 136337 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/'
   [junit4]   2> 136348 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/solr.xml
   [junit4]   2> 136354 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 136354 T1460 oasc.CoreContainer.<init> New CoreContainer 18741746
   [junit4]   2> 136355 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/]
   [junit4]   2> 136355 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 136355 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 136358 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 136358 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 136358 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 136359 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 136359 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 136359 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 136359 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 136360 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136360 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136362 T1512 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f8a8bb name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136362 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136363 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136365 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136366 T1515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8c6fd name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136367 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136371 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 137375 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52980_
   [junit4]   2> 137377 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52980_
   [junit4]   2> 137381 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 137385 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 137391 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/conf/
   [junit4]   2> 137511 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 137512 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 137512 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 137763 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 137764 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 137765 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@160f77c{/,null,AVAILABLE}
   [junit4]   2> 137766 T1460 oejs.AbstractConnector.doStart Started ServerConnector@bd8155{HTTP/1.1}{127.0.0.1:48955}
   [junit4]   2> 137766 T1460 oejs.Server.doStart Started @139043ms
   [junit4]   2> 137767 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 137767 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 137767 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 137768 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/'
   [junit4]   2> 137779 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/solr.xml
   [junit4]   2> 137784 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 137785 T1460 oasc.CoreContainer.<init> New CoreContainer 3577483
   [junit4]   2> 137785 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/]
   [junit4]   2> 137785 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 137788 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 137788 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 137788 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 137789 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 137789 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 137789 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 137789 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 137790 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 137790 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 137791 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 137792 T1530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc3021 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 137792 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 137793 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 137795 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 137796 T1533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e74805 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 137796 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 137800 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 138803 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48955_
   [junit4]   2> 138804 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48955_
   [junit4]   2> 138809 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 138813 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 138822 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/conf/
   [junit4]   2> 138976 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 138977 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 138977 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 139243 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 139244 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 139245 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@87aa5f{/,null,AVAILABLE}
   [junit4]   2> 139246 T1460 oejs.AbstractConnector.doStart Started ServerConnector@1fe29e8{HTTP/1.1}{127.0.0.1:39630}
   [junit4]   2> 139246 T1460 oejs.Server.doStart Started @140522ms
   [junit4]   2> 139247 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 139247 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 139247 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 139248 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/'
   [junit4]   2> 139259 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/solr.xml
   [junit4]   2> 139264 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 139264 T1460 oasc.CoreContainer.<init> New CoreContainer 3704360
   [junit4]   2> 139265 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/]
   [junit4]   2> 139265 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 139265 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 139268 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 139268 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 139268 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 139269 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 139269 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 139269 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 139269 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 139270 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 139270 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 139272 T1548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3f7dc name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139273 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139273 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 139276 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 139277 T1551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11455c4 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139277 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139282 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 140285 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39630_
   [junit4]   2> 140286 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39630_
   [junit4]   2> 140289 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 140292 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 140299 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/conf/
   [junit4]   2> 140451 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 140452 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 140452 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 140730 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 140731 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 140733 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@f87fa2{/,null,AVAILABLE}
   [junit4]   2> 140733 T1460 oejs.AbstractConnector.doStart Started ServerConnector@3c9fd6{HTTP/1.1}{127.0.0.1:39298}
   [junit4]   2> 140734 T1460 oejs.Server.doStart Started @142010ms
   [junit4]   2> 140734 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/'
   [junit4]   2> 140746 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/solr.xml
   [junit4]   2> 140751 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 140752 T1460 oasc.CoreContainer.<init> New CoreContainer 27949791
   [junit4]   2> 140752 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/]
   [junit4]   2> 140752 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 140755 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 140755 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 140755 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 140756 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 140756 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 140756 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 140756 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 140757 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 140757 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 140758 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 140759 T1566 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ea416 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 140759 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 140760 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 140762 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 140764 T1569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11eb2be name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 140764 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 140768 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141771 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39298_
   [junit4]   2> 141772 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39298_
   [junit4]   2> 141775 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 141777 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 141783 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/conf/
   [junit4]   2> 141932 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 141933 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 141933 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 141935 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 141935 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 141937 T1573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@164e654 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 141937 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 141937 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 141939 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141940 T1460 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
   [junit4]   2> 141941 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 141941 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 141942 T1576 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d675ec name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 141943 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 141943 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 141944 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141951 T1523 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=delLiveColl&action=CREATE&numShards=2&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 141952 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 141953 T1490 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 141955 T1578 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 141956 T1578 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 141956 T1578 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl 
   [junit4]   2> 141957 T1578 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 141959 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 141960 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 141960 T1489 oasco.ClusterStateMutator.createCollection building a new cName: delLiveColl
   [junit4]   2> 141961 T1489 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/delLiveColl/state.json
   [junit4]   2> 141962 T1495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1551 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1533 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1576 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1573 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:39298_
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:36085_
   [junit4]   2> 142163 T1559 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 142169 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:48955_
   [junit4]   2> 142169 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:39630_
   [junit4]   2> 142169 T1559 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
   [junit4]   2> 142171 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142173 T1559 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142173 T1559 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142173 T1559 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142173 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142174 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142175 T1559 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142175 T1559 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142182 T1541 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 142182 T1524 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 142182 T1478 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 142184 T1541 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
   [junit4]   2> 142184 T1524 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
   [junit4]   2> 142185 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142187 T1524 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142187 T1524 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142188 T1524 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142189 T1524 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142189 T1524 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142196 T1541 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142196 T1541 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142197 T1541 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142197 T1478 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
   [junit4]   2> 142198 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39630",
   [junit4]   2> 	  "node_name":"127.0.0.1:39630_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142198 T1541 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142199 T1541 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142199 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39630",
   [junit4]   2> 	  "node_name":"127.0.0.1:39630_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142200 T1478 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142201 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48955",
   [junit4]   2> 	  "node_name":"127.0.0.1:48955_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142202 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48955",
   [junit4]   2> 	  "node_name":"127.0.0.1:48955_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142203 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142204 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36085",
   [junit4]   2> 	  "node_name":"127.0.0.1:36085_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142204 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36085",
   [junit4]   2> 	  "node_name":"127.0.0.1:36085_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142206 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142306 T1489 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 0
   [junit4]   2> 142307 T1569 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142307 T1551 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142307 T1533 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142308 T1569 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142308 T1551 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142308 T1533 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142311 T1478 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142311 T1478 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142312 T1478 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142312 T1478 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142313 T1478 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 142313 T1478 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 142313 T1478 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 142314 T1478 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 142315 T1478 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 142315 T1478 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/'
   [junit4]   2> 142332 T1478 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 142334 T1478 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 142340 T1478 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 142346 T1478 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 142357 T1478 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 142358 T1478 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 142368 T1478 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 142486 T1478 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 142489 T1478 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 142491 T1478 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 142498 T1478 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 142500 T1478 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 142501 T1478 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 142502 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 142502 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 142503 T1478 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 142503 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 142504 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 142504 T1478 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using configuration from collection delLiveColl
   [junit4]   2> 142504 T1478 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 142504 T1478 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/
   [junit4]   2> 142505 T1478 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d29aa9
   [junit4]   2> 142505 T1478 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data
   [junit4]   2> 142506 T1478 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 142506 T1478 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 142506 T1478 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 142507 T1478 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 142507 T1478 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ae757e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13fc62e),segFN=segments_1,generation=1}
   [junit4]   2> 142508 T1478 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 142513 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 142513 T1478 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 142513 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 142516 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 142516 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 142519 T1478 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 142519 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 142520 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142521 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142522 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142523 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142523 T1478 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 142524 T1478 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 142524 T1478 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 142525 T1478 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2255999651279229]
   [junit4]   2> 142525 T1478 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ae757e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13fc62e),segFN=segments_1,generation=1}
   [junit4]   2> 142525 T1478 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 142526 T1478 oass.SolrIndexSearcher.<init> Opening Searcher@8f43c9[delLiveColl_shard1_replica2] main
   [junit4]   2> 142526 T1478 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 142527 T1478 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 142527 T1478 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 142527 T1478 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 142527 T1478 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 142529 T1478 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 142529 T1478 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 142530 T1586 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@8f43c9[delLiveColl_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 142530 T1478 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 142530 T1478 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:36085 collection:delLiveColl shard:shard1
   [junit4]   2> 142531 T1478 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 142534 T1478 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 142535 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142535 T1478 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999ms
   [junit4]   2> 142535 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl"} current state version: 2
   [junit4]   2> 142637 T1489 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 1
   [junit4]   2> 142637 T1551 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1569 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1488 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1569 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142637 T1533 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1551 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142639 T1533 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142639 T1488 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 143175 T1559 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 143176 T1559 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143176 T1559 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143176 T1559 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143177 T1559 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143177 T1559 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/'
   [junit4]   2> 143187 T1559 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143189 T1559 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143189 T1524 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 143190 T1524 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143190 T1524 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143190 T1524 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143191 T1524 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143191 T1524 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/delLiveColl_shard2_replica1/'
   [junit4]   2> 143193 T1559 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143199 T1541 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 143199 T1541 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143200 T1541 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143200 T1541 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143201 T1541 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143201 T1541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/delLiveColl_shard2_replica2/'
   [junit4]   2> 143202 T1559 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143207 T1524 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143211 T1559 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143212 T1559 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143212 T1524 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143213 T1541 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143215 T1541 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143216 T1559 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 143218 T1524 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143219 T1541 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143224 T1524 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143225 T1541 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143238 T1524 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143239 T1541 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143239 T1524 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143240 T1541 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143246 T1524 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 143246 T1541 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 143343 T1559 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 143349 T1559 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 143351 T1559 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 143365 T1559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 143368 T1559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 143370 T1559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 143372 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 143372 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 143373 T1559 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using configuration from collection delLiveColl
   [junit4]   2> 143373 T1559 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 143373 T1559 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/
   [junit4]   2> 143374 T1559 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d29aa9
   [junit4]   2> 143374 T1559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data
   [junit4]   2> 143375 T1559 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 143375 T1559 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 143375 T1559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 143376 T1559 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 143377 T1559 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@123db4b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@215677),segFN=segments_1,generation=1}
   [junit4]   2> 143377 T1559 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 143382 T1559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 143382 T1559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 143383 T1559 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 143383 T1559 

[...truncated too long message...]

iveColl_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@119fc6e
   [junit4]   2> 153607 T1460 oasc.ZkController.unRegisterConfListener  a listener was removed because of core close
   [junit4]   2> 153607 T1460 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 153607 T1460 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 153608 T1460 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 153608 T1460 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 153609 T1460 oasc.SolrCore.closeSearcher [delLiveColl_shard1_replica1] Closing main searcher on request.
   [junit4]   2> 153610 T1625 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 16 
   [junit4]   2> 153611 T1625 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 153611 T1625 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 153612 T1625 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 16
   [junit4]   2> 153613 T1607 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 153621 T1460 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 153621 T1460 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index;done=false>>]
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data;done=false>>]
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data
   [junit4]   2> 153623 T1460 oasc.Overseer.close Overseer (id=93157454132019213-127.0.0.1:39298_-n_0000000004) closing
   [junit4]   2> 153623 T1625 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:39298_
   [junit4]   2> 155124 T1607 oascc.ZkStateReader$7.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 155124 T1569 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 155125 T1460 oejs.AbstractConnector.doStop Stopped ServerConnector@3c9fd6{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 155126 T1460 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@f87fa2{/,null,UNAVAILABLE}
   [junit4]   2> 155127 T1460 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 155128 T1460 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39666 39666
   [junit4]   2> 155671 T1461 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39666 39666
   [junit4]   2> 155674 T1461 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/aliases.json
   [junit4]   2> 		8	/solr/clusterstate.json
   [junit4]   2> 		4	/solr/collections/delLiveColl/state.json
   [junit4]   2> 		4	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=409FEEBDB49636E3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Pacific/Rarotonga -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 20.2s J0 | DeleteReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Should have had a good message here
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([409FEEBDB49636E3:C17960A5C3C956DF]:0)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:138)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 155689 T1460 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_SA, timezone=Pacific/Rarotonga
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=49578496,total=252182528
   [junit4]   2> NOTE: All tests run in this JVM: [TestRebalanceLeaders, TestSort, HdfsRecoveryZkTest, SortByFunctionTest, TestShortCircuitedRequests, TestLRUCache, TestSolrQueryParserDefaultOperatorResource, TestDistributedSearch, AnalysisAfterCoreReloadTest, TestReplicationHandler, SolrTestCaseJ4Test, CacheHeaderTest, TestFastOutputStream, IndexSchemaRuntimeFieldTest, DeleteReplicaTest]
   [junit4] Completed on J0 in 20.24s, 1 test, 1 failure <<< FAILURES!

[...truncated 1322 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 failure, 47 ignored (21 assumptions)

Total time: 61 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b20) - Build # 11628 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11628/
Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)


FAILED:  org.apache.solr.cloud.OverseerStatusTest.testDistribSearch

Error Message:
Error from server at http://127.0.0.1:51111: reload the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51111: reload the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA:8E16D25740B1D086]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.invokeCollectionApi(AbstractFullDistribZkTestBase.java:1870)
	at org.apache.solr.cloud.OverseerStatusTest.doTest(OverseerStatusTest.java:103)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9115 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerStatusTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/init-core-data-001
   [junit4]   2> 667781 T2952 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 667783 T2952 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 667783 T2952 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 667784 T2953 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 667884 T2952 oasc.ZkTestServer.run start zk server on port:39574
   [junit4]   2> 667884 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 667885 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667890 T2960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ed65f1c name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667890 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667890 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 667891 T2952 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 667893 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 667894 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667898 T2963 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@411d26bb name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667899 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667899 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 667899 T2952 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 667901 T2952 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 667902 T2952 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 667903 T2952 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 667904 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 667904 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 667906 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 667906 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 667907 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 667908 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 667909 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 667909 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 667910 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 667911 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 667912 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 667912 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 667913 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 667914 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 667915 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 667915 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 667916 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 667916 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 667918 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 667918 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 667920 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 667920 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 668259 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 668260 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@259cd35a{/,null,AVAILABLE}
   [junit4]   2> 668269 T2952 oejs.AbstractConnector.doStart Started ServerConnector@6fa1c9b7{HTTP/1.1}{127.0.0.1:39621}
   [junit4]   2> 668270 T2952 oejs.Server.doStart Started @669634ms
   [junit4]   2> 668270 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 668270 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 668271 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002
   [junit4]   2> 668271 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/'
   [junit4]   2> 668286 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/solr.xml
   [junit4]   2> 668298 T2952 oasc.CoreContainer.<init> New CoreContainer 648521723
   [junit4]   2> 668298 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/]
   [junit4]   2> 668299 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 668299 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 668302 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 668302 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 668303 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 668303 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 668303 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 668304 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 668304 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 668304 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 668305 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 668307 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668308 T2977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e23bd52 name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668309 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668310 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 668312 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668313 T2980 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30765cf0 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668313 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668315 T2952 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 668317 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 668318 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 668320 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 668322 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 668323 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 668324 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39621_
   [junit4]   2> 668325 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39621_
   [junit4]   2> 668327 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 668328 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 668329 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 668330 T2952 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39621_
   [junit4]   2> 668330 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 668331 T2952 oasc.Overseer.start Overseer (id=93160157159030787-127.0.0.1:39621_-n_0000000000) starting
   [junit4]   2> 668333 T2952 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 668337 T2952 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 668337 T2982 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 668338 T2952 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 668340 T2952 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 668341 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 668345 T2981 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 668349 T2984 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 668349 T2984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 668350 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 668350 T2984 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 668352 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 668352 T2981 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 668352 T2981 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 668353 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 668354 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669350 T2984 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 669351 T2984 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 669351 T2984 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 669351 T2984 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 669352 T2984 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 669352 T2984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/collection1/'
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 669365 T2984 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 669366 T2984 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 669370 T2984 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 669384 T2984 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 669393 T2984 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 669394 T2984 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 669398 T2984 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 669493 T2984 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 669494 T2984 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 669495 T2984 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 669497 T2984 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 669507 T2984 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 669507 T2984 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 669507 T2984 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/
   [junit4]   2> 669508 T2984 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 669508 T2984 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data
   [junit4]   2> 669508 T2984 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index/
   [junit4]   2> 669508 T2984 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 669508 T2984 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index
   [junit4]   2> 669509 T2984 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 669510 T2984 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10eaef17 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@672875cd),segFN=segments_1,generation=1}
   [junit4]   2> 669510 T2984 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 669511 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 669511 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 669513 T2984 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 669517 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 669525 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669526 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669527 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669527 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669528 T2984 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 669529 T2984 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 669529 T2984 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 669529 T2984 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 669530 T2984 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10eaef17 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@672875cd),segFN=segments_1,generation=1}
   [junit4]   2> 669530 T2984 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 669530 T2984 oass.SolrIndexSearcher.<init> Opening Searcher@376810cc[collection1] main
   [junit4]   2> 669530 T2984 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 669531 T2984 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 669531 T2984 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 669532 T2984 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 669532 T2984 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 669532 T2984 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 669533 T2984 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 669533 T2984 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 669533 T2984 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 669533 T2984 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 669534 T2984 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 669534 T2985 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@376810cc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 669534 T2988 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39621 collection:control_collection shard:shard1
   [junit4]   2> 669534 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 669535 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 669535 T2988 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 669535 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669536 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669538 T2991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@635190b8 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669538 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669538 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669539 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 669539 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 669540 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2990 name=collection1 org.apache.solr.core.SolrCore@2b4d1ed8 url=http://127.0.0.1:39621/collection1 node=127.0.0.1:39621_ C2990_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:39621, node_name=127.0.0.1:39621_, state=down}
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39621/collection1/
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.syncToMe http://127.0.0.1:39621/collection1/ has no replicas
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39621/collection1/ shard1
   [junit4]   2> 669541 T2988 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 669542 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 669542 T2952 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 669542 T2952 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 669542 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669543 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669544 T2994 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@329c1998 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669545 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669545 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669546 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669546 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 669548 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 1
   [junit4]   2> 669548 T2981 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 669549 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669549 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669554 T2981 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 669554 T2995 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669555 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669596 T2988 oasc.ZkController.register We are http://127.0.0.1:39621/collection1/ and leader is http://127.0.0.1:39621/collection1/
   [junit4]   2> 669597 T2988 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39621
   [junit4]   2> 669597 T2988 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 669597 T2988 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 669597 T2988 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 669598 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669599 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 669600 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 669702 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669702 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669791 T2952 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 669792 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 669793 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@73b406e4{/,null,AVAILABLE}
   [junit4]   2> 669794 T2952 oejs.AbstractConnector.doStart Started ServerConnector@59727343{HTTP/1.1}{127.0.0.1:45877}
   [junit4]   2> 669794 T2952 oejs.Server.doStart Started @671158ms
   [junit4]   2> 669794 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/'
   [junit4]   2> 669806 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/solr.xml
   [junit4]   2> 669815 T2952 oasc.CoreContainer.<init> New CoreContainer 1588537754
   [junit4]   2> 669815 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/]
   [junit4]   2> 669815 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 669818 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 669818 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 669818 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 669818 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 669819 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 669819 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 669819 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669820 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669822 T3009 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8e77fc name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669822 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669823 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669826 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669827 T3012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ada0a70 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669827 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669834 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 670838 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45877_
   [junit4]   2> 670840 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45877_
   [junit4]   2> 670845 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 670853 T3013 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 670853 T3013 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 670855 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 670856 T3013 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 670857 T3013 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 670857 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 670857 T3013 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 670859 T3013 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 670859 T3013 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 670962 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 670963 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 670964 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 671860 T3013 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 671860 T3013 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 671861 T3013 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 671861 T3013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 671863 T3013 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 671863 T3013 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/collection1/'
   [junit4]   2> 671864 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 671865 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 671865 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 671884 T3013 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 671887 T3013 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 671891 T3013 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 671909 T3013 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 671924 T3013 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 671926 T3013 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 671929 T3013 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 672047 T3013 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 672049 T3013 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 672049 T3013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 672052 T3013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 672061 T3013 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 672061 T3013 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 672061 T3013 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/
   [junit4]   2> 672061 T3013 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 672062 T3013 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1
   [junit4]   2> 672062 T3013 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index/
   [junit4]   2> 672062 T3013 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 672063 T3013 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index
   [junit4]   2> 672063 T3013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 672063 T3013 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@24384936 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@270bb23e),segFN=segments_1,generation=1}
   [junit4]   2> 672064 T3013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 672067 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 672067 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 672067 T3013 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 672067 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 672069 T3013 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 672069 T3013 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 672072 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 672072 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 672073 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 672073 T3013 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 672074 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 672076 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672076 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672077 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672078 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672079 T3013 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 672080 T3013 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 672080 T3013 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 672081 T3013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 672082 T3013 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@24384936 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@270bb23e),segFN=segments_1,generation=1}
   [junit4]   2> 672082 T3013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 672082 T3013 oass.SolrIndexSearcher.<init> Opening Searcher@2110a1d2[collection1] main
   [junit4]   2> 672082 T3013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 672084 T3013 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 672084 T3013 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 672084 T3013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 672085 T3013 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 672085 T3013 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 672085 T3013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 672086 T3013 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 672086 T3013 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 672086 T3013 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 672094 T3014 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2110a1d2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 672095 T3013 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 672096 T3017 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45877 collection:collection1 shard:shard2
   [junit4]   2> 672096 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 672097 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 672098 T3017 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 672101 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 672102 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672102 T3017 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 672103 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2991 name=collection1 org.apache.solr.core.SolrCore@3d512af1 url=http://127.0.0.1:45877/collection1 node=127.0.0.1:45877_ C2991_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:45877, node_name=127.0.0.1:45877_, state=down}
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45877/collection1/
   [junit4]   2> 672103 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.syncToMe http://127.0.0.1:45877/collection1/ has no replicas
   [junit4]   2> 672103 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45877/collection1/ shard2
   [junit4]   2> 672104 T3017 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 672104 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 672104 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672105 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 672106 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672106 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 672107 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 672107 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672108 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 672156 T3017 oasc.ZkController.register We are http://127.0.0.1:45877/collection1/ and leader is http://127.0.0.1:45877/collection1/
   [junit4]   2> 672157 T3017 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45877
   [junit4]   2> 672157 T3017 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 672157 T3017 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 672157 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 672158 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672159 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 672159 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 672160 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 672160 T3017 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 672160 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672161 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 672333 T2952 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 672334 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 672336 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@14316aed{/,null,AVAILABLE}
   [junit4]   2> 672336 T2952 oejs.AbstractConnector.doStart Started ServerConnector@79500b40{HTTP/1.1}{127.0.0.1:51111}
   [junit4]   2> 672336 T2952 oejs.Server.doStart Started @673701ms
   [junit4]   2> 672337 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/'
   [junit4]   2> 672348 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/solr.xml
   [junit4]   2> 672357 T2952 oasc.CoreContainer.<init> New CoreContainer 316805325
   [junit4]   2> 672357 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/]
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 672361 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 672361 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 672361 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 672362 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 672362 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 672362 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 672363 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 672363 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 672363 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672364 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672366 T3031 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33f998cb name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672366 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672367 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672369 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672371 T3034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ff885a4 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672371 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672375 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 673379 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51111_
   [junit4]   2> 673380 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51111_
   [junit4]   2> 673384 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 673392 T3035 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 673392 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 673394 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 673394 T3035 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 673395 T3035 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 673395 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 673395 T3035 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 673395 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 673396 T2981 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 673396 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 673396 T3035 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 673396 T3035 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 673397 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 673398 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 673398 T3034 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 673398 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 673399 T3034 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 674397 T3035 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 674397 T3035 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 674398 T3035 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 674398 T3035 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 674400 T3035 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 674400 T3035 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/collection1/'
   [junit4]   2> 674401 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 674402 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 674402 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 674421 T3035 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 674424 T3035 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 674430 T3035 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 674450 T3035 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 674465 T3035 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 674466 T3035 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 674469 T3035 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 674564 T3035 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 674565 T3035 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 674566 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 674568 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 674579 T3035 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 674579 T3035 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 674579 T3035 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/
   [junit4]   2> 674579 T3035 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 674580 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2
   [junit4]   2> 674580 T3035 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index/
   [junit4]   2> 674581 T3035 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 674581 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index
   [junit4]   2> 674581 T3035 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 674582 T3035 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3bfc58a2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7dea9369),segFN=segments_1,generation=1}
   [junit4]   2> 674582 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 674585 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 674585 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 674585 T3035 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 674585 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 674587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 674587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 674589 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 674589 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 674590 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 674590 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 674591 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 674591 T3035 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 674592 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 674594 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674596 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674597 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674598 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674599 T3035 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 674601 T3035 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 674601 T3035 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 674602 T3035 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 674603 T3035 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3bfc58a2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7dea9369),segFN=segments_1,generation=1}
   [junit4]   2> 674603 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 674604 T3035 oass.SolrIndexSearcher.<init> Opening Searcher@6dfc8efa[collection1] main
   [junit4]   2> 674604 T3035 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 674605 T3035 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 674606 T3035 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 674607 T3035 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 674607 T3035 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 674608 T3035 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 674608 T3035 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 674608 T3036 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dfc8efa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 674609 T3035 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 674610 T3039 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51111 collection:collection1 shard:shard1
   [junit4]   2> 674610 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 674610 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 674611 T3039 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 674614 T3039 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 674615 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 674615 T3039 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 674615 T3039 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 674615 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C2992 name=collection1 org.apache.solr.core.SolrCore@26096e79 url=http://127.0.0.1:51111/collection1 node=127.0.0.1:51111_ C2992_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:51111, node_name=127.0.0.1:51111_, state=down}
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51111/collection1/
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrate

[...truncated too long message...]

dPoolExecutor.java:1066)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  21) Thread[id=7536, name=zkCallback-869-thread-115, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  22) Thread[id=6789, name=qtp901606055-6789-selector-ServerConnectorManager@6e891376/1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2> 	        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
   [junit4]   2> 	        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  23) Thread[id=6797, name=org.eclipse.jetty.server.session.HashSessionManager@57d04011Timer, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  24) Thread[id=6816, name=Scheduler-1401580675, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=LuceneFixedGap), _version_=FSTOrd50, a_t=FSTOrd50, a_i=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=DefaultSimilarity, locale=fr_LU, timezone=America/Grenada
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=34,free=61724600,total=285704192
   [junit4]   2> NOTE: All tests run in this JVM: [TestLMDirichletSimilarityFactory, TestOverriddenPrefixQueryForCustomFieldType, TestSchemaSimilarityResource, PathHierarchyTokenizerFactoryTest, SliceStateTest, AlternateDirectoryTest, TestSolrConfigHandlerConcurrent, TestSurroundQueryParser, TestBinaryField, TestSearcherReuse, TestSort, TestShardHandlerFactory, PeerSyncTest, NotRequiredUniqueKeyTest, DeleteInactiveReplicaTest, DistributedFacetPivotWhiteBoxTest, TestXIncludeConfig, TestReRankQParserPlugin, TestHashPartitioner, RankQueryTest, TestStressRecovery, PrimUtilsTest, TestFieldTypeResource, CustomCollectionTest, HighlighterConfigTest, TestRequestStatusCollectionAPI, TestWordDelimiterFilterFactory, TestSolrQueryParserDefaultOperatorResource, RequiredFieldsTest, SolrTestCaseJ4Test, RollingRestartTest, ConnectionManagerTest, TestCollectionAPI, TestSimpleQParserPlugin, TestManagedResourceStorage, QueryElevationComponentTest, DocValuesTest, TestQuerySenderListener, TestFastWriter, CoreAdminCreateDiscoverTest, DocExpirationUpdateProcessorFactoryTest, HdfsRecoveryZkTest, DirectUpdateHandlerTest, ExternalCollectionsTest, RecoveryZkTest, TestSolrConfigHandlerCloud, TestBulkSchemaConcurrent, PreAnalyzedFieldTest, TestStressReorder, OutputWriterTest, TestNonDefinedSimilarityFactory, ShowFileRequestHandlerTest, UnloadDistributedZkTest, TriLevelCompositeIdRoutingTest, CoreAdminHandlerTest, SolrCoreCheckLockOnStartupTest, DateMathParserTest, TestShortCircuitedRequests, TestObjectReleaseTracker, TestRandomMergePolicy, ClusterStateUpdateTest, RemoteQueryErrorTest, HdfsChaosMonkeySafeLeaderTest, TestIndexSearcher, SaslZkACLProviderTest, SortByFunctionTest, TestComponentsName, TestFunctionQuery, SolrXmlInZkTest, LegacyHTMLStripCharFilterTest, TestReplicationHandlerBackup, CircularListTest, DistributedFacetPivotLongTailTest, TestRandomDVFaceting, NumericFieldsTest, LeaderInitiatedRecoveryOnCommitTest, TestCloudPivotFacet, SolrIndexConfigTest, FacetPivotSmallTest, DocValuesMissingTest, TestCSVResponseWriter, TestOmitPositions, DefaultValueUpdateProcessorTest, UniqFieldsUpdateProcessorFactoryTest, SystemInfoHandlerTest, TestPivotHelperCode, SyncSliceTest, UUIDFieldTest, AddBlockUpdateTest, DateRangeFieldTest, LeaderElectionTest, BasicDistributedZkTest, TestStressLucene, TestRestManager, SimpleCollectionCreateDeleteTest, TestSweetSpotSimilarityFactory, TestImplicitCoreProperties, SpatialRPTFieldTypeTest, TestDocumentBuilder, TestCursorMarkWithoutUniqueKey, TestAnalyzeInfixSuggestions, TestLeaderElectionZkExpiry, TestGroupingSearch, CursorPagingTest, ZkCLITest, TestInitParams, DeleteReplicaTest, TestIndexingPerformance, TestMissingGroups, SolrInfoMBeanTest, NoCacheHeaderTest, JSONWriterTest, AssignTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=FF05C4F37EEB0BA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_LU -Dtests.timezone=America/Grenada -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)
   [junit4] Completed on J0 in 7224.00s, 1 test, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 suite-level error, 2 errors, 47 ignored (21 assumptions)

Total time: 167 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11627 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11627/
Java: 32bit/jdk1.8.0_25 -client -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([9E0EABB85A598E76:C65DCA9D477C414C]:0)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:280)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)




Build Log:
[...truncated 13425 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> 19647 T81 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 19673 T81 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 19673 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/'
   [junit4]   2> 19710 T81 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 19722 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 19737 T81 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 19737 T81 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 19745 T81 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 19758 T81 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 19759 T81 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 19759 T81 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/'
   [junit4]   2> 19794 T81 oasc.CoreContainer.<init> New CoreContainer 4179699
   [junit4]   2> 19795 T81 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/]
   [junit4]   2> 19795 T81 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19798 T81 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 19798 T81 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 19798 T81 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19799 T81 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19799 T81 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 19801 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/'
   [junit4]   2> 19827 T82 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 19835 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 19854 T82 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 19855 T82 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 19862 T82 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 19878 T82 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 19879 T82 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 19879 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 19880 T82 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/
   [junit4]   2> 19880 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 19880 T82 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/
   [junit4]   2> 19881 T82 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 19881 T82 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 19882 T82 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 19897 T82 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> 19898 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index/
   [junit4]   2> 19899 T82 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 19899 T82 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index
   [junit4]   2> 19900 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 19901 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19902 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 19903 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 19903 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19903 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 19905 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 19905 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 19906 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 19906 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 19908 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 19908 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 19913 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19914 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19916 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19917 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19918 T82 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 19919 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 19919 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 19921 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 19921 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19921 T82 oass.SolrIndexSearcher.<init> Opening Searcher@1415085[collection1] main
   [junit4]   2> 19922 T82 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf
   [junit4]   2> 19922 T82 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf}
   [junit4]   2> 19923 T82 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf
   [junit4]   2> 19923 T82 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 19923 T82 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 19924 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1415085[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 19924 T82 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 19924 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 19925 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1415085[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 19926 T81 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 19928 T81 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@b6ae2e
   [junit4]   2> 19931 T81 C9 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 19932 T81 C9 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 19932 T81 C9 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 19938 T81 C9 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 19947 T81 C9 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 19947 T81 C9 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 19972 T81 C9 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_2,generation=2}
   [junit4]   2> 19972 T81 C9 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 19982 T81 C9 oass.SolrIndexSearcher.<init> Opening Searcher@1b22fb2[collection1] main
   [junit4]   2> 19983 T81 C9 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 19983 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1b22fb2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C30)))}
   [junit4]   2> 19986 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 19987 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 19988 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 19988 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 19989 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b22fb2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C30)))}
   [junit4]   2> 19989 T81 C9 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 19990 T81 C9 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 19990 T81 C9 oashd.DocBuilder.execute Time taken = 0:0:0.52
   [junit4]   2> 19991 T81 C9 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 1, 3],commit=} 0 62
   [junit4]   2> 19994 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=30 status=0 QTime=1 
   [junit4]   2> 20002 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 20004 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=12 status=0 QTime=1 
   [junit4]   2> 20006 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=15 status=0 QTime=1 
   [junit4]   2> 20022 T81 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 20025 T81 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 51284 T81 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=9E0EABB85A598E76 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in_ID -Dtests.timezone=Europe/Vilnius -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   31.4s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9E0EABB85A598E76:C65DCA9D477C414C]:0)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:280)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 51379 T81 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 51380 T81 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=4179699
   [junit4]   2> 51380 T81 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@b6ae2e
   [junit4]   2> 51380 T81 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 51381 T81 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 51381 T81 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 51381 T81 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 51382 T81 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 51382 T81 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 51382 T81 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001;done=false>>]
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=PostingsFormat(name=LuceneFixedGap), _root_=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), desc=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), timestamp=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=DefaultSimilarity, locale=in_ID, timezone=Europe/Vilnius
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=285039384,total=302776320
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestWriterImpl, TestSqlEntityProcessorDelta, TestSqlEntityProcessor, TestDocBuilder2, TestXPathEntityProcessor, TestNumberFormatTransformer, TestPlainTextEntityProcessor, TestLineEntityProcessor, TestURLDataSource, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 31.77s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_141649_716.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid25119.hprof ...
   [junit4] Heap dump file created [478614752 bytes in 7.029 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 68 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_25) - Build # 11626 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11626/
Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([2E73AA5B93046FDE:7620CB7E8E21A0E4]:0)
	at java.util.HashMap.resize(HashMap.java:703)
	at java.util.HashMap.putVal(HashMap.java:628)
	at java.util.HashMap.put(HashMap.java:611)
	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:163)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)




Build Log:
[...truncated 13430 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> 7550 T43 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 7584 T43 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 7585 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/'
   [junit4]   2> 7627 T43 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 7672 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 7690 T43 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 7691 T43 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 7707 T43 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 7778 T43 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 7779 T43 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 7780 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/'
   [junit4]   2> 7808 T43 oasc.CoreContainer.<init> New CoreContainer 1665522085
   [junit4]   2> 7808 T43 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/]
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 7812 T43 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 7838 T43 oejul.Log.initialized Logging initialized @9160ms
   [junit4]   2> 7849 T43 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 7851 T43 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 7852 T43 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 7852 T43 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 7855 T44 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/'
   [junit4]   2> 7915 T44 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 7934 T44 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 7953 T44 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 7954 T44 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 7969 T44 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 8007 T44 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 8008 T44 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 8009 T44 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 8010 T44 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/
   [junit4]   2> 8011 T44 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 8012 T44 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/
   [junit4]   2> 8013 T44 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 8022 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 8023 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 8024 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> 8024 T44 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index/
   [junit4]   2> 8025 T44 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 8026 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index
   [junit4]   2> 8028 T44 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 8029 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 8031 T44 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 8032 T44 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 8034 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8039 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8039 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8040 T44 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 8040 T44 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 8041 T44 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 8043 T44 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 8044 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 8045 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 8045 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 8059 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 8069 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 8069 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 8070 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 8070 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 8071 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 8071 T44 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 8079 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8084 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8087 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8089 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8091 T44 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 8091 T44 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 8092 T44 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 8093 T44 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 8094 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 8095 T44 oass.SolrIndexSearcher.<init> Opening Searcher@6070b61d[collection1] main
   [junit4]   2> 8096 T44 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf
   [junit4]   2> 8105 T44 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf}
   [junit4]   2> 8106 T44 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf
   [junit4]   2> 8106 T44 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 8107 T44 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 8108 T44 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 8108 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@6070b61d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 8109 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 8110 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6070b61d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 8110 T43 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 8116 T43 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@be0b0df
   [junit4]   2> 8121 T43 C2 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 8121 T43 C2 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 8152 T43 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 8162 T43 C2 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 8187 T43 C2 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 8188 T43 C2 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 8234 T43 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_2,generation=2}
   [junit4]   2> 8235 T43 C2 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 8270 T43 C2 oass.SolrIndexSearcher.<init> Opening Searcher@2e2a214e[collection1] main
   [junit4]   2> 8272 T43 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 8272 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@2e2a214e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c27)))}
   [junit4]   2> 8286 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=13 
   [junit4]   2> 8288 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 8293 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=4 
   [junit4]   2> 8293 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 8293 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e2a214e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c27)))}
   [junit4]   2> 8295 T43 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 8296 T43 C2 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 8297 T43 C2 oashd.DocBuilder.execute Time taken = 0:0:0.144
   [junit4]   2> 8298 T43 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 3, 1],commit=} 0 180
   [junit4]   2> 8307 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=27 status=0 QTime=8 
   [junit4]   2> 8335 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 8348 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=9 status=0 QTime=1 
   [junit4]   2> 8357 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=15 status=0 QTime=1 
   [junit4]   2> 8407 T43 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 8421 T43 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 54500 T43 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=2E73AA5B93046FDE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi -Dtests.timezone=Asia/Karachi -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   46.1s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E73AA5B93046FDE:7620CB7E8E21A0E4]:0)
   [junit4]    > 	at java.util.HashMap.resize(HashMap.java:703)
   [junit4]    > 	at java.util.HashMap.putVal(HashMap.java:628)
   [junit4]    > 	at java.util.HashMap.put(HashMap.java:611)
   [junit4]    > 	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:163)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 54542 T43 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 54542 T43 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1665522085
   [junit4]   2> 54543 T43 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@be0b0df
   [junit4]   2> 54543 T43 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 54544 T43 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 54544 T43 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 54544 T43 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 54545 T43 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 54546 T43 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 54547 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 54547 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index
   [junit4]   2> 54548 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001;done=false>>]
   [junit4]   2> 54548 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=Lucene50(blocksize=128), _root_=Lucene50(blocksize=128), id=PostingsFormat(name=Direct), desc=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=Lucene50(blocksize=128)}, docValues:{}, sim=DefaultSimilarity, locale=vi, timezone=Asia/Karachi
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=53073904,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 47.07s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_120345_310.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid13598.hprof ...
   [junit4] Heap dump file created [797826427 bytes in 7.268 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 66 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11625 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11625/
Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([3F11AE594FB86051:6742CF7C529DAF6B]:0)
	at java.util.Arrays.copyOf(Arrays.java:3210)
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.ArrayList.grow(ArrayList.java:261)
	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
	at java.util.ArrayList.addAll(ArrayList.java:579)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:299)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)




Build Log:
[...truncated 13443 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> 25407 T190 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 25444 T190 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 25444 T190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/'
   [junit4]   2> 25498 T190 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 25516 T190 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 25541 T190 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 25542 T190 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 25552 T190 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 25596 T190 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 25598 T190 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 25598 T190 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/'
   [junit4]   2> 25659 T190 oasc.CoreContainer.<init> New CoreContainer 12284623
   [junit4]   2> 25659 T190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/]
   [junit4]   2> 25660 T190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 25664 T190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 25665 T190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 25665 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25668 T190 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 25669 T190 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 25669 T190 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25669 T190 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25670 T190 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 25672 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/'
   [junit4]   2> 25712 T191 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 25721 T191 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 25773 T191 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 25776 T191 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 25789 T191 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 25819 T191 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 25820 T191 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 25821 T191 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 25822 T191 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/
   [junit4]   2> 25823 T191 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 25824 T191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/
   [junit4]   2> 25825 T191 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 25825 T191 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 25826 T191 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 25827 T191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> 25828 T191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index/
   [junit4]   2> 25828 T191 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 25829 T191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index
   [junit4]   2> 25830 T191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 25831 T191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 25833 T191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 25844 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 25845 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25846 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25846 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25847 T191 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 25847 T191 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 25848 T191 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 25848 T191 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 25849 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 25849 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 25850 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 25851 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 25851 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 25852 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 25852 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 25853 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 25853 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 25854 T191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 25854 T191 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 25855 T191 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 25855 T191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 25859 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25861 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25863 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25864 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25865 T191 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 25866 T191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 25866 T191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 25867 T191 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 25867 T191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 25867 T191 oass.SolrIndexSearcher.<init> Opening Searcher@19db2ae[collection1] main
   [junit4]   2> 25868 T191 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf
   [junit4]   2> 25868 T191 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf}
   [junit4]   2> 25869 T191 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf
   [junit4]   2> 25869 T191 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 25869 T191 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 25870 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@19db2ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 25870 T191 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 25870 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 25870 T192 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19db2ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 25871 T190 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 25874 T190 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C68 name=collection1 org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 25877 T190 C68 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 25878 T190 C68 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 25879 T190 C68 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 25879 T190 C68 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 25890 T190 C68 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 25891 T190 C68 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> ASYNC  NEW_CORE C69 name=collection1 org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 25915 T190 C69 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_2,generation=2}
   [junit4]   2> 25916 T190 C69 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 25920 T190 C69 oass.SolrIndexSearcher.<init> Opening Searcher@1c7e5f2[collection1] main
   [junit4]   2> 25920 T190 C69 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 25921 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1c7e5f2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C25)))}
   [junit4]   2> 25922 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25923 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25924 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25924 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 25924 T192 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c7e5f2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C25)))}
   [junit4]   2> 25925 T190 C69 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 25926 T190 C69 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 25926 T190 C69 oashd.DocBuilder.execute Time taken = 0:0:0.46
   [junit4]   2> 25926 T190 C69 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 51
   [junit4]   2> 25928 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=25 status=0 QTime=1 
   [junit4]   2> 25937 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=1 
   [junit4]   2> 25940 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=10 status=0 QTime=1 
   [junit4]   2> 25943 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=12 status=0 QTime=0 
   [junit4]   2> 25958 T190 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 25962 T190 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 56720 T190 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=3F11AE594FB86051 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt_PT -Dtests.timezone=Australia/NSW -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   31.9s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3F11AE594FB86051:6742CF7C529DAF6B]:0)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3210)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3181)
   [junit4]    > 	at java.util.ArrayList.grow(ArrayList.java:261)
   [junit4]    > 	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
   [junit4]    > 	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
   [junit4]    > 	at java.util.ArrayList.addAll(ArrayList.java:579)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:299)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 57823 T190 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 57823 T190 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=12284623
   [junit4]   2> 57824 T190 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 57824 T190 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 57825 T190 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 57825 T190 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 57826 T190 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 57828 T190 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 57829 T190 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 57829 T190 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 57830 T190 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index
   [junit4]   2> 57831 T190 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001;done=false>>]
   [junit4]   2> 57831 T190 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), _root_=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=Memory doPackFST= false), desc=FSTOrd50, timestamp=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=pt_PT, timezone=Australia/NSW
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=495680048,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestWriterImpl, TestSolrEntityProcessorEndToEnd, TestSqlEntityProcessorDelta, TestSimplePropertiesWriter, TestXPathEntityProcessor, TestBuiltInEvaluators, TestXPathRecordReader, TestNumberFormatTransformer, TestDateFormatTransformer, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 32.47s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_094442_528.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid30912.hprof ...
   [junit4] Heap dump file created [592115515 bytes in 8.211 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 80 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11624 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11624/
Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([A5A0DD89F042C7CC:FDF3BCACED6708F6]:0)
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457)
	at java.lang.StringBuilder.append(StringBuilder.java:166)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2913)
	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
	at java.util.Formatter.format(Formatter.java:2520)
	at java.util.Formatter.format(Formatter.java:2455)
	at java.lang.String.format(String.java:2969)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)




Build Log:
[...truncated 13377 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> 6139 T43 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 6192 T43 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 6193 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/'
   [junit4]   2> 6253 T43 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 6268 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 6302 T43 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 6304 T43 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 6321 T43 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 6358 T43 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 6359 T43 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 6360 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6361 T43 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6361 T43 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001
   [junit4]   2> 6362 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/'
   [junit4]   2> 6437 T43 oasc.CoreContainer.<init> New CoreContainer 17131600
   [junit4]   2> 6438 T43 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/]
   [junit4]   2> 6439 T43 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 6441 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 6441 T43 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 6443 T43 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 6444 T43 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 6446 T43 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 6449 T43 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 6449 T43 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 6449 T43 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 6453 T44 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/'
   [junit4]   2> 6492 T44 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 6501 T44 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 6530 T44 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 6531 T44 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 6542 T44 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 6561 T44 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 6561 T44 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 6562 T44 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6562 T44 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/
   [junit4]   2> 6563 T44 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 6563 T44 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/
   [junit4]   2> 6564 T44 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 6564 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 6565 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 6565 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> 6566 T44 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index/
   [junit4]   2> 6566 T44 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 6566 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index
   [junit4]   2> 6567 T44 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 6568 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6569 T44 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 6570 T44 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 6570 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6571 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6571 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 6573 T44 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 6573 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 6574 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 6574 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 6578 T44 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 6580 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6582 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6588 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6590 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6591 T44 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 6591 T44 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 6592 T44 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 6593 T44 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 6593 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6594 T44 oass.SolrIndexSearcher.<init> Opening Searcher@c2c128[collection1] main
   [junit4]   2> 6595 T44 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf
   [junit4]   2> 6596 T44 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf}
   [junit4]   2> 6596 T44 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf
   [junit4]   2> 6597 T44 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 6597 T44 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 6598 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@c2c128[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6598 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 6598 T44 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 6599 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c2c128[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6600 T43 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 6604 T43 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 54876 T43 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=A5A0DD89F042C7CC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-13 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   48.3s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A5A0DD89F042C7CC:FDF3BCACED6708F6]:0)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > 	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
   [junit4]    > 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
   [junit4]    > 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:136)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:76)
   [junit4]    > 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:166)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:76)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2913)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
   [junit4]    > 	at java.util.Formatter.format(Formatter.java:2520)
   [junit4]    > 	at java.util.Formatter.format(Formatter.java:2455)
   [junit4]    > 	at java.lang.String.format(String.java:2969)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 54936 T43 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@1dbd1a7
   [junit4]   2> 54942 T43 C3 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 54943 T43 C3 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 54956 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 54957 T43 C3 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 54968 T43 C3 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 54969 T43 C3 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 54978 T43 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_2,generation=2}
   [junit4]   2> 54978 T43 C3 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 54980 T43 C3 oass.SolrIndexSearcher.<init> Opening Searcher@16fdc04[collection1] main
   [junit4]   2> 54981 T43 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 54981 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@16fdc04[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C26)))}
   [junit4]   2> 54991 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=9 
   [junit4]   2> 54992 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 54994 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 54994 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 54994 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16fdc04[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C26)))}
   [junit4]   2> 54995 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 54997 T43 C3 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 54997 T43 C3 oashd.DocBuilder.execute Time taken = 0:0:0.40
   [junit4]   2> 54997 T43 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 59
   [junit4]   2> 54999 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=26 status=0 QTime=1 
   [junit4]   2> 55013 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 55016 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=10 status=0 QTime=0 
   [junit4]   2> 55021 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=13 status=0 QTime=1 
   [junit4]   2> 55049 T43 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 55051 T43 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 55052 T43 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=17131600
   [junit4]   2> 55052 T43 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1dbd1a7
   [junit4]   2> 55053 T43 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 55053 T43 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 55053 T43 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 55054 T43 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 55054 T43 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 55055 T43 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 55055 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 55056 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index
   [junit4]   2> 55056 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001;done=false>>]
   [junit4]   2> 55057 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001
   [junit4]   2> NOTE: test params are: codec=Lucene50, sim=DefaultSimilarity, locale=ar_KW, timezone=Etc/GMT-13
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=51361120,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 48.96s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_071446_963.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid14852.hprof ...
   [junit4] Heap dump file created [592095532 bytes in 8.216 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 64 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any