You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/17 00:05:33 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build # 11261 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11261/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([CE017CEBECA61E3A:4FE7F2F39BF97E06]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:706)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:728)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:194)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10834 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/init-core-data-001
   [junit4]   2> 83673 T232 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 83673 T232 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 83679 T232 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 83682 T232 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 83684 T233 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 83784 T232 oasc.ZkTestServer.run start zk server on port:33465
   [junit4]   2> 83787 T232 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 83834 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 83868 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1619a5c name:ZooKeeperConnection Watcher:127.0.0.1:33465 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 83869 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 83869 T232 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 83871 T232 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 83898 T232 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 83899 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 83901 T241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@104bbee name:ZooKeeperConnection Watcher:127.0.0.1:33465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 83901 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 83902 T232 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 83902 T232 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 83909 T232 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 83913 T232 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 83917 T232 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 83921 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 83928 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 83936 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 83936 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 83942 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 83942 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 83948 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 83948 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 83954 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 83954 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 83959 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 83959 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 83962 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 83962 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 83965 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 83966 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 83969 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 83969 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 83972 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 83972 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 83975 T232 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 83975 T232 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 84270 T232 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 84272 T232 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52509
   [junit4]   2> 84273 T232 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 84273 T232 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 84274 T232 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002
   [junit4]   2> 84274 T232 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/'
   [junit4]   2> 84285 T232 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/solr.xml
   [junit4]   2> 84308 T232 oasc.CoreContainer.<init> New CoreContainer 3798257
   [junit4]   2> 84309 T232 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/]
   [junit4]   2> 84309 T232 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 84310 T232 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 84310 T232 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 84310 T232 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 84311 T232 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 84311 T232 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 84311 T232 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 84311 T232 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 84312 T232 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 84312 T232 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 84313 T232 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 84313 T232 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 84313 T232 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 84314 T232 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33465/solr
   [junit4]   2> 84323 T232 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 84324 T232 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 84324 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84326 T252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2890fb name:ZooKeeperConnection Watcher:127.0.0.1:33465 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84326 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84327 T232 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 84330 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 84332 T254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e088e4 name:ZooKeeperConnection Watcher:127.0.0.1:33465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 84332 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 84338 T232 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 84344 T232 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 84349 T232 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 84354 T232 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 84360 T232 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 84368 T232 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 84372 T232 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52509_
   [junit4]   2> 84375 T232 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52509_
   [junit4]   2> 84379 T232 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 84382 T232 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 84384 T232 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 84390 T232 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52509_
   [junit4]   2> 84390 T232 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 84393 T232 oasc.Overseer.start Overseer (id=92464902341066755-127.0.0.1:52509_-n_0000000000) starting
   [junit4]   2> 84400 T232 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 84423 T232 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 84426 T256 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 84427 T232 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 84430 T232 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 84432 T232 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 84440 T255 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 84444 T258 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 84444 T258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 84449 T258 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 84450 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 84457 T255 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52509",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "node_name":"127.0.0.1:52509_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 84457 T255 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 84458 T255 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 84460 T255 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 84467 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 85449 T258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 85450 T258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 85451 T258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 85451 T258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 85452 T258 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 85453 T258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/collection1/'
   [junit4]   2> 85454 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 85454 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 85454 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 85499 T258 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 85529 T258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 85530 T258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 85537 T258 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 85848 T258 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 85850 T258 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 85851 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 85854 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 85883 T258 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 85883 T258 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 85892 T258 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/
   [junit4]   2> 85892 T258 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@365e59
   [junit4]   2> 85897 T258 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data
   [junit4]   2> 85897 T258 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index/
   [junit4]   2> 85897 T258 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 85898 T258 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index
   [junit4]   2> 85898 T258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2c95e7
   [junit4]   2> 85904 T258 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 85905 T258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 85910 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 85921 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 85921 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 85921 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 85921 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 85922 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 85922 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 85922 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 85922 T258 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 85923 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 85924 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 85924 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 85925 T258 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 85926 T258 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 85926 T258 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 85926 T258 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 85927 T258 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 85927 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 85928 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 85928 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 85931 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 85931 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 85932 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 85933 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 85935 T258 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 85935 T258 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 85936 T258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=72.9140625, floorSegmentMB=0.33203125, forceMergeDeletesPctAllowed=9.022644095143843, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 85937 T258 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 85937 T258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 85938 T258 oass.SolrIndexSearcher.<init> Opening Searcher@db0b4f[collection1] main
   [junit4]   2> 85938 T258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 85940 T258 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 85940 T258 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 85942 T258 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 85942 T258 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 85943 T258 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 85943 T258 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 85944 T258 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 85945 T258 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 85947 T258 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 85948 T258 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 85948 T258 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 85953 T260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@db0b4f[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 85959 T258 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 85960 T263 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52509 collection:control_collection shard:shard1
   [junit4]   2> 85960 T232 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 85961 T232 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 85968 T263 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 85973 T232 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 85974 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 85975 T265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16c9dec name:ZooKeeperConnection Watcher:127.0.0.1:33465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 85975 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 85976 T232 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 85977 T263 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 85978 T232 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 85980 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 85980 T263 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 85981 T263 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@4ff220 url=http://127.0.0.1:52509/collection1 node=127.0.0.1:52509_ C27_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:52509, node_name=127.0.0.1:52509_, core=collection1, state=down}
   [junit4]   2> 85982 T263 C27 P52509 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52509/collection1/
   [junit4]   2> 85983 T263 C27 P52509 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 85983 T263 C27 P52509 oasc.SyncStrategy.syncToMe http://127.0.0.1:52509/collection1/ has no replicas
   [junit4]   2> 85983 T263 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52509/collection1/ shard1
   [junit4]   2> 85984 T232 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 85986 T263 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 85987 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 85987 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 85992 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 86100 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 86100 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 86144 T263 oasc.ZkController.register We are http://127.0.0.1:52509/collection1/ and leader is http://127.0.0.1:52509/collection1/
   [junit4]   2> 86144 T263 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52509
   [junit4]   2> 86144 T263 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 86145 T263 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 86145 T263 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 86147 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 86151 T255 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52509",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "node_name":"127.0.0.1:52509_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 86255 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 86255 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 86267 T232 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 86270 T232 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 86271 T232 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59586
   [junit4]   2> 86272 T232 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 86273 T232 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 86273 T232 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003
   [junit4]   2> 86273 T232 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/'
   [junit4]   2> 86284 T232 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/solr.xml
   [junit4]   2> 86307 T232 oasc.CoreContainer.<init> New CoreContainer 12842513
   [junit4]   2> 86307 T232 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/]
   [junit4]   2> 86308 T232 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 86308 T232 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 86309 T232 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 86309 T232 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 86309 T232 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 86309 T232 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 86310 T232 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 86310 T232 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 86310 T232 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 86311 T232 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 86311 T232 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 86311 T232 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 86312 T232 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 86312 T232 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33465/solr
   [junit4]   2> 86312 T232 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 86313 T232 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 86313 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 86315 T277 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1501615 name:ZooKeeperConnection Watcher:127.0.0.1:33465 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 86315 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 86315 T232 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 86319 T232 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 86320 T279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdb1e6 name:ZooKeeperConnection Watcher:127.0.0.1:33465/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 86321 T232 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 86332 T232 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 87338 T232 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59586_
   [junit4]   2> 87340 T232 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59586_
   [junit4]   2> 87347 T232 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 87358 T281 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 87358 T281 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 87359 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 87360 T281 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 87364 T255 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59586",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "node_name":"127.0.0.1:59586_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 87364 T255 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 87364 T255 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 87365 T255 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 87469 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 87469 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 87469 T266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 88360 T281 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 88361 T281 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 88362 T281 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 88362 T281 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 88364 T281 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 88364 T281 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/collection1/'
   [junit4]   2> 88365 T281 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 88365 T281 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 88365 T281 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 88397 T281 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 88429 T281 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 88430 T281 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 88436 T281 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 88603 T281 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 88604 T281 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 88606 T281 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 88609 T281 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 88642 T281 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 88643 T281 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 88643 T281 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/
   [junit4]   2> 88644 T281 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@365e59
   [junit4]   2> 88645 T281 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1
   [junit4]   2> 88645 T281 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index/
   [junit4]   2> 88645 T281 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 88646 T281 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index
   [junit4]   2> 88646 T281 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@a970d0
   [junit4]   2> 88653 T281 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 88654 T281 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 88670 T281 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 88671 T281 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 88672 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 88672 T281 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 88673 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 88673 T281 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 88674 T281 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 88675 T281 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 88675 T281 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 88678 T281 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 88679 T281 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 88680 T281 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 88682 T281 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 88683 T281 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 88683 T281 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 88684 T281 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 88685 T281 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 88686 T281 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 88687 T281 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 88689 T281 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 88692 T281 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 88693 T281 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 88694 T281 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 88695 T281 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 88698 T281 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 88698 T281 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 88700 T281 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=72.9140625, floorSegmentMB=0.33203125, forceMergeDeletesPctAllowed=9.022644095143843, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 88701 T281 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 88701 T281 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 88701 T281 oass.SolrIndexSearcher.<init> Opening Searcher@b1bf89[collection1] main
   [junit4]   2> 88702 T281 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 88703 T281 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 88704 T281 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 88704 T281 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 88705 T281 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 88705 T281 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 88706 T281 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 88707 T281 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 88707 T281 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 88707 T281 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 88718 T282 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b1bf89[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 88728 T281 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 88729 T285 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59586 collection:collection1 shard:shard1
   [junit4]   2> 88730 T232 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 88730 T232 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 88730 T285 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 88734 T232 oasc.AbstractFullDistribZkTestBase.randomlyEnableAutoSoftCommit Not turning on auto soft commit
   [junit4]   2> 88740 T285 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> ASYNC  NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@4ff220 url=http://127.0.0.1:52509/collection1 node=127.0.0.1:52509_ C28_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:52509, node_name=127.0.0.1:52509_, core=collection1, state=active, leader=true}
   [junit4]   2> 88742 T248 C28 P52509 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1479438442704142336)} 0 6
   [junit4]   2> 88743 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88743 T285 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 88744 T232 oascsi.CloudSolrServer.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 0
   [junit4]   2> 88744 T285 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@1ea9f6d url=http://127.0.0.1:59586/collection1 node=127.0.0.1:59586_ C29_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:59586, node_name=127.0.0.1:59586_, core=collection1, state=down}
   [junit4]   2> 88744 T285 C29 P59586 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59586/collection1/
   [junit4]   2> 88745 T285 C29 P59586 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 88745 T285 C29 P59586 oasc.SyncStrategy.syncToMe http://127.0.0.1:59586/collection1/ has no replicas
   [junit4]   2> 88745 T285 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59586/collection1/ shard1
   [junit4]   2> 88746 T285 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 88746 T232 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 52509
   [junit4]   2> 88746 T232 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=3798257
   [junit4]   2> 88747 T232 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 88747 T232 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 88748 T232 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4ff220
   [junit4]   2> 88758 T255 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52509",
   [junit4]   2> 	  "node_name":"127.0.0.1:52509_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 88760 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88763 T232 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 88763 T232 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 88764 T232 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 88764 T232 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 88764 T259 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88765 T232 C28 P52509 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
   [junit4]   2> 88766 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 88772 T280 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> ASYNC  NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@4ff220 url=http://127.0.0.1:52509/collection1 node=127.0.0.1:52509_ C30_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:52509, node_name=127.0.0.1:52509_, core=collection1, state=active, leader=true}
   [junit4]   2> 88773 T232 C30 P52509 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index,segFN=segments_2,generation=2}
   [junit4]   2> 88774 T232 C30 P52509 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 88776 T232 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 88777 T232 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 88777 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data;done=false>>]
   [junit4]   2> 88778 T232 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data
   [junit4]   2> 88778 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index;done=false>>]
   [junit4]   2> 88779 T232 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/control/data/index
   [junit4]   2> 88779 T232 oasc.Overseer.close Overseer (id=92464902341066755-127.0.0.1:52509_-n_0000000000) closing
   [junit4]   2> 88780 T255 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52509_
   [junit4]   2> 88795 T286 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59586_
   [junit4]   2> 88796 T286 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 88798 T286 oasc.Overseer.start Overseer (id=92464902341066758-127.0.0.1:59586_-n_0000000001) starting
   [junit4]   2> 88811 T285 oasc.ZkController.register We are http://127.0.0.1:59586/collection1/ and leader is http://127.0.0.1:59586/collection1/
   [junit4]   2> 88811 T285 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59586
   [junit4]   2> 88812 T285 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 88812 T285 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 88812 T285 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 88814 T286 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 88814 T290 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 88818 T289 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 88823 T289 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59586",
   [junit4]   2> 	  "node_name":"127.0.0.1:59586_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 88827 T286 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 88829 T286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 90295 T259 oascc.ZkStateReader.getIndividualColls WARN Error reading collections nodes org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:313)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:310)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:310)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getIndividualColls(ZkStateReader.java:483)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.constructState(ZkStateReader.java:448)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:55)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:315)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 90295 T287 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 90296 T259 oascc.ZkStateReader.getIndividualColls WARN Error reading collections nodes org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/control_collection/state.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:313)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:310)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:310)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.getIndividualColls(ZkStateReader.java:483)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.constructState(ZkStateReader.java:448)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:55)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:315)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 90298 T288 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 90320 T232 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 95310 T232 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59586
   [junit4]   2> 95310 T232 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=12842513
   [junit4]   2> 95311 T232 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 95311 T232 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 95312 T286 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 95312 T232 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ea9f6d
   [junit4]   2> 95316 T289 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59586",
   [junit4]   2> 	  "node_name":"127.0.0.1:59586_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 95319 T286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 95329 T232 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 95329 T232 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 95330 T232 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 95330 T232 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 95332 T232 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 95332 T232 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 95332 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index;done=false>>]
   [junit4]   2> 95332 T232 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1/index
   [junit4]   2> 95333 T232 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1;done=false>>]
   [junit4]   2> 95333 T232 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001/tempDir-001/jetty1
   [junit4]   2> 95333 T232 oasc.Overseer.close Overseer (id=92464902341066758-127.0.0.1:59586_-n_0000000001) closing
   [junit4]   2> 95334 T289 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:59586_
   [junit4]   2> 96835 T292 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 96835 T286 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 96858 T232 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 101848 T232 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 101850 T232 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33465 33465
   [junit4]   2> 102612 T233 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:33465 33465
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=CE017CEBECA61E3A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_CA -Dtests.timezone=US/Pacific -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   19.0s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CE017CEBECA61E3A:4FE7F2F39BF97E06]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:706)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:728)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:194)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:112)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 102645 T232 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-CE017CEBECA61E3A-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=en_CA, timezone=US/Pacific
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=1,free=186078576,total=234356736
   [junit4]   2> NOTE: All tests run in this JVM: [SynonymTokenizerTest, TestSchemaNameResource, QueryElevationComponentTest, SliceStateTest, HardAutoCommitTest, SolrInfoMBeanTest, TestSolr4Spatial2, SOLR749Test, NumericFieldsTest, TestReloadAndDeleteDocs, TestReversedWildcardFilterFactory, TestDistributedMissingSort, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 19.01s, 1 test, 1 error <<< FAILURES!

[...truncated 1286 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1772 tests, 1 error, 45 ignored (23 assumptions)

Total time: 91 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11274 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11274/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:60044

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:60044
	at __randomizedtesting.SeedInfo.seed([3E18A4595A8DF826:BFFE2A412DD2981A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.doTest(DistributedFacetPivotLongTailTest.java:82)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
	... 46 more




Build Log:
[...truncated 11852 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedFacetPivotLongTailTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/init-core-data-001
   [junit4]   2> 2571055 T6654 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 2571056 T6654 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2571058 T6654 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2571059 T6654 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2571061 T6654 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2571063 T6654 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:47746
   [junit4]   2> 2571065 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2571065 T6654 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2571065 T6654 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 2571065 T6654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 2571076 T6654 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2571097 T6654 oasc.CoreContainer.<init> New CoreContainer 542015301
   [junit4]   2> 2571097 T6654 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 2571098 T6654 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2571098 T6654 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2571098 T6654 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2571099 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2571099 T6654 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2571099 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2571099 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2571099 T6654 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2571100 T6654 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2571101 T6654 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2571102 T6654 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2571102 T6654 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2571102 T6654 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2571104 T6664 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2571105 T6664 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2571105 T6664 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2571105 T6664 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2571146 T6664 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2571178 T6664 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2571178 T6664 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2571183 T6664 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2571363 T6664 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2571367 T6664 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2571368 T6664 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2571372 T6664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2571374 T6664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2571375 T6664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2571375 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2571376 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2571376 T6664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2571376 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2571376 T6664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2571376 T6664 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2571377 T6664 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2571377 T6664 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/control/data/
   [junit4]   2> 2571377 T6664 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d27722
   [junit4]   2> 2571377 T6664 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/control/data
   [junit4]   2> 2571377 T6664 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/control/data/index/
   [junit4]   2> 2571378 T6664 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2571378 T6664 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/control/data/index
   [junit4]   2> 2571378 T6664 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=64.7529296875, floorSegmentMB=0.4697265625, forceMergeDeletesPctAllowed=26.432904035571944, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2571379 T6664 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5c567df8))),segFN=segments_1,generation=1}
   [junit4]   2> 2571379 T6664 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2571379 T6664 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2571381 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2571381 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2571382 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2571383 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2571383 T6664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2571383 T6664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2571383 T6664 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2571383 T6664 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571384 T6664 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571384 T6664 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571384 T6664 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2571384 T6664 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2571384 T6664 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2571385 T6664 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2571385 T6664 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2571385 T6664 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2571385 T6664 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2571386 T6664 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2571386 T6664 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2571386 T6664 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2571386 T6664 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2571386 T6664 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2571387 T6664 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2571387 T6664 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571387 T6664 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571387 T6664 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571388 T6664 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571388 T6664 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571388 T6664 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571388 T6664 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571388 T6664 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571389 T6664 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2571389 T6664 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2571391 T6664 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571392 T6664 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571392 T6664 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571393 T6664 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571394 T6664 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2571395 T6664 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2571396 T6664 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2571396 T6664 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2571396 T6664 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5c567df8))),segFN=segments_1,generation=1}
   [junit4]   2> 2571396 T6664 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2571397 T6664 oass.SolrIndexSearcher.<init> Opening Searcher@2e03c23e[collection1] main
   [junit4]   2> 2571397 T6664 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2571397 T6664 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2571397 T6664 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2571397 T6664 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2571398 T6664 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2571398 T6664 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2571467 T6664 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2571489 T6665 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e03c23e[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2571492 T6664 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2571492 T6654 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2571493 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2571497 T6654 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2571501 T6654 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2571502 T6654 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52424
   [junit4]   2> 2571505 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2571505 T6654 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2571505 T6654 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 2571505 T6654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 2571518 T6654 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2571540 T6654 oasc.CoreContainer.<init> New CoreContainer 1439138029
   [junit4]   2> 2571540 T6654 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 2571541 T6654 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2571541 T6654 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2571541 T6654 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2571541 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2571542 T6654 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2571542 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2571542 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2571542 T6654 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2571542 T6654 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2571544 T6654 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2571545 T6654 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2571545 T6654 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2571545 T6654 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2571547 T6676 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2571548 T6676 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2571548 T6676 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2571549 T6676 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2571583 T6676 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2571615 T6676 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2571615 T6676 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2571620 T6676 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2571798 T6676 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2571801 T6676 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2571803 T6676 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2571807 T6676 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2571809 T6676 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2571810 T6676 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2571810 T6676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2571811 T6676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2571811 T6676 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2571811 T6676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2571811 T6676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2571811 T6676 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2571812 T6676 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2571812 T6676 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard0/data/
   [junit4]   2> 2571812 T6676 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d27722
   [junit4]   2> 2571812 T6676 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard0/data
   [junit4]   2> 2571813 T6676 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard0/data/index/
   [junit4]   2> 2571813 T6676 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2571813 T6676 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard0/data/index
   [junit4]   2> 2571813 T6676 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=64.7529296875, floorSegmentMB=0.4697265625, forceMergeDeletesPctAllowed=26.432904035571944, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2571814 T6676 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-7618a2e))),segFN=segments_1,generation=1}
   [junit4]   2> 2571814 T6676 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2571814 T6676 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2571817 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2571817 T6676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2571817 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2571817 T6676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2571817 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2571818 T6676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2571818 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2571818 T6676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2571818 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2571818 T6676 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2571819 T6676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2571819 T6676 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2571819 T6676 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571819 T6676 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571820 T6676 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2571820 T6676 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2571820 T6676 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2571820 T6676 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2571821 T6676 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2571821 T6676 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2571821 T6676 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2571822 T6676 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2571822 T6676 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2571822 T6676 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2571822 T6676 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2571823 T6676 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2571823 T6676 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2571823 T6676 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2571823 T6676 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571824 T6676 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571824 T6676 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571824 T6676 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571824 T6676 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571825 T6676 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571825 T6676 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571825 T6676 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2571826 T6676 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2571826 T6676 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2571828 T6676 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571829 T6676 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571830 T6676 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571830 T6676 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2571831 T6676 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2571833 T6676 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2571833 T6676 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2571833 T6676 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2571834 T6676 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-7618a2e))),segFN=segments_1,generation=1}
   [junit4]   2> 2571834 T6676 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2571834 T6676 oass.SolrIndexSearcher.<init> Opening Searcher@9aa1934[collection1] main
   [junit4]   2> 2571834 T6676 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2571834 T6676 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2571835 T6676 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2571835 T6676 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2571835 T6676 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2571835 T6676 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2571840 T6676 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2571857 T6677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9aa1934[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2571861 T6676 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2571862 T6654 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2571862 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2571864 T6654 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2571866 T6654 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2571867 T6654 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60044
   [junit4]   2> 2571869 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2571869 T6654 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2571870 T6654 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 2571870 T6654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 2571885 T6654 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2571906 T6654 oasc.CoreContainer.<init> New CoreContainer 858568044
   [junit4]   2> 2571907 T6654 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 2571907 T6654 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2571908 T6654 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2571908 T6654 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2571908 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2571908 T6654 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2571908 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2571909 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2571909 T6654 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2571909 T6654 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2571910 T6654 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2571911 T6654 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2571911 T6654 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2571911 T6654 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2571913 T6688 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2571914 T6688 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2571915 T6688 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2571915 T6688 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2571958 T6688 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2571990 T6688 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2571990 T6688 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2571995 T6688 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2572205 T6688 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2572208 T6688 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2572210 T6688 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2572214 T6688 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2572216 T6688 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2572218 T6688 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2572218 T6688 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2572218 T6688 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2572219 T6688 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2572219 T6688 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2572219 T6688 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2572219 T6688 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2572219 T6688 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2572219 T6688 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard1/data/
   [junit4]   2> 2572220 T6688 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d27722
   [junit4]   2> 2572220 T6688 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard1/data
   [junit4]   2> 2572220 T6688 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard1/data/index/
   [junit4]   2> 2572221 T6688 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2572221 T6688 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard1/data/index
   [junit4]   2> 2572221 T6688 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=64.7529296875, floorSegmentMB=0.4697265625, forceMergeDeletesPctAllowed=26.432904035571944, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2572222 T6688 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5f3c6876))),segFN=segments_1,generation=1}
   [junit4]   2> 2572222 T6688 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2572222 T6688 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2572225 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2572225 T6688 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2572225 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2572225 T6688 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2572226 T6688 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2572227 T6688 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2572227 T6688 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2572227 T6688 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572227 T6688 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572228 T6688 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572228 T6688 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2572228 T6688 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2572228 T6688 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2572229 T6688 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2572229 T6688 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2572229 T6688 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2572230 T6688 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2572230 T6688 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2572230 T6688 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2572230 T6688 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2572231 T6688 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2572231 T6688 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2572231 T6688 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2572231 T6688 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572232 T6688 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572232 T6688 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572232 T6688 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572233 T6688 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572233 T6688 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572233 T6688 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572234 T6688 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572234 T6688 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2572234 T6688 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2572236 T6688 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572237 T6688 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572238 T6688 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572239 T6688 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572239 T6688 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2572241 T6688 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2572241 T6688 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2572242 T6688 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2572242 T6688 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5f3c6876))),segFN=segments_1,generation=1}
   [junit4]   2> 2572242 T6688 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2572242 T6688 oass.SolrIndexSearcher.<init> Opening Searcher@2ab07f40[collection1] main
   [junit4]   2> 2572243 T6688 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2572243 T6688 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2572243 T6688 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2572243 T6688 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2572243 T6688 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2572243 T6688 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2572248 T6688 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2572273 T6689 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ab07f40[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2572280 T6688 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2572281 T6654 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2572281 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2572285 T6654 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2572290 T6654 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2572291 T6654 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57170
   [junit4]   2> 2572293 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2572293 T6654 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2572294 T6654 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr
   [junit4]   2> 2572294 T6654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/'
   [junit4]   2> 2572309 T6654 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2572333 T6654 oasc.CoreContainer.<init> New CoreContainer 1608175841
   [junit4]   2> 2572333 T6654 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/]
   [junit4]   2> 2572333 T6654 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2572334 T6654 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2572334 T6654 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2572334 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2572334 T6654 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2572335 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2572335 T6654 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2572335 T6654 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2572335 T6654 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2572336 T6654 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2572337 T6654 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2572337 T6654 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2572338 T6654 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2572340 T6700 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2572341 T6700 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2572341 T6700 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2572341 T6700 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2572371 T6700 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2572404 T6700 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2572404 T6700 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2572410 T6700 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2572599 T6700 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2572603 T6700 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2572605 T6700 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2572609 T6700 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2572611 T6700 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2572612 T6700 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2572613 T6700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2572613 T6700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2572613 T6700 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2572613 T6700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2572614 T6700 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2572614 T6700 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2572614 T6700 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2572614 T6700 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard2/data/
   [junit4]   2> 2572614 T6700 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d27722
   [junit4]   2> 2572615 T6700 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard2/data
   [junit4]   2> 2572615 T6700 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard2/data/index/
   [junit4]   2> 2572615 T6700 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2572615 T6700 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001/tempDir-001/shard2/data/index
   [junit4]   2> 2572616 T6700 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=64.7529296875, floorSegmentMB=0.4697265625, forceMergeDeletesPctAllowed=26.432904035571944, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2572616 T6700 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-1335a58e))),segFN=segments_1,generation=1}
   [junit4]   2> 2572616 T6700 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2572616 T6700 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2572619 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2572619 T6700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2572620 T6700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2572621 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2572621 T6700 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2572621 T6700 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2572621 T6700 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2572622 T6700 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572622 T6700 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572622 T6700 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2572623 T6700 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2572623 T6700 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2572623 T6700 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2572624 T6700 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2572624 T6700 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2572624 T6700 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2572625 T6700 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2572625 T6700 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2572625 T6700 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2572626 T6700 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2572626 T6700 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2572626 T6700 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2572626 T6700 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2572627 T6700 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572627 T6700 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572627 T6700 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572627 T6700 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572628 T6700 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572628 T6700 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572628 T6700 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572629 T6700 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2572629 T6700 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2572630 T6700 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2572632 T6700 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572633 T6700 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572633 T6700 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572635 T6700 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2572635 T6700 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2572637 T6700 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2572637 T6700 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2572638 T6700 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2572638 T6700 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-1335a58e))),segFN=segments_1,generation=1}
   [junit4]   2> 2572638 T6700 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2572638 T6700 oass.SolrIndexSearcher.<init> Opening Searcher@5ad57256[collection1] main
   [junit4]   2> 2572639 T6700 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2572639 T6700 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2572639 T6700 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2572639 T6700 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2572639 T6700 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2572639 T6700 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2572644 T6700 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2572661 T6701 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ad57256[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2572666 T6700 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2572666 T6654 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2572667 T6654 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C5468 name=collection1 org.apache.solr.core.SolrCore@438c32ae
   [junit4]   2> 2572736 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1479524213001814016)]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C5469 name=collection1 org.apache.solr.core.SolrCore@2d8babe0
   [junit4]   2> 2572796 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1479524213065777152)]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C5470 name=collection1 org.apache.solr.core.SolrCore@70d233dc
   [junit4]   2> 2572856 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1479524213128691712)]} 0 1
   [junit4]   2> 2572860 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1479524213132886016)]} 0 1
   [junit4]   2> 2572863 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1479524213137080320)]} 0 0
   [junit4]   2> 2572867 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1479524213141274624)]} 0 0
   [junit4]   2> 2572870 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1479524213144420352)]} 0 0
   [junit4]   2> 2572873 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1479524213147566080)]} 0 0
   [junit4]   2> 2572875 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1479524213149663232)]} 0 0
   [junit4]   2> 2572878 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1479524213151760384)]} 0 1
   [junit4]   2> 2572880 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1479524213154906112)]} 0 1
   [junit4]   2> 2572882 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1479524213157003264)]} 0 0
   [junit4]   2> 2572884 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1479524213159100416)]} 0 0
   [junit4]   2> 2572886 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1479524213161197568)]} 0 0
   [junit4]   2> 2572889 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1479524213164343296)]} 0 1
   [junit4]   2> 2572891 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1479524213166440448)]} 0 0
   [junit4]   2> 2572893 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1479524213168537600)]} 0 0
   [junit4]   2> 2572895 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1479524213170634752)]} 0 0
   [junit4]   2> 2572897 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1479524213172731904)]} 0 0
   [junit4]   2> 2572899 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1479524213174829056)]} 0 0
   [junit4]   2> 2572901 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1479524213176926208)]} 0 0
   [junit4]   2> 2572903 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1479524213179023360)]} 0 0
   [junit4]   2> 2572905 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1479524213181120512)]} 0 0
   [junit4]   2> 2572908 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1479524213183217664)]} 0 1
   [junit4]   2> 2572910 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1479524213186363392)]} 0 0
   [junit4]   2> 2572912 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1479524213188460544)]} 0 0
   [junit4]   2> 2572914 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1479524213190557696)]} 0 0
   [junit4]   2> 2572916 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1479524213192654848)]} 0 0
   [junit4]   2> 2572918 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1479524213194752000)]} 0 0
   [junit4]   2> 2572921 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1479524213196849152)]} 0 1
   [junit4]   2> 2572923 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1479524213199994880)]} 0 1
   [junit4]   2> 2572925 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1479524213202092032)]} 0 0
   [junit4]   2> 2572927 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1479524213204189184)]} 0 0
   [junit4]   2> 2572929 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1479524213206286336)]} 0 0
   [junit4]   2> 2572931 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1479524213208383488)]} 0 0
   [junit4]   2> 2572933 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1479524213210480640)]} 0 0
   [junit4]   2> 2572935 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1479524213212577792)]} 0 0
   [junit4]   2> 2572937 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1479524213214674944)]} 0 0
   [junit4]   2> 2572940 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1479524213216772096)]} 0 1
   [junit4]   2> 2572942 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1479524213219917824)]} 0 0
   [junit4]   2> 2572944 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1479524213222014976)]} 0 0
   [junit4]   2> 2572946 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1479524213224112128)]} 0 0
   [junit4]   2> 2572948 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1479524213226209280)]} 0 0
   [junit4]   2> 2572950 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1479524213228306432)]} 0 0
   [junit4]   2> 2572952 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1479524213230403584)]} 0 0
   [junit4]   2> 2572954 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1479524213232500736)]} 0 0
   [junit4]   2> 2572957 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1479524213234597888)]} 0 1
   [junit4]   2> 2572959 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1479524213237743616)]} 0 0
   [junit4]   2> 2572961 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1479524213239840768)]} 0 0
   [junit4]   2> 2572963 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1479524213241937920)]} 0 0
   [junit4]   2> 2572965 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1479524213244035072)]} 0 0
   [junit4]   2> 2572967 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1479524213246132224)]} 0 0
   [junit4]   2> 2572969 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1479524213248229376)]} 0 0
   [junit4]   2> 2572972 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1479524213250326528)]} 0 0
   [junit4]   2> 2572974 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1479524213252423680)]} 0 1
   [junit4]   2> 2572976 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1479524213255569408)]} 0 0
   [junit4]   2> 2572978 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1479524213257666560)]} 0 0
   [junit4]   2> 2572980 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1479524213259763712)]} 0 0
   [junit4]   2> 2572982 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1479524213261860864)]} 0 0
   [junit4]   2> 2572985 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1479524213263958016)]} 0 1
   [junit4]   2> 2572987 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1479524213267103744)]} 0 0
   [junit4]   2> 2572989 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1479524213269200896)]} 0 0
   [junit4]   2> 2572991 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1479524213271298048)]} 0 0
   [junit4]   2> 2572993 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1479524213273395200)]} 0 0
   [junit4]   2> 2572995 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1479524213275492352)]} 0 0
   [junit4]   2> 2572997 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1479524213277589504)]} 0 0
   [junit4]   2> 2572999 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67 (1479524213279686656)]} 0 0
   [junit4]   2> 2573001 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[68 (1479524213281783808)]} 0 0
   [junit4]   2> 2573003 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[69 (1479524213283880960)]} 0 0
   [junit4]   2> 2573006 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[70 (1479524213285978112)]} 0 1
   [junit4]   2> 2573008 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[71 (1479524213289123840)]} 0 0
   [junit4]   2> 2573010 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[72 (1479524213291220992)]} 0 0
   [junit4]   2> 2573012 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[73 (1479524213293318144)]} 0 0
   [junit4]   2> 2573014 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[74 (1479524213295415296)]} 0 0
   [junit4]   2> 2573016 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[75 (1479524213297512448)]} 0 0
   [junit4]   2> 2573018 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[76 (1479524213299609600)]} 0 0
   [junit4]   2> 2573021 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[77 (1479524213302755328)]} 0 1
   [junit4]   2> 2573023 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[78 (1479524213304852480)]} 0 0
   [junit4]   2> 2573025 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[79 (1479524213306949632)]} 0 0
   [junit4]   2> 2573027 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[80 (1479524213309046784)]} 0 0
   [junit4]   2> 2573030 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[81 (1479524213311143936)]} 0 1
   [junit4]   2> 2573032 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[82 (1479524213314289664)]} 0 0
   [junit4]   2> 2573034 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[83 (1479524213316386816)]} 0 0
   [junit4]   2> 2573037 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[84 (1479524213319532544)]} 0 0
   [junit4]   2> 2573039 T6672 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[85 (1479524213321629696)]} 0 0
   [junit4]   2> 2573041 T6685 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[86 (1479524213323726848)]} 0 0
   [junit4]   2> 2573044 T6696 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[87 (1479524213325824000)]} 0 1
   [junit4]   2> 2573046 T6673 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[88 (1479524213328969728)]} 0 0
   [junit4]   2> 2573048 T6686 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[89 (1479524213331066880)]} 0 0
   [junit4]   2> 2573051 T6697 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[90 (1479524213333164032)]} 0 1
   [junit4]   2> 2573053 T6674 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[91 (1479524213336309760)]} 0 0
   [junit4]   2> 2573055 T6687 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[92 (1479524213338406912)]} 0 0
   [junit4]   2> 2573057 T6698 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[93 (1479524213340504064)]} 0 0
   [junit4]   2> 2573059 T6675 C5468 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[94 (1479524213342601216)]} 0 0
   [junit4]   2> 2573062 T6684 C5469 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[95 (1479524213345746944)]} 0 0
   [junit4]   2> 2573064 T6699 C5470 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[96 (14795

[...truncated too long message...]

 J1 | DistributedFacetPivotLongTailTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:60044
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3E18A4595A8DF826:BFFE2A412DD2981A]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:558)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
   [junit4]    > 	at org.apache.solr.handler.component.DistributedFacetPivotLongTailTest.doTest(DistributedFacetPivotLongTailTest.java:82)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:448)
   [junit4]    > 	... 46 more
   [junit4]   2> 2688639 T6654 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.handler.component.DistributedFacetPivotLongTailTest-3E18A4595A8DF826-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=is, timezone=Etc/GMT+9
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=105335864,total=445755392
   [junit4]   2> NOTE: All tests run in this JVM: [ScriptEngineTest, CoreAdminRequestStatusTest, TestManagedResourceStorage, TestFiltering, QueryElevationComponentTest, URLClassifyProcessorTest, AddSchemaFieldsUpdateProcessorFactoryTest, MultiTermTest, SolrXmlInZkTest, TestJmxMonitoredMap, RankQueryTest, SOLR749Test, ConnectionManagerTest, TestNonNRTOpen, TestRawResponseWriter, DistributedQueryComponentCustomSortTest, LegacyHTMLStripCharFilterTest, DateMathParserTest, TestRealTimeGet, TestSchemaSimilarityResource, TestCoreContainer, AnalyticsQueryTest, TestElisionMultitermQuery, AutoCommitTest, TestQueryUtils, HighlighterMaxOffsetTest, HardAutoCommitTest, TestCloudSchemaless, TestRandomFaceting, DocExpirationUpdateProcessorFactoryTest, TestMergePolicyConfig, TestIndexingPerformance, TestLFUCache, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest, SolrCoreTest, LeaderElectionIntegrationTest, TestBinaryField, TestManagedStopFilterFactory, SolrIndexConfigTest, TestGroupingSearch, DebugComponentTest, DateFieldTest, DistributedIntervalFacetingTest, StatsComponentTest, CSVRequestHandlerTest, SoftAutoCommitTest, TestUtils, TestDefaultSearchFieldResource, TestCustomSort, TestWriterPerf, TestPseudoReturnFields, TestValueSourceCache, UUIDFieldTest, TestFunctionQuery, TestJoin, HdfsChaosMonkeySafeLeaderTest, SpatialFilterTest, TestReloadAndDeleteDocs, TestCSVResponseWriter, TestSolrQueryParserDefaultOperatorResource, TestImplicitCoreProperties, TestQuerySenderNoQuery, BlockCacheTest, CopyFieldTest, CursorMarkTest, RemoteQueryErrorTest, TestCharFilters, ConvertedLegacyTest, AddBlockUpdateTest, ClusterStateTest, OverseerRolesTest, TestCloudManagedSchemaConcurrent, SSLMigrationTest, TestZkChroot, ExternalCollectionsTest, TestSimpleQParserPlugin, DirectSolrConnectionTest, TestCloudManagedSchema, FullSolrCloudDistribCmdsTest, BasicDistributedZk2Test, TestBadConfig, TestStressLucene, SharedFSAutoReplicaFailoverTest, TestPhraseSuggestions, CoreContainerCoreInitFailuresTest, TestOverriddenPrefixQueryForCustomFieldType, ChaosMonkeySafeLeaderTest, SliceStateTest, TestLeaderElectionZkExpiry, DistribCursorPagingTest, DistributedQueryElevationComponentTest, TestXIncludeConfig, LukeRequestHandlerTest, TestExceedMaxTermLength, TestRequestStatusCollectionAPI, TestIBSimilarityFactory, MigrateRouteKeyTest, TestSolrQueryParser, SuggestComponentTest, AnalysisAfterCoreReloadTest, TestUniqueKeyFieldResource, ShardRoutingTest, TestReplicationHandler, TestRangeQuery, SolrCmdDistributorTest, PeerSyncTest, HighlighterTest, CurrencyFieldOpenExchangeTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, DocValuesTest, TestTrie, TestCSVLoader, PolyFieldTest, SchemaVersionSpecificBehaviorTest, TestUpdate, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, TestRemoteStreaming, IndexBasedSpellCheckerTest, TestSurroundQueryParser, StandardRequestHandlerTest, TestReversedWildcardFilterFactory, TestQueryTypes, FileBasedSpellCheckerTest, RequestHandlersTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, JSONWriterTest, TestAnalyzedSuggestions, JsonLoaderTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, TestComponentsName, TestBinaryResponseWriter, AlternateDirectoryTest, BadComponentTest, SampleTest, NumericFieldsTest, TestConfig, OutputWriterTest, TestSolrCoreProperties, TestSystemIdResolver, FileUtilsTest, CursorPagingTest, TestHighlightDedupGrouping, TestTolerantSearch, AliasIntegrationTest, HttpPartitionTest, MultiThreadedOCPTest, ShardSplitTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsRecoveryZkTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, CachingDirectoryFactoryTest, TestSolrXmlPersistence, DistributedFacetPivotLongTailTest]
   [junit4] Completed on J1 in 117.60s, 1 test, 1 error <<< FAILURES!

[...truncated 127 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1772 tests, 1 error, 45 ignored (23 assumptions)

Total time: 88 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_20) - Build # 11273 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11273/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseParallelGC

112 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestNRTOpen

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCoreInc(TestHarness.java:219)
	at org.apache.solr.util.TestHarness.update(TestHarness.java:235)
	at org.apache.solr.util.BaseTestHarness.checkUpdateStatus(BaseTestHarness.java:282)
	at org.apache.solr.util.BaseTestHarness.validateUpdate(BaseTestHarness.java:252)
	at org.apache.solr.SolrTestCaseJ4.checkUpdateU(SolrTestCaseJ4.java:677)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:656)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:650)
	at org.apache.solr.core.TestNRTOpen.beforeClass(TestNRTOpen.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:767)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestNonNRTOpen

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCoreInc(TestHarness.java:219)
	at org.apache.solr.util.TestHarness.update(TestHarness.java:235)
	at org.apache.solr.util.BaseTestHarness.checkUpdateStatus(BaseTestHarness.java:282)
	at org.apache.solr.util.BaseTestHarness.validateUpdate(BaseTestHarness.java:252)
	at org.apache.solr.SolrTestCaseJ4.checkUpdateU(SolrTestCaseJ4.java:677)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:656)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:650)
	at org.apache.solr.core.TestNonNRTOpen.beforeClass(TestNonNRTOpen.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:767)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.EchoParamsTest.test

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:604B1232DEAACE34]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.util.TestHarness$LocalRequestFactory.makeRequest(TestHarness.java:410)
	at org.apache.solr.SolrTestCaseJ4.req(SolrTestCaseJ4.java:1004)
	at org.apache.solr.EchoParamsTest.defaultEchoParams(EchoParamsTest.java:48)
	at org.apache.solr.EchoParamsTest.test(EchoParamsTest.java:37)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.OutputWriterTest.testUselessWriter

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:77A9093DDE742AC4]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.util.TestHarness$LocalRequestFactory.makeRequest(TestHarness.java:410)
	at org.apache.solr.SolrTestCaseJ4.req(SolrTestCaseJ4.java:1004)
	at org.apache.solr.OutputWriterTest.testUselessWriter(OutputWriterTest.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.OutputWriterTest.testLazy

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:D9C620BAC00A7767]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.OutputWriterTest.testLazy(OutputWriterTest.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.OutputWriterTest.testTrivialXsltWriter

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:56D346358C6EB860]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.util.TestHarness$LocalRequestFactory.makeRequest(TestHarness.java:410)
	at org.apache.solr.SolrTestCaseJ4.req(SolrTestCaseJ4.java:1004)
	at org.apache.solr.OutputWriterTest.testTrivialXsltWriter(OutputWriterTest.java:82)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.OutputWriterTest.testTrivialXsltWriterInclude

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:7182E4435C6FC557]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.util.TestHarness$LocalRequestFactory.makeRequest(TestHarness.java:410)
	at org.apache.solr.SolrTestCaseJ4.req(SolrTestCaseJ4.java:1004)
	at org.apache.solr.OutputWriterTest.testTrivialXsltWriterInclude(OutputWriterTest.java:91)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.OutputWriterTest.testSOLR59responseHeaderVersions

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:4521AEBE47170237]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCore(TestHarness.java:209)
	at org.apache.solr.util.TestHarness$LocalRequestFactory.makeRequest(TestHarness.java:410)
	at org.apache.solr.SolrTestCaseJ4.req(SolrTestCaseJ4.java:1004)
	at org.apache.solr.OutputWriterTest.testSOLR59responseHeaderVersions(OutputWriterTest.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.SampleTest.testSimple

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:D0AC091657A5771D]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCoreInc(TestHarness.java:219)
	at org.apache.solr.util.TestHarness.update(TestHarness.java:235)
	at org.apache.solr.util.BaseTestHarness.checkUpdateStatus(BaseTestHarness.java:282)
	at org.apache.solr.util.BaseTestHarness.validateUpdate(BaseTestHarness.java:252)
	at org.apache.solr.SolrTestCaseJ4.checkUpdateU(SolrTestCaseJ4.java:677)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:656)
	at org.apache.solr.SampleTest.testSimple(SampleTest.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.SampleTest.testAdvanced

Error Message:
SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single

Stack Trace:
org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:33A1E9CFA59D0EF9]:0)
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.util.TestHarness.getCoreInc(TestHarness.java:219)
	at org.apache.solr.util.TestHarness.update(TestHarness.java:235)
	at org.apache.solr.util.BaseTestHarness.checkUpdateStatus(BaseTestHarness.java:282)
	at org.apache.solr.util.BaseTestHarness.validateUpdate(BaseTestHarness.java:252)
	at org.apache.solr.SolrTestCaseJ4.checkUpdateU(SolrTestCaseJ4.java:677)
	at org.apache.solr.SolrTestCaseJ4.assertU(SolrTestCaseJ4.java:656)
	at org.apache.solr.SampleTest.testAdvanced(SampleTest.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:793)
	... 8 more


REGRESSION:  org.apache.solr.cloud.AsyncMigrateRouteKeyTest.testDistribSearch

Error Message:
DocCount on target collection does not match expected:<19> but was:<5>

Stack Trace:
java.lang.AssertionError: DocCount on target collection does not match expected:<19> but was:<5>
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:69F9A3F00709C3F0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.MigrateRouteKeyTest.multipleShardMigrateTest(MigrateRouteKeyTest.java:220)
	at org.apache.solr.cloud.AsyncMigrateRouteKeyTest.doTest(AsyncMigrateRouteKeyTest.java:48)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 10000ms , collection: collection1 slice: shard1
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:69F9A3F00709C3F0]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:641)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:130)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 {msg=SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single,trace=org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:294)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)  at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)  at org.eclipse.jetty.server.Server.handle(Server.java:368)  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)  at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)  at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:890)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:651)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single  at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:793)  ... 8 more ,code=500}</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /txk/ic/collection1/update. Reason: <pre>    {msg=SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single,trace=org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:294)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)  at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)  at org.eclipse.jetty.server.Server.handle(Server.java:368)  at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)  at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)  at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)  at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)  at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)  at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:890)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:651)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)  at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single  at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)  at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)  at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)  at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:793)  ... 8 more ,code=500}</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 {msg=SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single,trace=org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:294)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:368)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:793)
	... 8 more
,code=500}</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /txk/ic/collection1/update. Reason:
<pre>    {msg=SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single,trace=org.apache.solr.common.SolrException: SolrCore 'collection1' is not available due to init failure: Unrecognized lockType: single
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:747)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:294)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:368)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:890)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:651)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
Caused by: org.apache.solr.common.SolrException: Unrecognized lockType: single
	at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:793)
	... 8 more
,code=500}</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:69F9A3F00709C3F0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:727)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:194)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.testDistribSearch

Error Message:
There are still nodes recoverying - waited for 30 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds
	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:69F9A3F00709C3F0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:754)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1373)
	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRul

[...truncated too long message...]

at org.apache.solr.core.StandardDirectoryFactory.createLockFactory(StandardDirectoryFactory.java:73)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487)
   [junit4]    > 	at org.apache.solr.core.SolrCore.&lt;init&gt;(SolrCore.java:793)
   [junit4]    > 	... 8 more
   [junit4]    > ,code=500}</pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    >                                                 
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E81F2DE87056A3CC:69F9A3F00709C3F0]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
   [junit4]    > 	at org.apache.solr.schema.TestCloudSchemaless.doTest(TestCloudSchemaless.java:140)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3098996 T6916 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.schema.TestCloudSchemaless-E81F2DE87056A3CC-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene410), sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=el_CY, timezone=America/Ojinaga
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.8.0_20 (32-bit)/cpus=8,threads=1,free=82032000,total=475529216
   [junit4]   2> NOTE: All tests run in this JVM: [TestNoOpRegenerator, LukeRequestHandlerTest, BasicZkTest, CacheHeaderTest, TestNonNRTOpen, CollectionsAPIDistributedZkTest, UniqFieldsUpdateProcessorFactoryTest, TestPhraseSuggestions, TestWriterPerf, UpdateRequestProcessorFactoryTest, TermVectorComponentTest, SOLR749Test, CustomCollectionTest, SoftAutoCommitTest, SpatialFilterTest, AtomicUpdatesTest, HdfsDirectoryTest, ExternalCollectionsTest, TestIntervalFaceting, HttpPartitionTest, TestExtendedDismaxParser, CoreMergeIndexesAdminHandlerTest, TestLazyCores, TestNRTOpen, TestSolrQueryParserDefaultOperatorResource, TestPerFieldSimilarity, TestSchemaResource, QueryParsingTest, TestRandomMergePolicy, TestMissingGroups, ChaosMonkeySafeLeaderTest, TestManagedSchemaDynamicFieldResource, TestGroupingSearch, AnalyticsQueryTest, DistributedTermsComponentTest, TestSchemaVersionResource, TestXIncludeConfig, ChangedSchemaMergeTest, RankQueryTest, OutOfBoxZkACLAndCredentialsProvidersTest, ResourceLoaderTest, TestOmitPositions, TestSolrXmlPersistence, MultiTermTest, TestValueSourceCache, TestBadConfig, TestCustomSort, TestIBSimilarityFactory, TestStressLucene, TestJmxMonitoredMap, SolrIndexConfigTest, StatsComponentTest, CursorMarkTest, ScriptEngineTest, SearchHandlerTest, BinaryUpdateRequestHandlerTest, SolrRequestParserTest, TestMultiCoreConfBootstrap, SampleTest, TestRecovery, PrimUtilsTest, CircularListTest, CurrencyFieldXmlFileTest, TestAtomicUpdateErrorCases, RequestHandlersTest, TestSearcherReuse, TestFieldSortValues, TestNonDefinedSimilarityFactory, TestInfoStreamLogging, TestManagedStopFilterFactory, OutputWriterTest, DefaultValueUpdateProcessorTest, TestIndexingPerformance, ClusterStateTest, TestJoin, TestSchemaSimilarityResource, TestSimpleQParserPlugin, TestSolrQueryParserResource, DeleteLastCustomShardedReplicaTest, CoreContainerCoreInitFailuresTest, OverriddenZkACLAndCredentialsProvidersTest, TestUpdate, TestFieldTypeResource, DocValuesMissingTest, DocValuesTest, AnalysisErrorHandlingTest, TermVectorComponentDistributedTest, HdfsRecoveryZkTest, TestFileDictionaryLookup, SynonymTokenizerTest, BlockCacheTest, PolyFieldTest, TestFunctionQuery, DistributedQueryComponentCustomSortTest, FullSolrCloudDistribCmdsTest, HdfsChaosMonkeySafeLeaderTest, SliceStateTest, SolrCoreTest, ShardRoutingCustomTest, HdfsUnloadDistributedZkTest, DistanceFunctionTest, MigrateRouteKeyTest, ConvertedLegacyTest, BasicDistributedZk2Test, OverseerTest, ShardRoutingTest, TestRandomDVFaceting, TestHashPartitioner, TestRealTimeGet, TestStressVersions, TestCoreContainer, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, TestIndexSearcher, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, IndexBasedSpellCheckerTest, TestQueryUtils, TestReversedWildcardFilterFactory, TestQueryTypes, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, JSONWriterTest, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, CSVRequestHandlerTest, HighlighterConfigTest, AlternateDirectoryTest, CopyFieldTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestPostingsSolrHighlighter, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCharFilters, EchoParamsTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, TestFastOutputStream, DateMathParserTest, TestSystemIdResolver, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, TestDocumentBuilder, DistributedIntervalFacetingTest, AsyncMigrateRouteKeyTest, DeleteReplicaTest, DistribDocExpirationUpdateProcessorTest, OverseerRolesTest, OverseerStatusTest, RollingRestartTest, TestShortCircuitedRequests, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsSyncSliceTest, StressHdfsTest, CachingDirectoryFactoryTest, QueryResultKeyTest, TestConfigSets, TestParamSet, TestReloadAndDeleteDocs, TestShardHandlerFactory, TestSolrXml, TestReplicationHandlerBackup, CoreAdminRequestStatusTest, InfoHandlerTest, DistributedDebugComponentTest, DistributedExpandComponentTest, DistributedFacetPivotSmallTest, TestExpandComponent, HighlighterMaxOffsetTest, TestChildDocTransformer, TestRawResponseWriter, TestSortingResponseWriter, TestManagedResource, TestCopyFieldCollectionResource, TestDefaultSearchFieldResource, TestDynamicFieldResource, TestFieldCollectionResource, TestFieldResource, TestFieldTypeCollectionResource, TestManagedSchemaFieldResource, TestSchemaNameResource, TestSerializedLuceneMatchVersion, BadCopyFieldTest, EnumFieldTest, ModifyConfFileTest, TestCloudManagedSchema, TestCloudManagedSchemaConcurrent, TestCloudSchemaless]
   [junit4] Completed on J1 in 74.16s, 1 test, 1 error <<< FAILURES!

[...truncated 76 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1768 tests, 2 suite-level errors, 103 errors, 7 failures, 45 ignored (23 assumptions)

Total time: 98 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11272 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11272/
Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

6 tests failed.
REGRESSION:  org.apache.solr.client.solrj.SolrExampleBinaryTest.testChildDoctransformer

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /solr/collection1/select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /solr/collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.client.solrj.SolrExampleXMLTest.testChildDoctransformer

Error Message:
java.lang.AssertionError

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.client.solrj.embedded.SolrExampleEmbeddedTest.testChildDoctransformer

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.lucene.search.join.ToChildBlockJoinQuery$ToChildBlockJoinScorer.advance(ToChildBlockJoinQuery.java:286)
	at org.apache.lucene.search.FilteredQuery$LeapFrogScorer.advanceToNextCommonDoc(FilteredQuery.java:274)
	at org.apache.lucene.search.FilteredQuery$LeapFrogScorer.nextDoc(FilteredQuery.java:286)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:192)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:614)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:289)
	at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:224)
	at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1631)
	at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1454)
	at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:535)
	at org.apache.solr.search.SolrIndexSearcher.getDocList(SolrIndexSearcher.java:1289)
	at org.apache.solr.response.transform.ChildDocTransformer.transform(ChildDocTransformerFactory.java:144)
	at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResultsBody(BinaryResponseWriter.java:159)
	at org.apache.solr.response.BinaryResponseWriter$Resolver.writeResults(BinaryResponseWriter.java:183)
	at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:88)
	at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:158)
	at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:148)
	at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:242)
	at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:153)
	at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:96)
	at org.apache.solr.response.BinaryResponseWriter.getParsedResponse(BinaryResponseWriter.java:256)
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.getParsedResponse(EmbeddedSolrServer.java:241)
	at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:213)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.client.solrj.embedded.SolrExampleJettyTest.testChildDoctransformer

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /solr/collection1/select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /solr/collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryTest.testChildDoctransformer

Error Message:
Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 Server Error</title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /solr/collection1/select. Reason: <pre>    Server Error</pre></p> <hr /><i><small>Powered by Jetty://</small></i>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /solr/collection1/select. Reason:
<pre>    Server Error</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
                                                
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:512)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(ConcurrentUpdateSolrServer.java:281)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.client.solrj.embedded.SolrExampleStreamingTest.testChildDoctransformer

Error Message:
java.lang.AssertionError

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([20CF0595A5B13DAC:53151A0F29A94AAA]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.request(ConcurrentUpdateSolrServer.java:281)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12479 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.SolrExampleXMLTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/init-core-data-001
   [junit4]   2> 1518 T16 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1519 T16 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1722 T16 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1853 T16 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56834
   [junit4]   2> 1886 T16 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1986 T16 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1988 T16 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 1989 T16 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/'
   [junit4]   2> 2141 T16 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/solr.xml
   [junit4]   2> 2529 T16 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 2541 T16 oasc.CoreContainer.<init> New CoreContainer 1960030174
   [junit4]   2> 2542 T16 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/]
   [junit4]   2> 2625 T16 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 2626 T16 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 2635 T16 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 2636 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2637 T16 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2638 T16 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2639 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2640 T16 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2641 T16 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2869 T16 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 2873 T16 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2875 T16 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2877 T16 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 2945 T16 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr
   [junit4]   2> 2971 T16 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 2972 T16 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2975 T28 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/'
   [junit4]   2> 3110 T28 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 3112 T28 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 3113 T28 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 3114 T28 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 3115 T28 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 3116 T28 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/langid/lib).
   [junit4]   2> 3117 T28 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 3118 T28 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 3118 T28 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/../../../dist).
   [junit4]   2> 3298 T28 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 3304 T28 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 3453 T28 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3463 T28 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3566 T28 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 4420 T28 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4448 T28 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
   [junit4]   2> 4449 T28 oasr.RestManager$Registry.registerManagedResource Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
   [junit4]   2> 4650 T28 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 4667 T28 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 4842 T28 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/
   [junit4]   2> 4872 T28 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4883 T28 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/tempDir-001/
   [junit4]   2> 4885 T28 oasc.JmxMonitoredMap.<init> No JMX servers found, not exposing Solr information with JMX.
   [junit4]   2> 4895 T28 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 4896 T28 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 4921 T28 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/tempDir-001
   [junit4]   2> 4922 T28 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/tempDir-001/index/
   [junit4]   2> 4923 T28 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/tempDir-001/index' doesn't exist. Creating new index...
   [junit4]   2> 4924 T28 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.SolrExampleXMLTest-20CF0595A5B13DAC-001/tempDir-001/index
   [junit4]   2> 4950 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: init: current segments file is "null"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@68f457aa
   [junit4]   2> 4951 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 4952 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: 0 msec to checkpoint
   [junit4]   2> 4952 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: init: create=true
   [junit4]   2> 4953 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: 
   [junit4]   2> 	dir=MockDirectoryWrapper(RAMDirectory(id=lucene-6ec0e8a7))
   [junit4]   2> 	index=
   [junit4]   2> 	version=5.0.0
   [junit4]   2> 	analyzer=null
   [junit4]   2> 	ramBufferSizeMB=100.0
   [junit4]   2> 	maxBufferedDocs=-1
   [junit4]   2> 	maxBufferedDeleteTerms=-1
   [junit4]   2> 	mergedSegmentWarmer=null
   [junit4]   2> 	delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
   [junit4]   2> 	commit=null
   [junit4]   2> 	openMode=CREATE
   [junit4]   2> 	similarity=org.apache.lucene.search.similarities.DefaultSimilarity
   [junit4]   2> 	mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=1, maxMergeCount=2, mergeThreadPriority=-1
   [junit4]   2> 	default WRITE_LOCK_TIMEOUT=1000
   [junit4]   2> 	writeLockTimeout=1000
   [junit4]   2> 	codec=Lucene410
   [junit4]   2> 	infoStream=org.apache.solr.update.LoggingInfoStream
   [junit4]   2> 	mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 	indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@73deac39
   [junit4]   2> 	readerPooling=false
   [junit4]   2> 	perThreadHardLimitMB=1945
   [junit4]   2> 	useCompoundFile=false
   [junit4]   2> 	checkIntegrityAtMerge=false
   [junit4]   2> 	commitOnClose=true
   [junit4]   2> 	writer=org.apache.lucene.util.SetOnce@1ffd0d9d
   [junit4]   2> 	
   [junit4]   2> 4954 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: now flush at close
   [junit4]   2> 4955 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]:   start flush: applyAllDeletes=true
   [junit4]   2> 4955 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]:   index before flush 
   [junit4]   2> 4956 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: startFullFlush
   [junit4]   2> 4956 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 4957 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: coreLoadExecutor-5-thread-1 finishFullFlush success=true
   [junit4]   2> 4957 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: apply all deletes during flush
   [junit4]   2> 4959 T28 oasu.LoggingInfoStream.message [BD][coreLoadExecutor-5-thread-1]: prune sis=null:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 4959 T28 oasu.LoggingInfoStream.message [CMS][coreLoadExecutor-5-thread-1]: now merge
   [junit4]   2> 4960 T28 oasu.LoggingInfoStream.message [CMS][coreLoadExecutor-5-thread-1]:   index: 
   [junit4]   2> 4960 T28 oasu.LoggingInfoStream.message [CMS][coreLoadExecutor-5-thread-1]:   no more merges pending; now return
   [junit4]   2> 4961 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: waitForMerges
   [junit4]   2> 4961 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: waitForMerges done
   [junit4]   2> 4961 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: start
   [junit4]   2> 4962 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: enter lock
   [junit4]   2> 4962 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: now prepare
   [junit4]   2> 4963 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: prepareCommit: flush
   [junit4]   2> 4963 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]:   index before flush 
   [junit4]   2> 4963 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: startFullFlush
   [junit4]   2> 4964 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 4964 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: apply all deletes during flush
   [junit4]   2> 4965 T28 oasu.LoggingInfoStream.message [BD][coreLoadExecutor-5-thread-1]: prune sis=null:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 4965 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: coreLoadExecutor-5-thread-1 finishFullFlush success=true
   [junit4]   2> 4965 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: startCommit(): start
   [junit4]   2> 4966 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: startCommit index= changeCount=1
   [junit4]   2> 4972 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: done all syncs: []
   [junit4]   2> 4972 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: pendingCommit != null
   [junit4]   2> 4973 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: wrote segments file "segments_1"
   [junit4]   2> 4973 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: now checkpoint "" [0 segments ; isCommit = true]
   [junit4]   2> 4977 T28 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-6ec0e8a7)),segFN=segments_1,generation=1}
   [junit4]   2> 4979 T28 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4979 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: 6 msec to checkpoint
   [junit4]   2> 4981 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: took 17.0 msec
   [junit4]   2> 4981 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: commit: done
   [junit4]   2> 4982 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: rollback
   [junit4]   2> 4982 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: all running merges have aborted
   [junit4]   2> 4982 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: rollback: done finish merges
   [junit4]   2> 4983 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: abort
   [junit4]   2> 4984 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: done abort; abortedFiles=[] success=true
   [junit4]   2> 4985 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: rollback: infos=
   [junit4]   2> 4985 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 4985 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: 0 msec to checkpoint
   [junit4]   2> 4992 T28 oasc.SolrCore.initWriters created json: solr.JSONResponseWriter
   [junit4]   2> 4992 T28 oasc.SolrCore.initWriters adding lazy queryResponseWriter: solr.VelocityResponseWriter
   [junit4]   2> 4994 T28 oasc.SolrCore.initWriters created velocity: solr.VelocityResponseWriter
   [junit4]   2> 5001 T28 oasc.SolrCore.initWriters created xslt: solr.XSLTResponseWriter
   [junit4]   2> 5002 T28 oasr.XSLTResponseWriter.init xsltCacheLifetimeSeconds=5
   [junit4]   2> 5244 T28 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 5256 T28 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 5257 T28 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 5257 T28 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 5258 T28 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 5269 T28 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 5269 T28 oasc.RequestHandlers.initHandlersFromConfig created /query: solr.SearchHandler
   [junit4]   2> 5273 T28 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 5273 T28 oasc.RequestHandlers.initHandlersFromConfig created /export: solr.SearchHandler
   [junit4]   2> 5274 T28 oasc.RequestHandlers.initHandlersFromConfig created /browse: solr.SearchHandler
   [junit4]   2> 5274 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.extraction.ExtractingRequestHandler
   [junit4]   2> 5275 T28 oasc.RequestHandlers.initHandlersFromConfig created /update/extract: solr.extraction.ExtractingRequestHandler
   [junit4]   2> 5275 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.FieldAnalysisRequestHandler
   [junit4]   2> 5276 T28 oasc.RequestHandlers.initHandlersFromConfig created /analysis/field: solr.FieldAnalysisRequestHandler
   [junit4]   2> 5276 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 5276 T28 oasc.RequestHandlers.initHandlersFromConfig created /analysis/document: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 5281 T28 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 5285 T28 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 5289 T28 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 5303 T28 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 5303 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 5303 T28 oasc.RequestHandlers.initHandlersFromConfig created /spell: solr.SearchHandler
   [junit4]   2> 5304 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 5304 T28 oasc.RequestHandlers.initHandlersFromConfig created /suggest: solr.SearchHandler
   [junit4]   2> 5304 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 5305 T28 oasc.RequestHandlers.initHandlersFromConfig created /tvrh: solr.SearchHandler
   [junit4]   2> 5305 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 5305 T28 oasc.RequestHandlers.initHandlersFromConfig created /terms: solr.SearchHandler
   [junit4]   2> 5306 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 5306 T28 oasc.RequestHandlers.initHandlersFromConfig created /elevate: solr.SearchHandler
   [junit4]   2> 5332 T28 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 5337 T28 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 5338 T28 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 5339 T28 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 5365 T28 oasu.CommitTracker.<init> Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 5365 T28 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 5370 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: init: current segments file is "segments_1"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@68f457aa
   [junit4]   2> 5371 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: init: load commit "segments_1"
   [junit4]   2> 5374 T28 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-6ec0e8a7)),segFN=segments_1,generation=1}
   [junit4]   2> 5375 T28 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 5376 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 5377 T28 oasu.LoggingInfoStream.message [IFD][coreLoadExecutor-5-thread-1]: 0 msec to checkpoint
   [junit4]   2> 5378 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: init: create=false
   [junit4]   2> 5379 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: 
   [junit4]   2> 	dir=MockDirectoryWrapper(RAMDirectory(id=lucene-6ec0e8a7))
   [junit4]   2> 	index=
   [junit4]   2> 	version=5.0.0
   [junit4]   2> 	analyzer=null
   [junit4]   2> 	ramBufferSizeMB=100.0
   [junit4]   2> 	maxBufferedDocs=-1
   [junit4]   2> 	maxBufferedDeleteTerms=-1
   [junit4]   2> 	mergedSegmentWarmer=null
   [junit4]   2> 	delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
   [junit4]   2> 	commit=null
   [junit4]   2> 	openMode=APPEND
   [junit4]   2> 	similarity=org.apache.lucene.search.similarities.DefaultSimilarity
   [junit4]   2> 	mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=1, maxMergeCount=2, mergeThreadPriority=-1
   [junit4]   2> 	default WRITE_LOCK_TIMEOUT=1000
   [junit4]   2> 	writeLockTimeout=1000
   [junit4]   2> 	codec=Lucene410
   [junit4]   2> 	infoStream=org.apache.solr.update.LoggingInfoStream
   [junit4]   2> 	mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 	indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@5b7ffdd6
   [junit4]   2> 	readerPooling=false
   [junit4]   2> 	perThreadHardLimitMB=1945
   [junit4]   2> 	useCompoundFile=false
   [junit4]   2> 	checkIntegrityAtMerge=false
   [junit4]   2> 	commitOnClose=true
   [junit4]   2> 	writer=org.apache.lucene.util.SetOnce@2ff21b88
   [junit4]   2> 	
   [junit4]   2> 5382 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: flush at getReader
   [junit4]   2> 5382 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: startFullFlush
   [junit4]   2> 5383 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 5383 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: apply all deletes during flush
   [junit4]   2> 5384 T28 oasu.LoggingInfoStream.message [BD][coreLoadExecutor-5-thread-1]: prune sis=segments_1:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 5386 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: return reader version=1 reader=StandardDirectoryReader(segments_1:1:nrt)
   [junit4]   2> 5387 T28 oasu.LoggingInfoStream.message [DW][coreLoadExecutor-5-thread-1]: coreLoadExecutor-5-thread-1 finishFullFlush success=true
   [junit4]   2> 5387 T28 oasu.LoggingInfoStream.message [IW][coreLoadExecutor-5-thread-1]: getReader took 5 msec
   [junit4]   2> 5405 T28 oass.SolrIndexSearcher.<init> Opening Searcher@5aae7573[collection1] main
   [junit4]   2> 5415 T28 oasr.ManagedResourceStorage.newStorageIO WARN Cannot write to config directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf; switching to use InMemory storage instead.
   [junit4]   2> 5417 T28 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 5419 T28 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 5420 T28 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 5422 T28 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 5422 T28 oasr.RestManager.init Initializing 2 registered ManagedResources
   [junit4]   2> 5423 T28 oasr.ManagedResourceStorage.load Reading _schema_analysis_stopwords_english.json using InMemoryStorage
   [junit4]   2> 5423 T28 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /schema/analysis/stopwords/english
   [junit4]   2> 5424 T28 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _schema_analysis_stopwords_english.json using InMemoryStorage
   [junit4]   2> 5424 T28 oasrsa.ManagedWordSetResource.onManagedDataLoadedFromStorage Loaded 0 words for /schema/analysis/stopwords/english
   [junit4]   2> 5425 T28 oasr.ManagedResource.notifyObserversDuringInit Notified 1 observers of /schema/analysis/stopwords/english
   [junit4]   2> 5425 T28 oasr.ManagedResourceStorage.load Reading _schema_analysis_synonyms_english.json using InMemoryStorage
   [junit4]   2> 5426 T28 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /schema/analysis/synonyms/english
   [junit4]   2> 5426 T28 oasrsa.ManagedSynonymFilterFactory$SynonymManager.onManagedDataLoadedFromStorage Loaded 0 synonym mappings for /schema/analysis/synonyms/english
   [junit4]   2> 5429 T28 oasr.ManagedResource.notifyObserversDuringInit Notified 1 observers of /schema/analysis/synonyms/english
   [junit4]   2> 5430 T28 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 5449 T28 oass.DirectSolrSpellChecker.init init: {name=default,field=text,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 5466 T28 oashc.SpellCheckComponent.inform No queryConverter defined, using default converter
   [junit4]   2> 5468 T28 oashc.SuggestComponent.inform Initializing SuggestComponent
   [junit4]   2> 5471 T28 oasss.SolrSuggester.init init: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string}
   [junit4]   2> 5519 T28 oasss.SolrSuggester.init Dictionary loaded with params: {name=mySuggester,lookupImpl=FuzzyLookupFactory,dictionaryImpl=DocumentDictionaryFactory,field=cat,weightField=price,suggestAnalyzerFieldType=string}
   [junit4]   2> 5525 T28 oashc.QueryElevationComponent.inform Loading QueryElevation from: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/example/solr/collection1/conf/elevate.xml
   [junit4]   2> 5585 T28 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 5587 T28 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 5587 T29 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@5aae7573[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 5591 T16 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 5592 T16 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 5592 T16 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#56834
   [junit4]   2> 5614 T16 oas.SolrTestCaseJ4.setUp ###Starting testQueryWithParams
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 5690 T29 C0 oasc.SolrCore.execute [collection1] webapp=null path=null params={distrib=false&q=static+firstSearcher+warming+in+solrconfig.xml&event=firstSearcher} hits=0 status=0 QTime=99 
   [junit4]   2> 5693 T29 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 5693 T29 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: default
   [junit4]   2> 5693 T29 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: wordbreak
   [junit4]   2> 5694 T29 oashc.SuggestComponent$SuggesterListener.newSearcher Loading suggester index for: mySuggester
   [junit4]   2> 5694 T29 oasss.SolrSuggester.reload reload()
   [junit4]   2> 5695 T29 oasss.SolrSuggester.build build()
   [junit4]   2> 6044 T29 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5aae7573[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 6154 T25 C1 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={version=2.2&wt=xml&debug=true&q=query} hits=0 status=0 QTime=6 
   [junit4]   2> 6336 T16 oas.SolrTestCaseJ4.tearDown ###Ending testQueryWithParams
   [junit4]   2> 6357 T16 oas.SolrTestCaseJ4.setUp ###Starting testChildDoctransformer
   [junit4]   2> 6413 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6414 T27 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-27]: nrtIsCurrent: infoVersion matches: true; DW changes: true; BD changes: false
   [junit4]   2> 6415 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6415 T27 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-27]: flush at getReader
   [junit4]   2> 6416 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: startFullFlush
   [junit4]   2> 6417 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: anyChanges? numDocsInRam=0 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6417 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: qtp2049549722-27: flush naked frozen global deletes
   [junit4]   2> 6424 T27 C1 oasu.LoggingInfoStream.message [BD][qtp2049549722-27]: push deletes  1 deleted queries bytesUsed=84 delGen=2 packetCount=1 totBytesUsed=84
   [junit4]   2> 6425 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: flush: push buffered updates:  1 deleted queries bytesUsed=84
   [junit4]   2> 6425 T27 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-27]: apply all deletes during flush
   [junit4]   2> 6426 T27 C1 oasu.LoggingInfoStream.message [BD][qtp2049549722-27]: prune sis=segments_1:  minGen=9223372036854775807 packetCount=1
   [junit4]   2> 6427 T27 C1 oasu.LoggingInfoStream.message [BD][qtp2049549722-27]: pruneDeletes: prune 1 packets; 0 packets remain
   [junit4]   2> 6427 T27 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-27]: return reader version=1 reader=StandardDirectoryReader(segments_1:1:nrt)
   [junit4]   2> 6428 T27 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-27]: qtp2049549722-27 finishFullFlush success=true
   [junit4]   2> 6428 T27 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-27]: getReader took 13 msec
   [junit4]   2> 6432 T27 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {deleteByQuery=*:* (-1479512258681765888)} 0 56
   [junit4]   2> 6444 T25 C1 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 6446 T25 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-25]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6447 T25 C1 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 6448 T25 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-25]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6449 T25 C1 oasu.LoggingInfoStream.message [IW][qtp2049549722-25]: nrtIsCurrent: infoVersion matches: true; DW changes: false; BD changes: false
   [junit4]   2> 6450 T25 C1 oasu.LoggingInfoStream.message [DW][qtp2049549722-25]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 6451 T25 C1 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 6453 T25 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 6454 T25 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml&softCommit=false&commit=true&waitSearcher=true} {commit=} 0 10
   [junit4]   2> 6634 T26 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[0 (1479512258802352128)]} 0 159
   [junit4]   2> 6642 T24 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[215 (1479512258949152768)]} 0 2
   [junit4]   2> 6651 T27 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[216 (1479512258958589952)]} 0 2
   [junit4]   2> 6758 T25 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[217 (1479512259006824448)]} 0 86
   [junit4]   2> 6784 T26 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[404 (1479512259085467648)]} 0 18
   [junit4]   2> 6790 T27 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[435 (1479512259106439168)]} 0 0
   [junit4]   2> 6856 T25 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[436 (1479512259139993600)]} 0 54
   [junit4]   2> 7015 T26 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[532 (1479512259223879680)]} 0 142
   [junit4]   2> 7068 T24 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[812 (1479512259360194560)]} 0 43
   [junit4]   2> 7073 T25 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[949 (1479512259402137600)]} 0 1
   [junit4]   2> 7078 T26 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[950 (1479512259407380480)]} 0 2
   [junit4]   2> 7215 T27 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[952 (1479512259450372096)]} 0 117
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 7332 T25 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[1413 (1479512259601367040)]} 0 80
   [junit4]   2> 7377 T26 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[1955 (1479512259690496000)]} 0 34
   [junit4]   2> 7399 T27 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[2182 (1479512259730341888)]} 0 16
   [junit4]   2> 7425 T24 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[2304 (1479512259754459136)]} 0 19
   [junit4]   2> 7554 T25 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[2460 (1479512259816325120)]} 0 107
   [junit4]   2> 7615 T26 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[3563 (1479512259926425600)]} 0 47
   [junit4]   2> 7620 T27 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4198 (1479512259975708672)]} 0 0
   [junit4]   2> 7623 T25 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4199 (1479512259979902976)]} 0 0
   [junit4]   2> 7668 T26 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4200 (1479512259996680192)]} 0 33
   [junit4]   2> 7673 T24 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4780 (1479512260031283200)]} 0 1
   [junit4]   2> 7682 T27 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4781 (1479512260037574656)]} 0 5
   [junit4]   2> 7725 T25 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[4876 (1479512260061691904)]} 0 27
   [junit4]   2> 7732 T26 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[5514 (1479512260092100608)]} 0 2
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 7816 T24 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[5567 (1479512260149772288)]} 0 61
   [junit4]   2> 7822 T27 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[6633 (1479512260187521024)]} 0 1
   [junit4]   2> 7862 T25 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[6661 (1479512260213735424)]} 0 20
   [junit4]   2> 7881 T24 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[7452 (1479512260243095552)]} 0 9
   [junit4]   2> 7897 T27 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[7769 (1479512260260921344)]} 0 7
   [junit4]   2> 7901 T26 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8043 (1479512260271407104)]} 0 0
   [junit4]   2> 7905 T25 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8044 (1479512260275601408)]} 0 0
   [junit4]   2> 7909 T24 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8045 (1479512260279795712)]} 0 0
   [junit4]   2> 7913 T27 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8050 (1479512260283990016)]} 0 1
   [junit4]   2> 7917 T26 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8051 (1479512260287135744)]} 0 1
   [junit4]   2> 7940 T25 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8052 (1479512260300767232)]} 0 13
   [junit4]   2> 7945 T24 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8519 (1479512260316495872)]} 0 1
   [junit4]   2> 7974 T27 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[8520 (1479512260332224512)]} 0 18
   [junit4]   2> 7979 T26 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[9207 (1479512260352147456)]} 0 1
   [junit4]   2> 8002 T25 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[9208 (1479512260364730368)]} 0 14
   [junit4]   2> 8020 T24 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[9753 (1479512260387799040)]} 0 10
   [junit4]   2> 8024 T26 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10108 (1479512260400381952)]} 0 0
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 8053 T25 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10109 (1479512260409819136)]} 0 21
   [junit4]   2> 8067 T24 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10496 (1479512260439179264)]} 0 7
   [junit4]   2> 8072 T27 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10790 (1479512260450713600)]} 0 0
   [junit4]   2> 8076 T26 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10791 (1479512260454907904)]} 0 0
   [junit4]   2> 8095 T25 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[10792 (1479512260466442240)]} 0 11
   [junit4]   2> 8101 T24 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[11154 (1479512260480073728)]} 0 2
   [junit4]   2> 8120 T27 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[11198 (1479512260491608064)]} 0 11
   [junit4]   2> 8155 T26 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[11603 (1479512260519919616)]} 0 21
   [junit4]   2> 8160 T25 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[12493 (1479512260541939712)]} 0 1
   [junit4]   2> 8164 T24 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[12494 (1479512260546134016)]} 0 1
   [junit4]   2> 8176 T27 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[12495 (1479512260554522624)]} 0 6
   [junit4]   2> 8193 T26 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[12721 (1479512260569202688)]} 0 9
   [junit4]   2> 8210 T25 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[13090 (1479512260588077056)]} 0 9
   [junit4]   2> 8239 T24 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[13461 (1479512260611145728)]} 0 17
   [junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 8294 T27 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[14166 (1479512260640505856)]} 0 45
   [junit4]   2> 8299 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[15192 (1479512260688740352)]} 0 1
   [junit4]   2> 8303 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[15194 (1479512260692934656)]} 0 1
   [junit4]   2> 8312 T26 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[15196 (1479512260698177536)]} 0 5
   [junit4]   2> 8327 T27 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[15384 (1479512260709711872)]} 0 9
   [junit4]   2> 8340 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[15635 (1479512260724391936)]} 0 8
   [junit4]   2> 8349 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16006 (1479512260738023424)]} 0 4
   [junit4]   2> 8354 T26 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16254 (1479512260745363456)]} 0 1
   [junit4]   2> 8368 T27 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16255 (1479512260752703488)]} 0 10
   [junit4]   2> 8372 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16579 (1479512260765286400)]} 0 0
   [junit4]   2> 8378 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16580 (1479512260770529280)]} 0 2
   [junit4]   2> 8382 T26 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16748 (1479512260775772160)]} 0 0
   [junit4]   2> 8386 T27 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16749 (1479512260779966464)]} 0 0
   [junit4]   2> 8392 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16750 (1479512260785209344)]} 0 2
   [junit4]   2> 8404 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[16857 (1479512260793597952)]} 0 7
   [junit4]   2> 8408 T26 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[17291 (1479512260803035136)]} 0 0
   [junit4]   2> 8413 T27 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[17293 (1479512260807229440)]} 0 1
   [junit4]   2> 8417 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[17335 (1479512260811423744)]} 0 0
   [junit4]   2> 8420 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[17351 (1479512260815618048)]} 0 0
   [junit4]   2> 8443 T26 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[17352 (1479512260827152384)]} 0 17
   [junit4]   2> 8448 T25 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[18423 (1479512260843929600)]} 0 1
   [junit4]   2> 8461 T24 C5 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[18424 (1479512260852318208)]} 0 9
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@4161a430
   [junit4]   2> 8490 T27 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[19014 (1479512260880629760)]} 0 24
   [junit4]   2> 8512 T26 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[19520 (1479512260901601280)]} 0 13
   [junit4]   2> 8525 T25 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[20303 (1479512260919427072)]} 0 8
   [junit4]   2> 8547 T24 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[20774 (1479512260938301440)]} 0 16
   [junit4]   2> 8552 T27 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[21619 (1479512260954030080)]} 0 1
   [junit4]   2> 8559 T26 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[21620 (1479512260960321536)]} 0 2
   [junit4]   2> 8572 T25 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[21686 (1479512260968710144)]} 0 8
   [junit4]   2> 8596 T24 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[21943 (1479512260987584512)]} 0 16
   [junit4]   2> 8607 T27 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[22543 (1479512261008556032)]} 0 5
   [junit4]   2> 8613 T26 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[22852 (1479512261015896064)]} 0 3
   [junit4]   2> 8617 T25 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[23004 (1479512261021138944)]} 0 1
   [junit4]   2> 8620 T24 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[23005 (1479512261025333248)]} 0 0
   [junit4]   2> 8623 T27 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2.2&wt=xml} {add=[23006 (1479512261028478976)]} 0 0
   [junit4]   2> 8627 T26 C6 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 8629 T26 C6 oasu.LoggingInfoStream.message [IW][qtp2049549722-26]: commit: start
   [junit4]   2> 8629 T26 C6 oasu.LoggingInfoStream.message [IW][qtp2049549722-26]: commit: enter lock
   [junit4]   2> 8630 T26 C6 oasu.LoggingInfoStream.message [IW][qtp2049549722-26]: commit: now prepare
   [junit4]   2> 8630 T26 C6 oasu.LoggingInfoStream.message [IW][qtp2049549722-26]: prepareCommit: flush
   [junit4]   2> 8631 T26 C6 oasu.LoggingInfoStream.message [IW][qtp2049549722-26]:   index before flush 
   [junit4]   2> 8631 T26 C6 oasu.LoggingInfoStream.message [DW][qtp2049549722-26]: startFullFlush
   [junit4]   2> 8632 T26 C6 oasu.LoggingInfoStream.message [DW][qtp2049549722-26]: anyChanges? numDocsInRam=23007 deletes=true hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 8633 T26 C6 oasu.LoggingInfoStream.message [DWFC][qtp2049549722-26]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 91 deleted terms (unique count=91) bytesUsed=21652, segment=_0, aborting=false, numDocsInRAM=2300

[...truncated too long message...]

message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: now flush at close
   [junit4]   2> 156164 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   start flush: applyAllDeletes=true
   [junit4]   2> 156165 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   index before flush 
   [junit4]   2> 156165 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: startFullFlush
   [junit4]   2> 156165 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 156165 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker finishFullFlush success=true
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: apply all deletes during flush
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [BD][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: prune sis=segments_1y:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: now merge
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   index: 
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [CMS][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   no more merges pending; now return
   [junit4]   2> 156166 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: waitForMerges
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: waitForMerges done
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: commit: start
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: commit: enter lock
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: commit: now prepare
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: prepareCommit: flush
   [junit4]   2> 156167 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   index before flush 
   [junit4]   2> 156168 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: startFullFlush
   [junit4]   2> 156168 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: anyChanges? numDocsInRam=0 deletes=false hasTickets:false pendingChangesInFullFlush: false
   [junit4]   2> 156168 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: apply all deletes during flush
   [junit4]   2> 156168 T286 C172 oasu.LoggingInfoStream.message [BD][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: prune sis=segments_1y:  minGen=9223372036854775807 packetCount=0
   [junit4]   2> 156168 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker finishFullFlush success=true
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: startCommit(): start
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]:   skip startCommit(): no changes pending
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: commit: pendingCommit == null; skip
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: commit: done
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: rollback
   [junit4]   2> 156169 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: all running merges have aborted
   [junit4]   2> 156170 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: rollback: done finish merges
   [junit4]   2> 156170 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: abort
   [junit4]   2> 156170 T286 C172 oasu.LoggingInfoStream.message [DW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: done abort; abortedFiles=[] success=true
   [junit4]   2> 156170 T286 C172 oasu.LoggingInfoStream.message [IW][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: rollback: infos=
   [junit4]   2> 156170 T286 C172 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: now checkpoint "" [0 segments ; isCommit = false]
   [junit4]   2> 156171 T286 C172 oasu.LoggingInfoStream.message [IFD][SUITE-SolrExampleEmbeddedTest-seed#[20CF0595A5B13DAC]-worker]: 0 msec to checkpoint
   [junit4]   2> 156171 T286 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 156172 T286 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 156172 T286 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001;done=false>>]
   [junit4]   2> 156172 T286 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001
   [junit4]   2> 156172 T286 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 156172 T286 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001/init-core-data-001/index
   [junit4]   2> 156173 T286 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 156173 T286 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1312837549
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.embedded.SolrExampleEmbeddedTest-20CF0595A5B13DAC-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ko, timezone=Europe/Athens
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=48796720,total=127401984
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastInputStream, MultiCoreExampleJettyTest, TestSolrProperties, SolrDocumentTest, MergeIndexesEmbeddedTest, SolrQueryTest, SolrExampleJettyTest, TestLBHttpSolrServer, QueryResponseTest, SolrExampleBinaryTest, TestRetryUtil, TestHash, SolrExampleEmbeddedTest]
   [junit4] Completed on J1 in 7.67s, 28 tests, 1 failure <<< FAILURES!

[...truncated 81 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 53 suites, 313 tests, 5 errors, 1 failure

Total time: 93 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b28) - Build # 11271 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11271/
Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 51533 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:430: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:105: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:208: Source checkout is dirty after running tests!!! Offending files:
* ./solr/contrib/analytics/test-lib

Total time: 105 minutes 1 second
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11270 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11270/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.NullPointerException
	at __randomizedtesting.SeedInfo.seed([EEECF3B1EDC920AE:6F0A7DA99A964092]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:706)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:728)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:194)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11486 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/init-core-data-001
   [junit4]   2> 1473542 T4128 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1473542 T4128 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1473546 T4128 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1473547 T4128 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1473548 T4129 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1473648 T4128 oasc.ZkTestServer.run start zk server on port:58419
   [junit4]   2> 1473648 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1473650 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1473653 T4135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d6d40ff name:ZooKeeperConnection Watcher:127.0.0.1:58419 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1473653 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1473653 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1473653 T4128 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1473656 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1473657 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1473658 T4137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58201c70 name:ZooKeeperConnection Watcher:127.0.0.1:58419/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1473658 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1473658 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1473659 T4128 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1473662 T4128 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1473664 T4128 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1473665 T4128 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1473666 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1473666 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1473667 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1473667 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1473669 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1473669 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1473670 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1473670 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1473671 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1473671 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1473672 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1473673 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1473674 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1473674 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1473675 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1473675 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1473677 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1473677 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1473678 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1473678 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1473679 T4128 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1473679 T4128 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1473681 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1473681 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1473682 T4139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79014e37 name:ZooKeeperConnection Watcher:127.0.0.1:58419/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1473683 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1473683 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1473928 T4128 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1473930 T4128 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1473995 T4128 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60333
   [junit4]   2> 1473996 T4128 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1473996 T4128 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1473997 T4128 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002
   [junit4]   2> 1473997 T4128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/'
   [junit4]   2> 1474006 T4128 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/solr.xml
   [junit4]   2> 1474013 T4128 oasc.CoreContainer.<init> New CoreContainer 1061219062
   [junit4]   2> 1474013 T4128 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/]
   [junit4]   2> 1474013 T4128 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1474014 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1474015 T4128 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1474015 T4128 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1474016 T4128 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1474016 T4128 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1474017 T4128 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1474017 T4128 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1474017 T4128 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58419/solr
   [junit4]   2> 1474017 T4128 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1474018 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1474018 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1474020 T4150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c2b78c7 name:ZooKeeperConnection Watcher:127.0.0.1:58419 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1474020 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1474020 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1474022 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1474023 T4152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@414c7539 name:ZooKeeperConnection Watcher:127.0.0.1:58419/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1474023 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1474025 T4128 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1474028 T4128 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1474032 T4128 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1474034 T4128 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1474037 T4128 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1474039 T4128 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1474040 T4128 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60333_
   [junit4]   2> 1474040 T4128 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60333_
   [junit4]   2> 1474041 T4128 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1474042 T4128 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1474043 T4128 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1474044 T4128 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60333_
   [junit4]   2> 1474044 T4128 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1474045 T4128 oasc.Overseer.start Overseer (id=92468659043631108-127.0.0.1:60333_-n_0000000000) starting
   [junit4]   2> 1474046 T4128 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1474053 T4128 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1474054 T4154 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1474054 T4128 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1474056 T4128 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1474058 T4128 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1474062 T4153 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1474064 T4156 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1474064 T4156 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1474065 T4156 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1474065 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1474068 T4153 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60333",
   [junit4]   2> 	  "node_name":"127.0.0.1:60333_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1474068 T4153 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1474068 T4153 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1474069 T4153 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1474071 T4157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1475065 T4156 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1475066 T4156 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1475066 T4156 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1475067 T4156 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1475068 T4156 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1475068 T4156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/collection1/'
   [junit4]   2> 1475069 T4156 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1475069 T4156 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1475070 T4156 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1475100 T4156 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1475116 T4156 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1475117 T4156 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1475122 T4156 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1475224 T4156 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1475225 T4156 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1475226 T4156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1475229 T4156 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1475243 T4156 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1475243 T4156 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1475244 T4156 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/
   [junit4]   2> 1475244 T4156 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f7fa2b4
   [junit4]   2> 1475245 T4156 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data
   [junit4]   2> 1475246 T4156 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index/
   [junit4]   2> 1475246 T4156 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1475247 T4156 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index
   [junit4]   2> 1475247 T4156 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1475471 T4156 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1475471 T4156 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1475474 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1475475 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1475475 T4156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1475475 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1475476 T4156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1475476 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1475476 T4156 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1475477 T4156 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1475477 T4156 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1475477 T4156 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1475478 T4156 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1475478 T4156 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1475478 T4156 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1475479 T4156 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1475479 T4156 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1475479 T4156 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1475480 T4156 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1475480 T4156 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1475481 T4156 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1475481 T4156 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1475483 T4156 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1475484 T4156 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1475486 T4156 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1475487 T4156 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1475488 T4156 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1475489 T4156 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1475489 T4156 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7320202192153836]
   [junit4]   2> 1475490 T4156 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1475491 T4156 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1475491 T4156 oass.SolrIndexSearcher.<init> Opening Searcher@600ad2d9[collection1] main
   [junit4]   2> 1475491 T4156 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1475493 T4156 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1475493 T4156 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1475494 T4156 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1475494 T4156 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1475494 T4156 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1475495 T4156 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1475496 T4156 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1475497 T4156 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1475499 T4156 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1475499 T4156 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1475500 T4156 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1475502 T4158 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@600ad2d9[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1475503 T4156 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1475504 T4161 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60333 collection:control_collection shard:shard1
   [junit4]   2> 1475505 T4128 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1475505 T4128 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1475505 T4161 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1475507 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1475508 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1475509 T4163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52df06ff name:ZooKeeperConnection Watcher:127.0.0.1:58419/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1475509 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1475509 T4161 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1475510 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1475511 T4128 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1475511 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1475511 T4161 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1475511 T4161 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2955 name=collection1 org.apache.solr.core.SolrCore@69edc412 url=https://127.0.0.1:60333/collection1 node=127.0.0.1:60333_ C2955_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:60333, node_name=127.0.0.1:60333_, state=down}
   [junit4]   2> 1475511 T4161 C2955 P60333 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60333/collection1/
   [junit4]   2> 1475512 T4161 C2955 P60333 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1475512 T4161 C2955 P60333 oasc.SyncStrategy.syncToMe https://127.0.0.1:60333/collection1/ has no replicas
   [junit4]   2> 1475512 T4161 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60333/collection1/ shard1
   [junit4]   2> 1475512 T4161 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1475512 T4128 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1475514 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1475616 T4157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1475616 T4164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1475665 T4161 oasc.ZkController.register We are https://127.0.0.1:60333/collection1/ and leader is https://127.0.0.1:60333/collection1/
   [junit4]   2> 1475665 T4161 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60333
   [junit4]   2> 1475665 T4161 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1475665 T4161 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1475666 T4161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1475666 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1475669 T4153 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60333",
   [junit4]   2> 	  "node_name":"127.0.0.1:60333_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1475740 T4128 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1475740 T4128 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1475743 T4128 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1475743 T4128 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43992
   [junit4]   2> 1475745 T4128 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1475745 T4128 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1475746 T4128 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003
   [junit4]   2> 1475746 T4128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/'
   [junit4]   2> 1475763 T4128 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/solr.xml
   [junit4]   2> 1475777 T4164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1475777 T4157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1475779 T4128 oasc.CoreContainer.<init> New CoreContainer 191519566
   [junit4]   2> 1475780 T4128 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/]
   [junit4]   2> 1475780 T4128 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1475780 T4128 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1475781 T4128 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1475781 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1475781 T4128 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1475781 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1475782 T4128 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1475782 T4128 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1475782 T4128 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1475783 T4128 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1475784 T4128 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1475785 T4128 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1475785 T4128 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1475785 T4128 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58419/solr
   [junit4]   2> 1475786 T4128 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1475786 T4128 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1475787 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1475789 T4175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35a7a69d name:ZooKeeperConnection Watcher:127.0.0.1:58419 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1475790 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1475790 T4128 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1475793 T4128 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1475795 T4177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cb6b549 name:ZooKeeperConnection Watcher:127.0.0.1:58419/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1475796 T4128 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1475800 T4128 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1476802 T4128 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43992_
   [junit4]   2> 1476803 T4128 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43992_
   [junit4]   2> 1476807 T4128 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1476813 T4179 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1476813 T4179 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1476814 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1476814 T4179 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1476819 T4153 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43992",
   [junit4]   2> 	  "node_name":"127.0.0.1:43992_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1476819 T4153 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1476819 T4153 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 1476820 T4153 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1476923 T4157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1476923 T4164 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1476923 T4178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1477815 T4179 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1477815 T4179 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1477816 T4179 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1477816 T4179 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1477817 T4179 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1477818 T4179 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/collection1/'
   [junit4]   2> 1477818 T4179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1477819 T4179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1477819 T4179 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1477844 T4179 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1477852 T4179 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1477854 T4179 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1477857 T4179 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1477936 T4179 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1477937 T4179 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1477938 T4179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1477940 T4179 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1477951 T4179 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1477951 T4179 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1477951 T4179 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/
   [junit4]   2> 1477952 T4179 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f7fa2b4
   [junit4]   2> 1477952 T4179 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1
   [junit4]   2> 1477952 T4179 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index/
   [junit4]   2> 1477953 T4179 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1477953 T4179 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index
   [junit4]   2> 1477953 T4179 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1478120 T4179 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1478121 T4179 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1478123 T4179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1478123 T4179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1478123 T4179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1478123 T4179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1478124 T4179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1478124 T4179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1478124 T4179 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1478124 T4179 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1478124 T4179 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1478125 T4179 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1478125 T4179 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1478125 T4179 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1478125 T4179 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1478125 T4179 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1478126 T4179 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1478126 T4179 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1478126 T4179 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1478126 T4179 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1478126 T4179 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1478127 T4179 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1478128 T4179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1478128 T4179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1478129 T4179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1478129 T4179 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1478130 T4179 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1478131 T4179 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1478131 T4179 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=12, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7320202192153836]
   [junit4]   2> 1478132 T4179 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1478132 T4179 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1478132 T4179 oass.SolrIndexSearcher.<init> Opening Searcher@6d6109c3[collection1] main
   [junit4]   2> 1478132 T4179 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1478133 T4179 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1478133 T4179 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1478133 T4179 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1478134 T4179 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1478134 T4179 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1478135 T4179 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1478135 T4179 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1478135 T4179 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1478135 T4179 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1478136 T4180 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d6109c3[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1478138 T4179 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1478138 T4183 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43992 collection:collection1 shard:shard1
   [junit4]   2> 1478139 T4128 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1478139 T4128 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1478139 T4183 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1478141 T4128 oasc.AbstractFullDistribZkTestBase.randomlyEnableAutoSoftCommit Not turning on auto soft commit
   [junit4]   2> 1478142 T4183 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1478143 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1478144 T4183 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1478144 T4183 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2956 name=collection1 org.apache.solr.core.SolrCore@787ed3a7 url=https://127.0.0.1:43992/collection1 node=127.0.0.1:43992_ C2956_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:43992, node_name=127.0.0.1:43992_, state=down}
   [junit4]   2> 1478145 T4183 C2956 P43992 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43992/collection1/
   [junit4]   2> 1478145 T4183 C2956 P43992 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1478145 T4183 C2956 P43992 oasc.SyncStrategy.syncToMe https://127.0.0.1:43992/collection1/ has no replicas
   [junit4]   2> 1478145 T4183 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43992/collection1/ shard1
   [junit4]   2> 1478145 T4183 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1478149 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C2957 name=collection1 org.apache.solr.core.SolrCore@69edc412 url=https://127.0.0.1:60333/collection1 node=127.0.0.1:60333_ C2957_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:60333, node_name=127.0.0.1:60333_, state=active, leader=true}
   [junit4]   2> 1478205 T4148 C2957 P60333 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1479498549579218944)} 0 1
   [junit4]   2> 1478207 T4128 oascsi.CloudSolrServer.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 0
   [junit4]   2> 1478210 T4128 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60333
   [junit4]   2> 1478210 T4128 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1061219062
   [junit4]   2> 1478211 T4128 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1478211 T4128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1478212 T4157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1478212 T4128 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@69edc412
   [junit4]   2> 1478213 T4153 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60333",
   [junit4]   2> 	  "node_name":"127.0.0.1:60333_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1478215 T4128 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1478216 T4128 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1478216 T4128 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1478217 T4128 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1478217 T4128 C2957 P60333 oasu.DirectUpdateHandler2.closeWriter Committing on IndexWriter close.
   [junit4]   2> 1478222 T4128 C2957 P60333 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index,segFN=segments_2,generation=2}
   [junit4]   2> 1478223 T4128 C2957 P60333 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1478224 T4128 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1478225 T4128 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1478225 T4128 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data;done=false>>]
   [junit4]   2> 1478226 T4128 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data
   [junit4]   2> 1478226 T4128 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index;done=false>>]
   [junit4]   2> 1478226 T4128 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/control/data/index
   [junit4]   2> 1478227 T4128 oasc.Overseer.close Overseer (id=92468659043631108-127.0.0.1:60333_-n_0000000000) closing
   [junit4]   2> 1478228 T4153 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60333_
   [junit4]   2> 1478230 T4184 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 1478233 T4178 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43992_
   [junit4]   2> 1478233 T4178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1478235 T4178 oasc.Overseer.start Overseer (id=92468659043631111-127.0.0.1:43992_-n_0000000001) starting
   [junit4]   2> 1478242 T4178 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1478243 T4187 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1478245 T4186 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1478246 T4178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1478249 T4186 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60333",
   [junit4]   2> 	  "node_name":"127.0.0.1:60333_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1478251 T4186 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1478299 T4183 oasc.ZkController.register We are https://127.0.0.1:43992/collection1/ and leader is https://127.0.0.1:43992/collection1/
   [junit4]   2> 1478300 T4183 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43992
   [junit4]   2> 1478300 T4183 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1478300 T4183 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1478300 T4183 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1478302 T4178 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1478307 T4186 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43992",
   [junit4]   2> 	  "node_name":"127.0.0.1:43992_",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1478411 T4178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1479730 T4157 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1479756 T4128 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1484744 T4128 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43992
   [junit4]   2> 1484745 T4128 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=191519566
   [junit4]   2> 1484745 T4128 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1484746 T4128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1484747 T4178 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1484748 T4128 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@787ed3a7
   [junit4]   2> 1484751 T4186 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43992",
   [junit4]   2> 	  "node_name":"127.0.0.1:43992_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1484754 T4178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1484756 T4128 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1484756 T4128 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1484757 T4128 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1484757 T4128 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1484758 T4128 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1484759 T4128 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1484759 T4128 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1;done=false>>]
   [junit4]   2> 1484759 T4128 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1
   [junit4]   2> 1484759 T4128 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index;done=false>>]
   [junit4]   2> 1484760 T4128 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001/tempDir-001/jetty1/index
   [junit4]   2> 1484760 T4128 oasc.Overseer.close Overseer (id=92468659043631111-127.0.0.1:43992_-n_0000000001) closing
   [junit4]   2> 1484760 T4186 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:43992_
   [junit4]   2> 1486263 T4189 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 1486263 T4178 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1486290 T4128 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1491278 T4128 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1491279 T4128 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58419 58419
   [junit4]   2> 1491857 T4129 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58419 58419
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=EEECF3B1EDC920AE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=Africa/Tunis -Dtests.file.encoding=UTF-8
   [junit4] ERROR   18.3s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.NullPointerException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EEECF3B1EDC920AE:6F0A7DA99A964092]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:706)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:285)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.deleteByQuery(SolrServer.java:271)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:728)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:194)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:112)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1491870 T4128 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-EEECF3B1EDC920AE-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=hu_HU, timezone=Africa/Tunis
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=131383776,total=236384256
   [junit4]   2> NOTE: All tests run in this JVM: [OverseerStatusTest, TestRandomMergePolicy, TestPostingsSolrHighlighter, TestOverriddenPrefixQueryForCustomFieldType, TestMultiCoreConfBootstrap, StatsComponentTest, TestQueryUtils, TestCustomSort, ShardRoutingCustomTest, URLClassifyProcessorTest, TestIBSimilarityFactory, TestMissingGroups, TestSerializedLuceneMatchVersion, TestLFUCache, SolrRequestParserTest, TestSchemaResource, DocValuesMissingTest, TestCursorMarkWithoutUniqueKey, TestBM25SimilarityFactory, TestSolrXmlPersistence, TestBlendedInfixSuggestions, HdfsRecoveryZkTest, LoggingHandlerTest, TestCollationFieldDocValues, DefaultValueUpdateProcessorTest, QueryElevationComponentTest, CustomCollectionTest, CacheHeaderTest, TestBinaryField, FieldAnalysisRequestHandlerTest, UUIDFieldTest, TestSolrQueryParser, ScriptEngineTest, DeleteLastCustomShardedReplicaTest, DistributedDebugComponentTest, RankQueryTest, OpenCloseCoreStressTest, TestOmitPositions, TestManagedStopFilterFactory, TestManagedSchemaDynamicFieldResource, TestFileDictionaryLookup, TestFiltering, TestFieldSortValues, DirectSolrConnectionTest, TestSolrQueryParserDefaultOperatorResource, TestReRankQParserPlugin, ExternalCollectionsTest, ResourceLoaderTest, TestPerFieldSimilarity, SliceStateTest, ShardSplitTest, DateFieldTest, SearchHandlerTest, SolrInfoMBeanTest, PrimUtilsTest, TestPHPSerializedResponseWriter, DeleteShardTest, OverseerRolesTest, RollingRestartTest, TestFieldTypeResource, TestRandomFaceting, AnalysisErrorHandlingTest, TestWriterPerf, TestCollationField, ConvertedLegacyTest, UniqFieldsUpdateProcessorFactoryTest, TestUpdate, SynonymTokenizerTest, BinaryUpdateRequestHandlerTest, DistributedTermsComponentTest, SolrPluginUtilsTest, SpatialFilterTest, DistributedFacetPivotSmallTest, TestCSVLoader, HdfsBasicDistributedZk2Test, LeaderElectionIntegrationTest, MigrateRouteKeyTest, MultiThreadedOCPTest, TestRealTimeGet, CurrencyFieldXmlFileTest, TestIntervalFaceting, ConnectionManagerTest, TestNonNRTOpen, TestCSVResponseWriter, SuggesterFSTTest, TestElisionMultitermQuery, OutputWriterTest, SOLR749Test, PolyFieldTest, TestIndexingPerformance, BasicZkTest, TestQueryTypes, TestFunctionQuery, HdfsDirectoryTest, TestPseudoReturnFields, TestNRTOpen, TestZkChroot, TestUtils, BasicDistributedZk2Test, TestCharFilters, TestSearcherReuse, SoftAutoCommitTest, TestManagedSchema, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 18.36s, 1 test, 1 error <<< FAILURES!

[...truncated 669 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1772 tests, 1 error, 45 ignored (23 assumptions)

Total time: 86 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11269 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11269/
Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 60273 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/docs/solr-analytics/overview-summary.html
     [exec]   missing: org.apache.solr.handler.component
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:78: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:548: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:564: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2471: exec returned: 1

Total time: 106 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_67) - Build # 11268 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11268/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 53718 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:430: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:105: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:208: Source checkout is dirty after running tests!!! Offending files:
* ./solr/contrib/analytics/test-lib

Total time: 111 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11267 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11267/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 51657 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:430: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:105: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:208: Source checkout is dirty after running tests!!! Offending files:
* ./solr/contrib/analytics/test-lib

Total time: 97 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b04) - Build # 11266 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11266/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 60279 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/docs/solr-analytics/overview-summary.html
     [exec]   missing: org.apache.solr.handler.component
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:78: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:548: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:564: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2471: exec returned: 1

Total time: 105 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b04 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11265 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11265/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 60050 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/docs/solr-analytics/overview-summary.html
     [exec]   missing: org.apache.solr.handler.component
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:78: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:548: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:564: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2471: exec returned: 1

Total time: 109 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11264 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11264/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:36558/pp, https://127.0.0.1:34877/pp, https://127.0.0.1:44146/pp]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:36558/pp, https://127.0.0.1:34877/pp, https://127.0.0.1:44146/pp]
	at __randomizedtesting.SeedInfo.seed([3E48EDCA82819F90:BFAE63D2F5DEFFAC]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 45 more




Build Log:
[...truncated 11812 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/init-core-data-001
   [junit4]   2> 2053209 T6603 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 2053209 T6603 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pp/
   [junit4]   2> 2053211 T6603 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2053212 T6603 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2053212 T6604 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2053312 T6603 oasc.ZkTestServer.run start zk server on port:37709
   [junit4]   2> 2053313 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053313 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053315 T6610 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23280ef4 name:ZooKeeperConnection Watcher:127.0.0.1:37709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053315 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053316 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053316 T6603 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2053318 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053318 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053319 T6612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ce8700 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053319 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053319 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053320 T6603 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2053321 T6603 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2053322 T6603 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2053322 T6603 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2053323 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2053324 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2053325 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2053326 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2053327 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2053327 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2053328 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2053329 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2053330 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2053330 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2053331 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2053331 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2053332 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2053333 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2053334 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2053334 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2053335 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2053335 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2053336 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2053337 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2053338 T6603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2053338 T6603 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2053339 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053340 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053340 T6614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4da65b88 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053341 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053341 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053631 T6603 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2053634 T6603 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2053635 T6603 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:44146
   [junit4]   2> 2053636 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2053637 T6603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2053637 T6603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002
   [junit4]   2> 2053637 T6603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/'
   [junit4]   2> 2053649 T6603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/solr.xml
   [junit4]   2> 2053654 T6603 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002
   [junit4]   2> 2053655 T6603 oasc.CoreContainer.<init> New CoreContainer 1166025992
   [junit4]   2> 2053655 T6603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/]
   [junit4]   2> 2053656 T6603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2053656 T6603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2053656 T6603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2053656 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2053657 T6603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2053657 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2053657 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2053657 T6603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2053657 T6603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2053659 T6603 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2053659 T6603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2053659 T6603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2053660 T6603 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2053660 T6603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37709/solr
   [junit4]   2> 2053660 T6603 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2053660 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053662 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053664 T6626 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e9701d9 name:ZooKeeperConnection Watcher:127.0.0.1:37709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053664 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053664 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053665 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053666 T6628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d1e43a1 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053666 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053667 T6603 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2053668 T6603 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2053669 T6603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2053670 T6603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2053671 T6603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2053673 T6603 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2053673 T6603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44146_pp
   [junit4]   2> 2053674 T6603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44146_pp
   [junit4]   2> 2053675 T6603 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2053675 T6603 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2053676 T6603 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2053677 T6603 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44146_pp
   [junit4]   2> 2053677 T6603 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2053678 T6603 oasc.Overseer.start Overseer (id=92466216750022660-127.0.0.1:44146_pp-n_0000000000) starting
   [junit4]   2> 2053679 T6603 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2053682 T6603 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2053683 T6630 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2053683 T6603 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2053684 T6603 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2053685 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2053687 T6629 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2053688 T6603 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002
   [junit4]   2> 2053694 T6603 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/conf/
   [junit4]   2> 2053846 T6603 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2053847 T6603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2053847 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2053849 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053849 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053850 T6634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2df15497 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053850 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053850 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053851 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2053853 T6603 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2053853 T6603 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2053853 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2053854 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2053854 T6636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eb72066 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2053854 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2053855 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2053857 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2053859 T6629 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 2053859 T6629 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2053859 T6629 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 2053861 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2053861 T6637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2053861 T6638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2054127 T6603 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2054128 T6603 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2054131 T6603 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2054132 T6603 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36558
   [junit4]   2> 2054134 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2054135 T6603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2054135 T6603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003
   [junit4]   2> 2054135 T6603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/'
   [junit4]   2> 2054151 T6603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/solr.xml
   [junit4]   2> 2054156 T6603 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003
   [junit4]   2> 2054157 T6603 oasc.CoreContainer.<init> New CoreContainer 2030779274
   [junit4]   2> 2054157 T6603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/]
   [junit4]   2> 2054172 T6603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2054173 T6603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2054173 T6603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2054173 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2054174 T6603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2054174 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2054174 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2054174 T6603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2054175 T6603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2054176 T6603 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2054176 T6603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2054177 T6603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2054177 T6603 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2054177 T6603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37709/solr
   [junit4]   2> 2054177 T6603 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2054178 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2054178 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2054179 T6649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d57cf1 name:ZooKeeperConnection Watcher:127.0.0.1:37709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2054179 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2054180 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2054181 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2054182 T6651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@718adc07 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2054182 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2054185 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2055188 T6603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36558_pp
   [junit4]   2> 2055189 T6603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36558_pp
   [junit4]   2> 2055191 T6603 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2055192 T6603 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003
   [junit4]   2> 2055199 T6603 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/conf/
   [junit4]   2> 2055345 T6603 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2055346 T6603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2055346 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2055638 T6603 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2055639 T6603 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2055642 T6603 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2055643 T6603 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34877
   [junit4]   2> 2055644 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2055645 T6603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2055645 T6603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004
   [junit4]   2> 2055646 T6603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004/'
   [junit4]   2> 2055661 T6603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004/solr.xml
   [junit4]   2> 2055667 T6603 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004
   [junit4]   2> 2055668 T6603 oasc.CoreContainer.<init> New CoreContainer 1898908745
   [junit4]   2> 2055668 T6603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004/]
   [junit4]   2> 2055669 T6603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2055669 T6603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2055670 T6603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2055670 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2055670 T6603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2055671 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2055671 T6603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2055671 T6603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2055672 T6603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2055673 T6603 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2055674 T6603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2055675 T6603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2055675 T6603 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2055675 T6603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37709/solr
   [junit4]   2> 2055676 T6603 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2055676 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2055677 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2055678 T6664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76603c87 name:ZooKeeperConnection Watcher:127.0.0.1:37709 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2055678 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2055678 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2055679 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2055680 T6666 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bfddde9 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2055680 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2055683 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2056686 T6603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34877_pp
   [junit4]   2> 2056686 T6603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34877_pp
   [junit4]   2> 2056688 T6603 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2056690 T6603 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004
   [junit4]   2> 2056697 T6603 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-004/conf/
   [junit4]   2> 2056846 T6603 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2056847 T6603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2056848 T6603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2056850 T6603 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 2056850 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2056850 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2056851 T6670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1529f8a9 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2056852 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2056852 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2056852 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2056918 T6624 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 2056918 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056921 T6630 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2056921 T6671 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2056922 T6671 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2056922 T6671 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 2056923 T6671 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 2056924 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2056926 T6629 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 2056926 T6629 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 2056926 T6629 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 2056928 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2056928 T6637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2056928 T6667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2056928 T6652 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2056928 T6638 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2056929 T6672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2057025 T6671 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 2057025 T6671 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:44146_pp
   [junit4]   2> 2057026 T6671 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:36558_pp
   [junit4]   2> 2057088 T6623 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2057088 T6646 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 2057090 T6623 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2057090 T6646 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2057091 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057091 T6623 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2057092 T6646 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2057092 T6646 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2057093 T6646 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2057093 T6629 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44146/pp",
   [junit4]   2> 	  "node_name":"127.0.0.1:44146_pp",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2057093 T6646 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2057094 T6646 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2057096 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2057097 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057097 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2057098 T6623 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2057098 T6623 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2057099 T6623 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2057099 T6623 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2057099 T6623 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 2057099 T6623 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2057100 T6623 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2057100 T6623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2057101 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2057101 T6623 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2057101 T6629 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36558/pp",
   [junit4]   2> 	  "node_name":"127.0.0.1:36558_pp",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2057101 T6623 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 2057102 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057102 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2057103 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057103 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057103 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2057104 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2057138 T6623 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2057147 T6623 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2057148 T6623 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2057152 T6623 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 2057234 T6623 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2057237 T6623 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2057238 T6623 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2057244 T6623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2057245 T6623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2057247 T6623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2057248 T6623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2057248 T6623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2057248 T6623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2057249 T6623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2057249 T6623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2057249 T6623 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2057249 T6623 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2057250 T6623 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 2057250 T6623 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62867889
   [junit4]   2> 2057251 T6623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 2057251 T6623 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 2057251 T6623 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2057252 T6623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 2057252 T6623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=14.3740234375, floorSegmentMB=1.810546875, forceMergeDeletesPctAllowed=11.1342598844669, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2057252 T6623 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-35cf1b9e))),segFN=segments_1,generation=1}
   [junit4]   2> 2057253 T6623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2057255 T6623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2057255 T6623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2057256 T6623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2057256 T6623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2057256 T6623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2057256 T6623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2057257 T6623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2057257 T6623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2057257 T6623 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2057258 T6623 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2057258 T6623 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2057258 T6623 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2057258 T6623 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2057259 T6623 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2057259 T6623 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2057259 T6623 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2057260 T6623 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2057260 T6623 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2057260 T6623 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2057261 T6623 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2057262 T6623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2057263 T6623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2057263 T6623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2057264 T6623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2057265 T6623 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2057266 T6623 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2057266 T6623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=52.4306640625, floorSegmentMB=1.69140625, forceMergeDeletesPctAllowed=14.881437673440196, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6530349530890306
   [junit4]   2> 2057267 T6623 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-35cf1b9e))),segFN=segments_1,generation=1}
   [junit4]   2> 2057267 T6623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2057267 T6623 oass.SolrIndexSearcher.<init> Opening Searcher@38420797[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 2057267 T6623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2057268 T6623 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2057268 T6623 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2057268 T6623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2057269 T6623 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2057269 T6623 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2057280 T6623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2057280 T6623 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2057280 T6623 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2057282 T6623 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2057282 T6623 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2057282 T6623 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2057284 T6675 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@38420797[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2057285 T6623 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 2057285 T6623 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:https://127.0.0.1:44146/pp collection:customcollreplicadeletion shard:a
   [junit4]   2> 2057286 T6623 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 2057289 T6623 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 2057290 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057290 T6623 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2057290 T6623 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4317 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@28d4adba url=https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1 node=127.0.0.1:44146_pp C4317_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=https://127.0.0.1:44146/pp, node_name=127.0.0.1:44146_pp, state=down}
   [junit4]   2> 2057290 T6623 C4317 P44146 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2057291 T6623 C4317 P44146 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2057291 T6623 C4317 P44146 oasc.SyncStrategy.syncToMe https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 2057291 T6623 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 2057291 T6623 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 2057292 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2057293 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2057294 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057294 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057294 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2057295 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2057297 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2057298 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057298 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2057298 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057298 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057299 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2057299 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2057345 T6623 oasc.ZkController.register We are https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1/ and leader is https://127.0.0.1:44146/pp/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2057345 T6623 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=https://127.0.0.1:44146/pp
   [junit4]   2> 2057345 T6623 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2057345 T6623 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2057346 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2057348 T6623 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2057349 T6623 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=261 
   [junit4]   2> 2057350 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2057350 T6629 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44146/pp",
   [junit4]   2> 	  "node_name":"127.0.0.1:44146_pp",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2057351 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2057351 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057352 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2057352 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2057353 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2058094 T6646 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 2058094 T6646 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2058095 T6646 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2058095 T6646 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2058096 T6646 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2058096 T6646 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 2058119 T6646 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2058140 T6646 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2058141 T6646 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2058147 T6646 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 2058254 T6646 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2058258 T6646 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2058260 T6646 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2058266 T6646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2058269 T6646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2058270 T6646 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2058271 T6646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2058272 T6646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2058272 T6646 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2058273 T6646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2058273 T6646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2058273 T6646 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2058274 T6646 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2058274 T6646 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 2058274 T6646 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@62867889
   [junit4]   2> 2058275 T6646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 2058276 T6646 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index/
   [junit4]   2> 2058276 T6646 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2058277 T6646 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 2058277 T6646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=14.3740234375, floorSegmentMB=1.810546875, forceMergeDeletesPctAllowed=11.1342598844669, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2058278 T6646 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-58a9c39f))),segFN=segments_1,generation=1}
   [junit4]   2> 2058278 T6646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2058281 T6646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2058282 T6646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2058282 T6646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2058283 T6646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2058283 T6646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2058283 T6646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2058284 T6646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2058284 T6646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2058285 T6646 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2058285 T6646 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2058285 T6646 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2058286 T6646 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2058286 T6646 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2058287 T6646 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2058287 T6646 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2058287 T6646 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2058288 T6646 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2058288 T6646 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2058289 T6646 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2058289 T6646 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2058291 T6646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2058291 T6646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2058292 T6646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2058293 T6646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2058295 T6646 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2058296 T6646 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2058296 T6646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=52.4306640625, floorSegmentMB=1.69140625, forceMergeDeletesPctAllowed=14.881437673440196, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6530349530890306
   [junit4]   2> 2058297 T6646 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-58a9c39f))),segFN=segments_1,generation=1}
   [junit4]   2> 2058297 T6646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2058298 T6646 oass.SolrIndexSearcher.<init> Opening Searcher@6aa0baae[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 2058298 T6646 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2058299 T6646 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2058299 T6646 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2058299 T6646 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2058300 T6646 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2058300 T6646 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2058301 T6646 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2058301 T6646 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2058301 T6646 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2058302 T6646 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2058303 T6677 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@6aa0baae[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2058306 T6646 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 2058306 T6646 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:https://127.0.0.1:36558/pp collection:customcollreplicadeletion shard:b
   [junit4]   2> 2058307 T6646 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 2058309 T6646 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 2058310 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058310 T6646 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2058311 T6646 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4318 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@36a61d0c url=https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1 node=127.0.0.1:36558_pp C4318_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=https://127.0.0.1:36558/pp, node_name=127.0.0.1:36558_pp, state=down}
   [junit4]   2> 2058311 T6646 C4318 P36558 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2058311 T6646 C4318 P36558 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2058312 T6646 C4318 P36558 oasc.SyncStrategy.syncToMe https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 2058312 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2058312 T6646 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 2058312 T6646 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 2058313 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2058314 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058314 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058314 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2058315 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2058317 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2058318 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058318 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2058318 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058319 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058319 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2058320 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2058366 T6646 oasc.ZkController.register We are https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1/ and leader is https://127.0.0.1:36558/pp/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2058366 T6646 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=https://127.0.0.1:36558/pp
   [junit4]   2> 2058366 T6646 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2058367 T6646 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2058368 T6637 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058369 T6646 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2058370 T6629 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2058370 T6629 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36558/pp",
   [junit4]   2> 	  "node_name":"127.0.0.1:36558_pp",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2058370 T6646 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=1282 
   [junit4]   2> 2058371 T6671 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 2058371 T6629 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2058372 T6671 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=261},core=customcollreplicadeletion_a_replica1},null={responseHeader={status=0,QTime=1282},core=customcollreplicadeletion_b_replica1}}}
   [junit4]   2> 2058381 T6637 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058381 T6652 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2058382 T6652 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2058382 T6679 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 2058383 T6637 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2058382 T6680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058385 T6624 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=1467 
   [junit4]   2> 2058386 T6603 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 2058391 T6603 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: customcollreplicadeletion
   [junit4]   2> 2058392 T6603 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2058392 T6603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2058393 T6682 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78cc2059 name:ZooKeeperConnection Watcher:127.0.0.1:37709/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2058394 T6603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2058394 T6603 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2058394 T6603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2058400 T6623 oasha.CollectionsHandler.handleRemoveReplica Remove replica: replica=core_node1&action=deletereplica&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2058401 T6680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2058402 T6630 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 2058402 T6683 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : deletereplica , {
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [junit4]   2> 2058407 T6619 oasc.SolrCore.close [customcollreplicadeletion_a_replica1]  

[...truncated too long message...]

ent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2183998 T6670 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@796e9dbb rejected from java.util.concurrent.ThreadPoolExecutor@194162f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteLastCustomShardedReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=3E48EDCA82819F90 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_CR -Dtests.timezone=Africa/Dakar -Dtests.file.encoding=UTF-8
   [junit4] ERROR    131s J0 | DeleteLastCustomShardedReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:36558/pp, https://127.0.0.1:34877/pp, https://127.0.0.1:44146/pp]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3E48EDCA82819F90:BFAE63D2F5DEFFAC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 45 more
   [junit4]   2> 2184017 T6603 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-3E48EDCA82819F90-001
   [junit4]   2> 130814 T6602 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_CR, timezone=Africa/Dakar
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=92502016,total=302149632
   [junit4]   2> NOTE: All tests run in this JVM: [SuggesterWFSTTest, TestArbitraryIndexDir, TestUniqueKeyFieldResource, UnloadDistributedZkTest, AssignTest, TestFieldTypeCollectionResource, TestDocSet, DistributedIntervalFacetingTest, TestChildDocTransformer, TestImplicitCoreProperties, TestComponentsName, ReturnFieldsTest, HdfsWriteToMultipleCollectionsTest, ZkNodePropsTest, DistributedQueryComponentOptimizationTest, TestCloudManagedSchema, TestAddFieldRealTimeGet, TestLRUCache, JSONWriterTest, TestStandardQParsers, SpellPossibilityIteratorTest, DisMaxRequestHandlerTest, TestManagedResourceStorage, TestAnalyzedSuggestions, TestConfigSets, TestCloudSchemaless, TestDefaultSearchFieldResource, ResponseHeaderTest, ExternalFileFieldSortTest, RecoveryZkTest, PingRequestHandlerTest, TestClassNameShortening, CursorPagingTest, SolrIndexSplitterTest, TestTrie, MoreLikeThisHandlerTest, TestPartialUpdateDeduplication, TestExpandComponent, SharedFSAutoReplicaFailoverTest, DistributedExpandComponentTest, TestSolrIndexConfig, SuggestComponentTest, TestSolr4Spatial2, QueryEqualityTest, TestStressRecovery, NotRequiredUniqueKeyTest, TestReplicationHandler, TimeZoneUtilsTest, StatelessScriptUpdateProcessorFactoryTest, HdfsLockFactoryTest, MBeansHandlerTest, AnalyticsMergeStrategyTest, DocumentAnalysisRequestHandlerTest, TestSystemIdResolver, MergeStrategyTest, TestRandomDVFaceting, TestFuzzyAnalyzedSuggestions, TestCloudInspectUtil, SSLMigrationTest, IndexBasedSpellCheckerTest, DistributedFacetPivotLongTailTest, TestJmxIntegration, BlockDirectoryTest, DirectUpdateHandlerTest, TestRawResponseWriter, MinimalSchemaTest, CoreAdminCreateDiscoverTest, DateRangeFieldTest, BadIndexSchemaTest, TestHighlightDedupGrouping, TestDefaultSimilarityFactory, ZkCLITest, TestQuerySenderListener, TestCopyFieldCollectionResource, HighlighterMaxOffsetTest, JsonLoaderTest, SolrXmlInZkTest, TestSolrJ, UpdateParamsTest, BadCopyFieldTest, PluginInfoTest, TestFastWriter, TestFaceting, BasicFunctionalityTest, TestAnalyzeInfixSuggestions, ZkControllerTest, TestDocumentBuilder, TestSolr4Spatial, TestPivotHelperCode, SimpleFacetsTest, TestStressReorder, TestSchemaNameResource, TestReplicationHandlerBackup, ShardRoutingTest, TestRequestStatusCollectionAPI, TestFreeTextSuggestions, SyncSliceTest, TestRTGBase, StandardRequestHandlerTest, ParsingFieldUpdateProcessorsTest, EchoParamsTest, TestLeaderElectionZkExpiry, TestTolerantSearch, QueryResultKeyTest, NoCacheHeaderTest, BasicDistributedZkTest, BasicDistributedZk2Test, OverseerTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ShardRoutingCustomTest, TestDistributedSearch, AutoCommitTest, TestRecovery, TestHashPartitioner, TermVectorComponentDistributedTest, TestMultiCoreConfBootstrap, TestReload, TestStressVersions, StatsComponentTest, TestGroupingSearch, QueryElevationComponentTest, PeerSyncTest, TestLazyCores, TestBadConfig, OverseerCollectionProcessorTest, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, TestRemoteStreaming, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, FastVectorHighlighterTest, RegexBoostProcessorTest, TestCollationField, BinaryUpdateRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, AlternateDirectoryTest, BadComponentTest, MultiTermTest, TestBinaryField, DirectSolrConnectionTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestFastOutputStream, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, AsyncMigrateRouteKeyTest, ConnectionManagerTest, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed on J0 in 131.85s, 1 test, 1 error <<< FAILURES!

[...truncated 227 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1772 tests, 1 error, 45 ignored (23 assumptions)

Total time: 96 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11263 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11263/
Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:39861, http://127.0.0.1:35520, http://127.0.0.1:56282]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:39861, http://127.0.0.1:35520, http://127.0.0.1:56282]
	at __randomizedtesting.SeedInfo.seed([5B0AC388B92F331E:DAEC4D90CE705322]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node2
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 45 more


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:35428, http://127.0.0.1:35087, http://127.0.0.1:46657, http://127.0.0.1:54458, http://127.0.0.1:41697]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:35428, http://127.0.0.1:35087, http://127.0.0.1:46657, http://127.0.0.1:54458, http://127.0.0.1:41697]
	at __randomizedtesting.SeedInfo.seed([5B0AC388B92F331E:DAEC4D90CE705322]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:139)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:123)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:87)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard2/core_node4
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 46 more




Build Log:
[...truncated 11881 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/init-core-data-001
   [junit4]   2> 1878060 T6330 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1878061 T6330 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1878063 T6330 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1878064 T6330 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1878064 T6331 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1878164 T6330 oasc.ZkTestServer.run start zk server on port:48200
   [junit4]   2> 1878164 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1878166 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878168 T6337 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47ba2da5 name:ZooKeeperConnection Watcher:127.0.0.1:48200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878168 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878169 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1878169 T6330 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1878173 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1878175 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878176 T6339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b31eca9 name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878176 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878177 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1878177 T6330 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1878180 T6330 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1878183 T6330 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1878184 T6330 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1878186 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1878186 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1878189 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1878189 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1878191 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1878192 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1878194 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1878194 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1878195 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1878196 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1878197 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1878197 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1878198 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1878198 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1878199 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1878200 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1878201 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1878201 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1878202 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1878202 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1878203 T6330 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1878204 T6330 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1878479 T6330 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1878509 T6330 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39861
   [junit4]   2> 1878511 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1878511 T6330 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1878512 T6330 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002
   [junit4]   2> 1878512 T6330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/'
   [junit4]   2> 1878525 T6330 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/solr.xml
   [junit4]   2> 1878532 T6330 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002
   [junit4]   2> 1878532 T6330 oasc.CoreContainer.<init> New CoreContainer 957549831
   [junit4]   2> 1878533 T6330 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/]
   [junit4]   2> 1878533 T6330 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1878533 T6330 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1878534 T6330 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1878534 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1878534 T6330 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1878534 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1878535 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1878535 T6330 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1878535 T6330 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1878535 T6330 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1878536 T6330 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1878536 T6330 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1878536 T6330 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1878536 T6330 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48200/solr
   [junit4]   2> 1878537 T6330 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1878537 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1878538 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878539 T6350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14284b21 name:ZooKeeperConnection Watcher:127.0.0.1:48200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878539 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878539 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1878541 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878542 T6352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@598732c5 name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878542 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878544 T6330 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1878547 T6330 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1878549 T6330 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1878552 T6330 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1878555 T6330 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1878558 T6330 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1878560 T6330 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39861_
   [junit4]   2> 1878561 T6330 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39861_
   [junit4]   2> 1878564 T6330 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1878566 T6330 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1878569 T6330 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1878572 T6330 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39861_
   [junit4]   2> 1878573 T6330 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1878575 T6330 oasc.Overseer.start Overseer (id=92465814551986179-127.0.0.1:39861_-n_0000000000) starting
   [junit4]   2> 1878578 T6330 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1878587 T6330 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1878588 T6354 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1878589 T6330 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1878591 T6330 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1878592 T6330 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1878594 T6353 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1878595 T6330 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002
   [junit4]   2> 1878606 T6330 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/conf/
   [junit4]   2> 1878725 T6330 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1878726 T6330 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1878726 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1878726 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1878727 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878728 T6358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5857666b name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878728 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878728 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1878729 T6330 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1878730 T6330 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1878731 T6330 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1878731 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1878731 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878732 T6360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@790d2353 name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1878732 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1878732 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1878734 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1878736 T6353 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1878736 T6353 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1878736 T6353 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1878737 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1878738 T6361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1878738 T6362 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1878993 T6330 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1878994 T6330 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1878996 T6330 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35520
   [junit4]   2> 1878997 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1878997 T6330 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1878997 T6330 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003
   [junit4]   2> 1878997 T6330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003/'
   [junit4]   2> 1879007 T6330 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003/solr.xml
   [junit4]   2> 1879025 T6330 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003
   [junit4]   2> 1879025 T6330 oasc.CoreContainer.<init> New CoreContainer 90869429
   [junit4]   2> 1879025 T6330 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003/]
   [junit4]   2> 1879026 T6330 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1879026 T6330 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1879026 T6330 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1879026 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1879027 T6330 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1879027 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1879027 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1879027 T6330 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1879028 T6330 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1879028 T6330 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1879028 T6330 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1879028 T6330 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1879029 T6330 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1879029 T6330 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48200/solr
   [junit4]   2> 1879029 T6330 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1879029 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1879030 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1879031 T6373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b82cbab name:ZooKeeperConnection Watcher:127.0.0.1:48200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1879031 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1879031 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1879032 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1879033 T6375 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d04be42 name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1879033 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1879037 T6330 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1880040 T6330 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35520_
   [junit4]   2> 1880041 T6330 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35520_
   [junit4]   2> 1880047 T6330 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1880050 T6330 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003
   [junit4]   2> 1880064 T6330 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-003/conf/
   [junit4]   2> 1880218 T6330 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1880219 T6330 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1880219 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1880472 T6330 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1880472 T6330 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1880474 T6330 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56282
   [junit4]   2> 1880475 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1880475 T6330 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1880476 T6330 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004
   [junit4]   2> 1880476 T6330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/'
   [junit4]   2> 1880487 T6330 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/solr.xml
   [junit4]   2> 1880493 T6330 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004
   [junit4]   2> 1880494 T6330 oasc.CoreContainer.<init> New CoreContainer 554351368
   [junit4]   2> 1880494 T6330 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/]
   [junit4]   2> 1880494 T6330 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1880495 T6330 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1880495 T6330 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1880495 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1880495 T6330 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1880496 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1880496 T6330 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1880496 T6330 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1880496 T6330 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1880497 T6330 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1880497 T6330 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1880497 T6330 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1880498 T6330 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1880498 T6330 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48200/solr
   [junit4]   2> 1880498 T6330 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1880498 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1880499 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1880500 T6388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4dd01904 name:ZooKeeperConnection Watcher:127.0.0.1:48200 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1880501 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1880501 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1880503 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1880504 T6390 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7e19ec name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1880504 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1880511 T6330 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1881516 T6330 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56282_
   [junit4]   2> 1881517 T6330 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56282_
   [junit4]   2> 1881521 T6330 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1881523 T6330 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004
   [junit4]   2> 1881528 T6330 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/conf/
   [junit4]   2> 1881644 T6330 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1881645 T6330 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 1881645 T6330 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1881647 T6330 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 1881647 T6330 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1881648 T6330 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1881649 T6394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fa42a47 name:ZooKeeperConnection Watcher:127.0.0.1:48200/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1881649 T6330 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1881649 T6330 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1881650 T6330 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1881653 T6348 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 1881653 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1881654 T6354 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 1881655 T6395 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 1881656 T6395 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1881656 T6395 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 1881657 T6395 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 1881659 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1881661 T6353 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 1881661 T6353 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 1881661 T6353 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 1881662 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1881663 T6361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1881663 T6391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1881663 T6396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1881663 T6362 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1881663 T6376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1881759 T6395 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 1881759 T6395 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:39861_
   [junit4]   2> 1881760 T6395 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:56282_
   [junit4]   2> 1881761 T6346 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 1881762 T6383 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 1881763 T6383 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 1881763 T6346 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 1881764 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1881765 T6383 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 1881765 T6383 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 1881765 T6346 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 1881765 T6383 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 1881766 T6383 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 1881766 T6353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56282",
   [junit4]   2> 	  "node_name":"127.0.0.1:56282_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1881766 T6383 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1881769 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1881769 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1881770 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1881770 T6346 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 1881770 T6346 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 1881771 T6346 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1881771 T6346 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1881773 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 1881773 T6353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39861",
   [junit4]   2> 	  "node_name":"127.0.0.1:39861_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1881774 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1881774 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1881774 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1881774 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1881775 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1881775 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1882767 T6383 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 1882767 T6383 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 1882768 T6383 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1882768 T6383 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1882768 T6383 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1882769 T6383 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 1882772 T6346 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 1882772 T6346 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 1882772 T6346 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1882773 T6346 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1882773 T6346 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1882774 T6346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 1882790 T6383 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1882791 T6346 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1882813 T6383 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1882814 T6346 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1882818 T6383 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1882818 T6346 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1882825 T6383 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 1882825 T6346 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 1882969 T6383 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1882975 T6383 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1882977 T6383 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1882977 T6346 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1882981 T6346 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1882982 T6346 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1882983 T6383 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1882986 T6383 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1882988 T6383 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1882988 T6346 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1882988 T6383 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1882989 T6383 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1882989 T6383 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1882990 T6383 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1882990 T6383 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1882990 T6346 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1882990 T6383 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 1882991 T6383 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1882991 T6383 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 1882991 T6383 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e599390
   [junit4]   2> 1882992 T6346 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1882992 T6383 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 1882992 T6346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1882993 T6346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1882992 T6383 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index/
   [junit4]   2> 1882993 T6346 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1882993 T6383 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1882994 T6346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1882994 T6346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1882994 T6383 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-004/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 1882994 T6346 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 1882995 T6383 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1882995 T6346 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1882995 T6346 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 1882996 T6346 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e599390
   [junit4]   2> 1882996 T6383 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-41e228d)),segFN=segments_1,generation=1}
   [junit4]   2> 1882996 T6383 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1882996 T6346 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 1882997 T6346 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 1882997 T6346 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1882997 T6346 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-5B0AC388B92F331E-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 1882998 T6346 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=49, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1882998 T6346 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-4ba8fbc3)),segFN=segments_1,generation=1}
   [junit4]   2> 1882998 T6346 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1882999 T6383 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1882999 T6383 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1882999 T6383 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1883000 T6383 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1883000 T6383 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1883000 T6383 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1883000 T6383 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1883001 T6383 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1883001 T6383 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1883001 T6346 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1883001 T6383 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883001 T6346 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1883002 T6383 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883002 T6346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1883002 T6383 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883002 T6346 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1883003 T6383 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1883003 T6346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1883003 T6383 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1883003 T6383 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1883003 T6346 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1883004 T6383 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1883004 T6346 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1883004 T6383 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1883004 T6346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1883005 T6346 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1883004 T6383 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1883005 T6346 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883005 T6383 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1883005 T6346 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883006 T6383 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1883006 T6346 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1883007 T6346 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1883007 T6346 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1883007 T6346 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1883007 T6346 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1883008 T6383 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883008 T6346 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1883008 T6346 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1883008 T6383 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883009 T6346 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1883009 T6346 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1883009 T6383 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883010 T6346 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883010 T6383 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883011 T6346 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883011 T6346 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883012 T6383 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1883012 T6383 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1883012 T6346 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1883012 T6383 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=93.044921875, floorSegmentMB=0.47265625, forceMergeDeletesPctAllowed=28.800744841940144, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1883013 T6383 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-41e228d)),segFN=segments_1,generation=1}
   [junit4]   2> 1883013 T6383 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1883014 T6383 oass.SolrIndexSearcher.<init> Opening Searcher@23bcef76[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 1883014 T6346 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1883014 T6383 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1883014 T6346 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1883015 T6383 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1883015 T6383 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1883015 T6346 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=93.044921875, floorSegmentMB=0.47265625, forceMergeDeletesPctAllowed=28.800744841940144, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1883015 T6383 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1883016 T6383 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1883016 T6346 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-4ba8fbc3)),segFN=segments_1,generation=1}
   [junit4]   2> 1883016 T6346 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1883016 T6383 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1883017 T6346 oass.SolrIndexSearcher.<init> Opening Searcher@2779fd3a[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 1883017 T6383 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1883017 T6346 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 1883018 T6383 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1883018 T6346 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1883018 T6346 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1883018 T6383 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1883019 T6346 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1883019 T6346 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1883020 T6346 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1883037 T6383 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1883038 T6383 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1883038 T6383 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1883038 T6346 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1883039 T6346 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1883039 T6346 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1883039 T6346 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1883040 T6399 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@23bcef76[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1883041 T6400 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@2779fd3a[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1883041 T6383 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 1883042 T6383 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:http://127.0.0.1:56282 collection:customcollreplicadeletion shard:b
   [junit4]   2> 1883043 T6383 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 1883043 T6346 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 1883043 T6346 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:http://127.0.0.1:39861 collection:customcollreplicadeletion shard:a
   [junit4]   2> 1883044 T6346 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 1883046 T6383 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 1883047 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883047 T6383 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1883048 T6346 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 1883048 T6383 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1558 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@f7d5222 url=http://127.0.0.1:56282/customcollreplicadeletion_b_replica1 node=127.0.0.1:56282_ C1558_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=http://127.0.0.1:56282, node_name=127.0.0.1:56282_, state=down}
   [junit4]   2> 1883048 T6383 C1558 P56282 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56282/customcollreplicadeletion_b_replica1/
   [junit4]   2> 1883049 T6383 C1558 P56282 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1883049 T6346 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1883050 T6346 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1883049 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 1883049 T6383 C1558 P56282 oasc.SyncStrategy.syncToMe http://127.0.0.1:56282/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> ASYNC  NEW_CORE C1559 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@4c456dfe url=http://127.0.0.1:39861/customcollreplicadeletion_a_replica1 node=127.0.0.1:39861_ C1559_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=http://127.0.0.1:39861, node_name=127.0.0.1:39861_, state=down}
   [junit4]   2> 1883050 T6346 C1559 P39861 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39861/customcollreplicadeletion_a_replica1/
   [junit4]   2> 1883051 T6383 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56282/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 1883051 T6346 C1559 P39861 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1883051 T6383 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 1883051 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883051 T6346 C1559 P39861 oasc.SyncStrategy.syncToMe http://127.0.0.1:39861/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 1883052 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883052 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883053 T6346 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39861/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 1883053 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1883054 T6346 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 1883054 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1883055 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883057 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 1883058 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883058 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883058 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883059 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1883060 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1883062 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 1883063 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883063 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883063 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883064 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883064 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1883065 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1883067 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 1883068 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883068 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883068 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883069 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883069 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1883069 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1883105 T6383 oasc.ZkController.register We are http://127.0.0.1:56282/customcollreplicadeletion_b_replica1/ and leader is http://127.0.0.1:56282/customcollreplicadeletion_b_replica1/
   [junit4]   2> 1883106 T6383 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=http://127.0.0.1:56282
   [junit4]   2> 1883106 T6383 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1883106 T6383 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 1883107 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883107 T6346 oasc.ZkController.register We are http://127.0.0.1:39861/customcollreplicadeletion_a_replica1/ and leader is http://127.0.0.1:39861/customcollreplicadeletion_a_replica1/
   [junit4]   2> 1883108 T6346 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=http://127.0.0.1:39861
   [junit4]   2> 1883108 T6346 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1883108 T6346 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 1883108 T6383 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1883109 T6383 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=1347 
   [junit4]   2> 1883110 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 1883110 T6353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56282",
   [junit4]   2> 	  "node_name":"127.0.0.1:56282_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1883111 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883112 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883112 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883112 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1883113 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1883114 T6346 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1883115 T6346 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=1354 
   [junit4]   2> 1883115 T6395 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 1883116 T6395 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=1347},core=customcollreplicadeletion_b_replica1},null={responseHeader={status=0,QTime=1354},core=customcollreplicadeletion_a_replica1}}}
   [junit4]   2> 1883117 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 1883117 T6353 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 1883117 T6353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39861",
   [junit4]   2> 	  "node_name":"127.0.0.1:39861_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1883118 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883118 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883119 T6348 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=1466 
   [junit4]   2> 1883119 T6353 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 1883119 T6330 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 1883119 T6361 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1883120 T6361 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883120 T6391 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 1883120 T6391 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 1883121 T6361 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadelet

[...truncated too long message...]

   [junit4]   2> 2109791 T5921 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:46657_
   [junit4]   2> 2111292 T5924 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:346)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:296)
   [junit4]   2> 
   [junit4]   2> 2111292 T5920 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2111317 T5773 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 2116307 T5773 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2116308 T5773 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53408 53408
   [junit4]   2> 2117123 T5774 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53408 53408
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=5B0AC388B92F331E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_DO -Dtests.timezone=Asia/Calcutta -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    220s J0 | DeleteReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:35428, http://127.0.0.1:35087, http://127.0.0.1:46657, http://127.0.0.1:54458, http://127.0.0.1:41697]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([5B0AC388B92F331E:DAEC4D90CE705322]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:139)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:123)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:87)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard2/core_node4
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 46 more
   [junit4]   2> 2117145 T5773 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest-5B0AC388B92F331E-001
   [junit4]   2> 220512 T5772 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_DO, timezone=Asia/Calcutta
   [junit4]   2> NOTE: Linux 3.13.0-35-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=200662864,total=519045120
   [junit4]   2> NOTE: All tests run in this JVM: [TestConfigSets, TestCodecSupport, TestReplicationHandler, TestRTGBase, TestCloudManagedSchema, FieldMutatingUpdateProcessorTest, TestRandomDVFaceting, TestDocumentBuilder, DateRangeFieldTest, HighlighterConfigTest, DisMaxRequestHandlerTest, CoreAdminCreateDiscoverTest, TestSolr4Spatial2, TestImplicitCoreProperties, MinimalSchemaTest, SharedFSAutoReplicaFailoverTest, DocValuesMultiTest, DistributedExpandComponentTest, TestCopyFieldCollectionResource, OpenExchangeRatesOrgProviderTest, TestModifyConfFiles, ClusterStateUpdateTest, IndexSchemaTest, DistribDocExpirationUpdateProcessorTest, FastVectorHighlighterTest, TestDefaultSearchFieldResource, PeerSyncTest, ZkControllerTest, TestCloudInspectUtil, BadComponentTest, TestFreeTextSuggestions, HdfsCollectionsAPIDistributedZkTest, HdfsWriteToMultipleCollectionsTest, PingRequestHandlerTest, DirectSolrSpellCheckerTest, OverseerTest, ResponseLogComponentTest, TestLRUCache, BadCopyFieldTest, TestDFRSimilarityFactory, TestTrie, TestManagedSynonymFilterFactory, TestArbitraryIndexDir, TestLMDirichletSimilarityFactory, TestAnalyzedSuggestions, RequiredFieldsTest, TestDistribDocBasedVersion, SolrIndexSplitterTest, TestSchemaNameResource, TestRecoveryHdfs, TestWordDelimiterFilterFactory, SolrCmdDistributorTest, QueryResultKeyTest, TestStandardQParsers, TestDefaultSimilarityFactory, PrimitiveFieldTypeTest, NoCacheHeaderTest, MergeStrategyTest, BadIndexSchemaTest, TestStressVersions, TestRestManager, PathHierarchyTokenizerFactoryTest, PluginInfoTest, TestSystemIdResolver, TestLMJelinekMercerSimilarityFactory, TestSolrDeletionPolicy2, NotRequiredUniqueKeyTest, StatelessScriptUpdateProcessorFactoryTest, HighlighterMaxOffsetTest, SpellingQueryConverterTest, TestFastOutputStream, TestSort, TermsComponentTest, ZkCLITest, TestHighlightDedupGrouping, TestBinaryResponseWriter, TestFastWriter, TestHighFrequencyDictionaryFactory, TestHashPartitioner, TestMaxScoreQueryParser, SuggesterWFSTTest, TestFuzzyAnalyzedSuggestions, TestReplicationHandlerBackup, TestSolrXml, AliasIntegrationTest, TestPartialUpdateDeduplication, AddBlockUpdateTest, DistributedQueryComponentOptimizationTest, TestStressReorder, TestComponentsName, TestSolrDeletionPolicy1, BlockDirectoryTest, ShardRoutingTest, TestShortCircuitedRequests, TestSolrXmlPersistor, DistributedIntervalFacetingTest, CoreAdminHandlerTest, ZkNodePropsTest, TestAnalyzeInfixSuggestions, SpellPossibilityIteratorTest, MBeansHandlerTest, TestSolrJ, LeaderElectionTest, BasicDistributedZk2Test, SyncSliceTest, RecoveryZkTest, ZkSolrClientTest, AutoCommitTest, TestDistributedGrouping, TestReload, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, StatsComponentTest, SpellCheckComponentTest, QueryElevationComponentTest, TestFiltering, HighlighterTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, TestUpdate, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, TestRemoteStreaming, TestOmitPositions, XmlUpdateRequestHandlerTest, DocumentBuilderTest, MoreLikeThisHandlerTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, TestJmxIntegration, JSONWriterTest, SearchHandlerTest, SOLR749Test, UpdateParamsTest, TestQuerySenderNoQuery, TestStressRecovery, TestSearchPerf, DirectSolrConnectionTest, TestLuceneMatchVersion, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestTolerantSearch, AssignTest, AsyncMigrateRouteKeyTest, DeleteInactiveReplicaTest, DeleteReplicaTest]
   [junit4] Completed on J0 in 220.79s, 1 test, 1 error <<< FAILURES!

[...truncated 267 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 429 suites, 1772 tests, 2 errors, 45 ignored (23 assumptions)

Total time: 89 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11262 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11262/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 60051 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/docs/solr-analytics/overview-summary.html
     [exec]   missing: org.apache.solr.handler.component
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:78: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:548: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:564: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2471: exec returned: 1

Total time: 108 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any