You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/13 03:39:52 UTC

[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_40-ea-b04) - Build # 4308 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4308/
Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]
	at __randomizedtesting.SeedInfo.seed([EE65860654A833E4:6F83081E23F753D8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	... 46 more




Build Log:
[...truncated 11187 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\init-core-data-001
   [junit4]   2> 2489168 T5111 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2489168 T5111 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2489180 T5111 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2489184 T5111 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2489186 T5112 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2489281 T5111 oasc.ZkTestServer.run start zk server on port:56184
   [junit4]   2> 2489283 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2489287 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2489297 T5118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e04284 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2489298 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2489299 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2489300 T5111 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2489314 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2489320 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2489323 T5120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a1b7aa name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2489323 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2489323 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2489324 T5111 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2489328 T5111 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2489334 T5111 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2489339 T5111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2489346 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2489347 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2489357 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2489357 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2489366 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2489366 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2489374 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2489374 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2489383 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2489383 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2489395 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2489395 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2489402 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2489403 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2489412 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2489413 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2489418 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2489419 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2489426 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2489426 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2489434 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2489434 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2491840 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2491871 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56191
   [junit4]   2> 2491871 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\'
   [junit4]   2> 2491913 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\solr.xml
   [junit4]   2> 2491932 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2491934 T5111 oasc.CoreContainer.<init> New CoreContainer 27308953
   [junit4]   2> 2491934 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\]
   [junit4]   2> 2491935 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2491936 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2491936 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2491937 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2491937 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2491938 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2491938 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2491939 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2491940 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2491940 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2491941 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2491941 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2491942 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2491942 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2491943 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2491944 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2491946 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2491947 T5131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27bd5f name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2491948 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2491949 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2491954 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2491957 T5133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d6ac57 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2491958 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2491961 T5111 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2491969 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2491977 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2491983 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2491990 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2491996 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2492000 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56191_
   [junit4]   2> 2492001 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56191_
   [junit4]   2> 2492007 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2492011 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2492016 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2492019 T5111 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:56191_
   [junit4]   2> 2492020 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2492023 T5111 oasc.Overseer.start Overseer (id=92443007120441347-127.0.0.1:56191_-n_0000000000) starting
   [junit4]   2> 2492029 T5111 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2492050 T5111 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2492052 T5135 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2492054 T5111 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2492059 T5111 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2492063 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2492072 T5134 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2492074 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2493317 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\conf\
   [junit4]   2> 2493540 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2493544 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2493544 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2493545 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2493548 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2493552 T5139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12b21ea name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2493552 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2493552 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2493553 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2493561 T5111 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2493561 T5111 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2493561 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2493565 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2493567 T5141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d7672d name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2493568 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2493568 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2493572 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 2493589 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2493596 T5142 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2493605 T5143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2495646 T5111 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2495648 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2495653 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56206
   [junit4]   2> 2495655 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\'
   [junit4]   2> 2495694 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\solr.xml
   [junit4]   2> 2495714 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2495715 T5111 oasc.CoreContainer.<init> New CoreContainer 19705578
   [junit4]   2> 2495715 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\]
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2495719 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2495720 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2495720 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2495720 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2495720 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2495721 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2495722 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2495722 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2495724 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2495727 T5154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@463656 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2495728 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2495728 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2495733 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2495737 T5156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d49c1a name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2495737 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2495753 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2496629 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56206_
   [junit4]   2> 2496631 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56206_
   [junit4]   2> 2496644 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2496649 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2497874 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\conf\
   [junit4]   2> 2498069 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2498074 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2498074 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2500365 T5111 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2500366 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2500372 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56215
   [junit4]   2> 2500374 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2500374 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2500374 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2500375 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\'
   [junit4]   2> 2500423 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\solr.xml
   [junit4]   2> 2500440 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2500441 T5111 oasc.CoreContainer.<init> New CoreContainer 10850303
   [junit4]   2> 2500441 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\]
   [junit4]   2> 2500442 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2500442 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2500443 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2500443 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2500445 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2500446 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2500446 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2500447 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2500447 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2500448 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2500448 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2500450 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2500453 T5169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@804a27 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2500454 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2500454 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2500461 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2500463 T5171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8d66 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2500464 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2500478 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2501354 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56215_
   [junit4]   2> 2501356 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56215_
   [junit4]   2> 2501370 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2501382 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2502839 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\conf\
   [junit4]   2> 2503112 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2503115 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2503115 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2503122 T5111 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 2503122 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2503124 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2503128 T5175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9622b name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2503128 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2503129 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2503132 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2503147 T5152 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 2503151 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503154 T5135 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2503157 T5176 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2503162 T5176 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2503163 T5176 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 2503164 T5176 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 2503178 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 2503204 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503210 T5143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503210 T5142 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503278 T5176 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 2503278 T5176 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:56191_
   [junit4]   2> 2503281 T5176 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:56206_
   [junit4]   2> 2503291 T5128 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2503292 T5148 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 2503312 T5128 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2503314 T5148 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2503324 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503327 T5128 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2503327 T5128 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2503328 T5148 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2503332 T5128 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2503333 T5148 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2503336 T5148 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2503338 T5128 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503340 T5128 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2503345 T5148 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503346 T5148 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2503356 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503357 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56191",
   [junit4]   2> 	  "node_name":"127.0.0.1:56191_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2503363 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503365 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503366 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503368 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503374 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503385 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503386 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56206",
   [junit4]   2> 	  "node_name":"127.0.0.1:56206_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2503392 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503394 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503398 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503398 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503401 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2503406 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2504341 T5128 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 2504342 T5128 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2504344 T5128 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2504346 T5148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 2504346 T5128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2504346 T5148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2504350 T5148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2504351 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2504351 T5128 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2504352 T5128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\'
   [junit4]   2> 2504357 T5148 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2504357 T5148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\'
   [junit4]   2> 2504519 T5128 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2504539 T5148 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2504565 T5148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2504574 T5128 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2504576 T5148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2504585 T5128 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2504646 T5128 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 2504677 T5148 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 2504958 T5128 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2504968 T5148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2504987 T5148 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2504992 T5148 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2504996 T5128 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2505005 T5128 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2505092 T5128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505095 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505107 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505107 T5128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505115 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505119 T5128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505122 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505125 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505126 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505128 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505129 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505131 T5128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505132 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505135 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505135 T5148 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2505136 T5148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2505137 T5148 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\
   [junit4]   2> 2505138 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a10908
   [junit4]   2> 2505138 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505140 T5148 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\
   [junit4]   2> 2505141 T5148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index/
   [junit4]   2> 2505142 T5148 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 2505140 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505144 T5128 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2505144 T5128 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2505143 T5148 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index
   [junit4]   2> 2505148 T5128 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\
   [junit4]   2> 2505150 T5128 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a10908
   [junit4]   2> 2505149 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=75.3212890625, floorSegmentMB=2.1005859375, forceMergeDeletesPctAllowed=3.1211462431648207, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44228484152719927
   [junit4]   2> 2505152 T5128 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\
   [junit4]   2> 2505152 T5128 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index/
   [junit4]   2> 2505152 T5128 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 2505153 T5128 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index
   [junit4]   2> 2505155 T5128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=75.3212890625, floorSegmentMB=2.1005859375, forceMergeDeletesPctAllowed=3.1211462431648207, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44228484152719927
   [junit4]   2> 2505157 T5148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5caee6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bd4bc0),segFN=segments_1,generation=1}
   [junit4]   2> 2505157 T5128 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dbbc23 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@159860e),segFN=segments_1,generation=1}
   [junit4]   2> 2505159 T5128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505158 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505186 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2505188 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505189 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505190 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505190 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505191 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2505191 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2505193 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505193 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505194 T5148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2505193 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505196 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505197 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505198 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505199 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2505196 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505201 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505203 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505203 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505204 T5128 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2505207 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505210 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505212 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505215 T5128 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2505216 T5128 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2505216 T5128 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2505218 T5128 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2505220 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2505205 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505222 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2505224 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2505225 T5148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2505226 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2505230 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2505231 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2505233 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2505235 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2505238 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505240 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505244 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505244 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505249 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505250 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505254 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505254 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505262 T5128 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2505262 T5148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2505266 T5148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2505265 T5128 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2505268 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1604624151325485]
   [junit4]   2> 2505269 T5128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1604624151325485]
   [junit4]   2> 2505271 T5148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5caee6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bd4bc0),segFN=segments_1,generation=1}
   [junit4]   2> 2505271 T5128 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dbbc23 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@159860e),segFN=segments_1,generation=1}
   [junit4]   2> 2505274 T5128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505275 T5128 oass.SolrIndexSearcher.<init> Opening Searcher@a56adc[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 2505276 T5128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2505273 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505277 T5148 oass.SolrIndexSearcher.<init> Opening Searcher@19fb027[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 2505279 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2505280 T5128 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2505281 T5128 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2505283 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2505285 T5128 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2505286 T5128 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505287 T5148 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2505288 T5148 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2505290 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505291 T5128 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505291 T5148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2505294 T5148 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2505295 T5128 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2505297 T5148 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505302 T5148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505302 T5148 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505307 T5148 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2505318 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505319 T5128 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505320 T5128 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2505319 T5148 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505323 T5148 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2505343 T5148 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 2505343 T5148 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:http://127.0.0.1:56206 collection:customcollreplicadeletion shard:b
   [junit4]   2> 2505346 T5181 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@19fb027[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2505354 T5180 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@a56adc[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2505361 T5148 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 2505363 T5128 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 2505364 T5128 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:http://127.0.0.1:56191 collection:customcollreplicadeletion shard:a
   [junit4]   2> 2505370 T5128 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 2505404 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 2505409 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 2505412 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505415 T5148 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2505416 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3139 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@61f90e url=http://127.0.0.1:56206/customcollreplicadeletion_b_replica1 node=127.0.0.1:56206_ C3139_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=http://127.0.0.1:56206, node_name=127.0.0.1:56206_, state=down}
   [junit4]   2> 2505418 T5148 C3139 P56206 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2505419 T5148 C3139 P56206 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2505422 T5148 C3139 P56206 oasc.SyncStrategy.syncToMe http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 2505423 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 2505424 T5148 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 2505424 T5128 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2505427 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3140 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@1ebd0d9 url=http://127.0.0.1:56191/customcollreplicadeletion_a_replica1 node=127.0.0.1:56191_ C3140_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=http://127.0.0.1:56191, node_name=127.0.0.1:56191_, state=down}
   [junit4]   2> 2505428 T5128 C3140 P56191 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2505429 T5128 C3140 P56191 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2505430 T5128 C3140 P56191 oasc.SyncStrategy.syncToMe http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 2505431 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 2505432 T5128 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 2505443 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2505458 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505464 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505466 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505468 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505475 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505490 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505497 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505500 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505506 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505507 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505512 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505524 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505535 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505544 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505546 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505548 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505550 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505552 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505558 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505563 T5148 oasc.ZkController.register We are http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ and leader is http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2505564 T5148 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=http://127.0.0.1:56206
   [junit4]   2> 2505564 T5148 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2505564 T5148 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2505571 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505577 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505586 T5148 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505588 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505593 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505593 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505593 T5148 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=2301 
   [junit4]   2> 2505597 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505605 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505617 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505618 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56206",
   [junit4]   2> 	  "node_name":"127.0.0.1:56206_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2505623 T5128 oasc.ZkController.register We are http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ and leader is http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2505624 T5128 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=http://127.0.0.1:56191
   [junit4]   2> 2505624 T5128 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2505627 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505628 T5128 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2505628 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505633 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505634 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505636 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505652 T5128 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505657 T5128 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=2366 
   [junit4]   2> 2505659 T5176 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 2505665 T5176 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=2301},core=customcollreplicadeletion_b_replica1},null={responseHeader={status=0,QTime=2366},core=customcollreplicadeletion_a_replica1}}}
   [junit4]   2> 2505666 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505669 T5157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 2505675 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505678 T5152 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=2531 
   [junit4]   2> 2505683 T5111 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 2505690 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505691 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56191",
   [junit4]   2> 	  "node_name":"127.0.0.1:56191_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2505698 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505699 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505701 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505701 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505703 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2505708 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2506702 T5111 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: customcollreplicadeletion
   [junit4]   2> 2506703 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2506706 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2506710 T5185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c27170 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2506710 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2506711 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2506715 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2506735 T5150 oasha.CollectionsHandler.handleRemoveReplica Remove replica: replica=core_node1&action=deletereplica&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2506738 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2506743 T5135 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [jun

[...truncated too long message...]

bstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2631993 T5175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9622b name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 2631993 T5175 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 2632013 T5175 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@182a1ac rejected from java.util.concurrent.ThreadPoolExecutor@1ad5ee3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteLastCustomShardedReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=EE65860654A833E4 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=Atlantic/Reykjavik -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    143s | DeleteLastCustomShardedReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EE65860654A833E4:6F83081E23F753D8]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
   [junit4]    > 	... 46 more
   [junit4]   2> 2632021 T5111 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001
   [junit4]   2> 142861 T5110 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=is, timezone=Atlantic/Reykjavik
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=2,threads=1,free=124461504,total=345505792
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, BadComponentTest, EnumFieldTest, DOMUtilTest, SharedFSAutoReplicaFailoverUtilsTest, RangeFacetTest, DistributedFacetPivotLongTailTest, ReturnFieldsTest, TestQueryTypes, TestManagedSchemaFieldTypeResource, TestCloudSchemaless, ShowFileRequestHandlerTest, OverseerRolesTest, StatelessScriptUpdateProcessorFactoryTest, DistributedIntervalFacetingTest, TestReload, TestDocBasedVersionConstraints, TestQueryUtils, TestSolrDeletionPolicy2, RAMDirectoryFactoryTest, BlockCacheTest, HighlighterTest, TestDFRSimilarityFactory, AlternateDirectoryTest, TestFaceting, ReplicationFactorTest, TestCursorMarkWithoutUniqueKey, PathHierarchyTokenizerFactoryTest, TestArbitraryIndexDir, TestManagedResource, TestHighlightDedupGrouping, TestSolr4Spatial, PluginInfoTest, TestFieldTypeResource, UpdateParamsTest, TestOverriddenPrefixQueryForCustomFieldType, SpellingQueryConverterTest, TestIBSimilarityFactory, TestBadConfig, CoreAdminCreateDiscoverTest, BadIndexSchemaTest, BasicFunctionalityTest, TermVectorComponentDistributedTest, SpatialFilterTest, HdfsRecoveryZkTest, TermVectorComponentTest, RequestHandlersTest, TestCloudInspectUtil, TestValueSourceCache, IndexSchemaTest, TestImplicitCoreProperties, TestTolerantSearch, TestRandomMergePolicy, TestCustomSort, TestExceedMaxTermLength, DeleteShardTest, QueryFacetTest, TestLRUCache, SoftAutoCommitTest, SuggesterFSTTest, ChaosMonkeySafeLeaderTest, TestRTGBase, NumericFieldsTest, PrimUtilsTest, LoggingHandlerTest, TestRestManager, QueryResultKeyTest, TestManagedSynonymFilterFactory, TestFreeTextSuggestions, TestJmxIntegration, TestClassNameShortening, TestSuggestSpellingConverter, TestCoreDiscovery, RankQueryTest, SolrIndexSplitterTest, CollectionsAPIDistributedZkTest, TestExtendedDismaxParser, DebugComponentTest, SimpleFacetsTest, HardAutoCommitTest, TestOrdValues, TestCloudPivotFacet, TestNonNRTOpen, AbstractAnalyticsStatsTest, SuggesterTest, TestSolrIndexConfig, MinimalSchemaTest, CursorMarkTest, TestStressLucene, UUIDFieldTest, TestConfigSets, CurrencyFieldXmlFileTest, TestShortCircuitedRequests, HighlighterConfigTest, SpellCheckCollatorTest, TestNRTOpen, TestCollationField, TestShardHandlerFactory, DistribDocExpirationUpdateProcessorTest, DeleteInactiveReplicaTest, UniqFieldsUpdateProcessorFactoryTest, OverriddenZkACLAndCredentialsProvidersTest, TestStressUserVersions, BlockDirectoryTest, OutputWriterTest, TestFieldTypeCollectionResource, DistributedFacetPivotSmallTest, TestCloudManagedSchemaConcurrent, DocExpirationUpdateProcessorFactoryTest, TestReversedWildcardFilterFactory, CopyFieldTest, TimeZoneUtilsTest, MigrateRouteKeyTest, TestJoin, SuggesterWFSTTest, PingRequestHandlerTest, TestDefaultSimilarityFactory, CursorPagingTest, TestRawResponseWriter, ExternalCollectionsTest, PrimitiveFieldTypeTest, URLClassifyProcessorTest, IndexBasedSpellCheckerTest, TestAddFieldRealTimeGet, PolyFieldTest, TestBlendedInfixSuggestions, TestReplicationHandlerBackup, SolrCoreCheckLockOnStartupTest, ResponseLogComponentTest, TestManagedSchemaDynamicFieldResource, ScriptEngineTest, FunctionTest, TestSchemaVersionResource, BadCopyFieldTest, TestPseudoReturnFields, FullSolrCloudDistribCmdsTest, ModifyConfFileTest, DateRangeFieldTest, TestJmxMonitoredMap, TestMaxScoreQueryParser, TestRangeQuery, TestExpandComponent, FileBasedSpellCheckerTest, ChangedSchemaMergeTest, TestChildDocTransformer, TestDynamicFieldResource, DisMaxRequestHandlerTest, TestSimpleQParserPlugin, TestDocSet, SharedFSAutoReplicaFailoverTest, TestManagedResourceStorage, TestReplicationHandler, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed in 143.76s, 1 test, 1 error <<< FAILURES!

[...truncated 887 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 error, 54 ignored (32 assumptions)

Total time: 155 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_67) - Build # 4313 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4313/
Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:55043/rpswv/k, http://127.0.0.1:55019/rpswv/k, http://127.0.0.1:55034/rpswv/k]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55043/rpswv/k, http://127.0.0.1:55019/rpswv/k, http://127.0.0.1:55034/rpswv/k]
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 45 more


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:64255, http://127.0.0.1:64273, http://127.0.0.1:64282, http://127.0.0.1:64240, http://127.0.0.1:64264]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:64255, http://127.0.0.1:64273, http://127.0.0.1:64282, http://127.0.0.1:64240, http://127.0.0.1:64264]
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:139)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:123)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:87)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard2/core_node4
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 47 more




Build Log:
[...truncated 10877 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\init-core-data-001
   [junit4]   2> 1226895 T2716 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1226895 T2716 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1226899 T2716 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1226900 T2716 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1226901 T2717 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1226981 T2716 oasc.ZkTestServer.run start zk server on port:64233
   [junit4]   2> 1226981 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1226985 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1226991 T2723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83fdac name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1226992 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1226992 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1226992 T2716 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1226999 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1227001 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1227003 T2725 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69416a name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1227004 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1227004 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1227004 T2716 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1227011 T2716 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1227015 T2716 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1227021 T2716 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1227026 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1227026 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1227032 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1227032 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1227038 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1227038 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1227043 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1227043 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1227048 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1227048 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1227054 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1227054 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1227059 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1227059 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1227064 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1227064 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1227068 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1227068 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1227074 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1227074 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1227080 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1227081 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1228507 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1228515 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64240
   [junit4]   2> 1228515 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\'
   [junit4]   2> 1228545 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\solr.xml
   [junit4]   2> 1228585 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1228585 T2716 oasc.CoreContainer.<init> New CoreContainer 33362955
   [junit4]   2> 1228585 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\]
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1228590 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1228590 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1228590 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1228590 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1228590 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1228590 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1228590 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1228593 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1228594 T2736 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f1383 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1228595 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1228595 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1228599 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1228600 T2738 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fdabab name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1228600 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1228602 T2716 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1228607 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1228611 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1228614 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1228619 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1228624 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1228627 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64240_
   [junit4]   2> 1228628 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64240_
   [junit4]   2> 1228633 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1228636 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1228639 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1228642 T2716 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64240_
   [junit4]   2> 1228642 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1228645 T2716 oasc.Overseer.start Overseer (id=92456901443452931-127.0.0.1:64240_-n_0000000000) starting
   [junit4]   2> 1228649 T2716 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1228663 T2716 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1228665 T2740 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1228666 T2716 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1228669 T2716 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1228672 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1228679 T2739 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1228681 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1229263 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\conf\
   [junit4]   2> 1229364 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1229366 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1229366 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1229368 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1229370 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1229372 T2744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d1035 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1229372 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1229373 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1229375 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1229382 T2716 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1229382 T2716 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1229382 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1229384 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1229385 T2746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8635b1 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1229386 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1229386 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1229391 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1229402 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1229406 T2747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1229412 T2748 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1230791 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1230791 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1230795 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64255
   [junit4]   2> 1230797 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1230797 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1230797 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1230798 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\'
   [junit4]   2> 1230826 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\solr.xml
   [junit4]   2> 1230881 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1230881 T2716 oasc.CoreContainer.<init> New CoreContainer 22393952
   [junit4]   2> 1230882 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\]
   [junit4]   2> 1230883 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1230883 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1230885 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1230885 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1230886 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1230886 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1230886 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1230887 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1230887 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1230887 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1230887 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1230890 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1230892 T2759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b520da name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1230893 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1230894 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1230900 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1230902 T2761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bda455 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1230902 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1230911 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1231713 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64255_
   [junit4]   2> 1231714 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64255_
   [junit4]   2> 1231722 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1231728 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1232714 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\conf\
   [junit4]   2> 1232897 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1232899 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1232899 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1234253 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1234254 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1234259 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64264
   [junit4]   2> 1234259 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\'
   [junit4]   2> 1234291 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\solr.xml
   [junit4]   2> 1234331 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1234331 T2716 oasc.CoreContainer.<init> New CoreContainer 13801278
   [junit4]   2> 1234332 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\]
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1234335 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1234335 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1234335 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1234335 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1234335 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1234337 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1234337 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1234337 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1234339 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1234340 T2774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c78390 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1234341 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1234342 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1234344 T2718 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148790fdf920008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1234345 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1234346 T2776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130e843 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1234347 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1234355 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1235227 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64264_
   [junit4]   2> 1235230 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64264_
   [junit4]   2> 1235242 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1235249 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1235904 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\conf\
   [junit4]   2> 1236010 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1236012 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1236012 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1237639 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1237640 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1237649 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64273
   [junit4]   2> 1237650 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\'
   [junit4]   2> 1237685 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\solr.xml
   [junit4]   2> 1237732 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1237733 T2716 oasc.CoreContainer.<init> New CoreContainer 12121304
   [junit4]   2> 1237733 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\]
   [junit4]   2> 1237734 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1237734 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1237736 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1237736 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1237737 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1237738 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1237738 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1237738 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1237738 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1237739 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1237739 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1237739 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1237740 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1237744 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1237749 T2789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2b948 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1237750 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1237750 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1237756 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1237758 T2791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f2539e name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1237758 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1237773 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1238656 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64273_
   [junit4]   2> 1238658 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64273_
   [junit4]   2> 1238672 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1238679 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1239291 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\conf\
   [junit4]   2> 1239414 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1239415 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1239415 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1240998 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1240998 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241004 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64282
   [junit4]   2> 1241005 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241005 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241005 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1241006 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\'
   [junit4]   2> 1241035 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\solr.xml
   [junit4]   2> 1241108 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1241108 T2716 oasc.CoreContainer.<init> New CoreContainer 4794936
   [junit4]   2> 1241108 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\]
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241114 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241114 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241114 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1241116 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241116 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241116 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241118 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1241118 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241118 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1241121 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241123 T2804 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@172eaaf name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241124 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241125 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1241128 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241130 T2806 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1582550 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241131 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241143 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1242026 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64282_
   [junit4]   2> 1242028 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64282_
   [junit4]   2> 1242042 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1242051 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1243114 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\conf\
   [junit4]   2> 1243304 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1243307 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1243307 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1243312 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1243313 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243314 T2810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1564a75 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243315 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243315 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1243318 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1243325 T2716 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
   [junit4]   2> 1243325 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1243327 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243328 T2812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b88624 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243328 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243329 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1243331 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1243344 T2733 oasha.CollectionsHandler.handleCreateAction Creating Collection : stateFormat=2&action=CREATE&version=2&maxShardsPerNode=2&wt=javabin&name=delLiveColl&replicationFactor=2&numShards=2
   [junit4]   2> 1243347 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243349 T2740 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1243351 T2813 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1243355 T2813 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1243355 T2813 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl 
   [junit4]   2> 1243356 T2813 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1243364 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243372 T2739 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1243373 T2739 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1243373 T2739 oasc.Overseer$ClusterStateUpdater.createCollection state version delLiveColl 2
   [junit4]   2> 1243377 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243380 T2748 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243381 T2747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243382 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243384 T2815 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243384 T2814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243461 T2813 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1243461 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:64282_
   [junit4]   2> 1243463 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:64273_
   [junit4]   2> 1243465 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:64240_
   [junit4]   2> 1243468 T2800 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard1&collection.configName=conf1&wt=javabin&name=delLiveColl_shard1_replica1&collection=delLiveColl&numShards=2
   [junit4]   2> 1243468 T2784 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard1&collection.configName=conf1&wt=javabin&name=delLiveColl_shard1_replica2&collection=delLiveColl&numShards=2
   [junit4]   2> 1243471 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:64264_
   [junit4]   2> 1243475 T2768 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard2&collection.configName=conf1&wt=javabin&name=delLiveColl_shard2_replica2&collection=delLiveColl&numShards=2
   [junit4]   2> 1243481 T2734 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard2&collection.configName=conf1&wt=javabin&name=delLiveColl_shard2_replica1&collection=delLiveColl&numShards=2
   [junit4]   2> 1243482 T2784 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
   [junit4]   2> 1243483 T2800 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
   [junit4]   2> 1243485 T2768 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
   [junit4]   2> 1243487 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243488 T2734 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
   [junit4]   2> 1243494 T2784 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243494 T2784 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243494 T2800 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243494 T2800 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243496 T2784 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243497 T2768 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243497 T2768 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243498 T2800 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243500 T2768 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243504 T2784 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243504 T2784 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243505 T2734 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243507 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64273",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64273_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243507 T2800 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243508 T2800 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243509 T2768 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243509 T2768 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243526 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243527 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243529 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243529 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243530 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243535 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243535 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243537 T2734 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243538 T2734 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243539 T2734 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243539 T2734 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243550 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243551 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64282",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64282_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243554 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243555 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243556 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243556 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243558 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243558 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243559 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243559 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243560 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243563 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243571 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243572 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64264",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64264_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243575 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243576 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243578 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243578 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243579 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243580 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243579 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243580 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243581 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243584 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243591 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243591 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64240",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64240_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243595 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243595 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243597 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243597 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243598 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243599 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243599 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243599 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243600 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243603 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1244465 T2784 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1244465 T2784 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244467 T2784 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244467 T2784 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244469 T2800 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1244469 T2800 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244470 T2784 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244470 T2784 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\'
   [junit4]   2> 1244470 T2768 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1244471 T2768 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244471 T2800 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244472 T2800 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244474 T2768 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244474 T2768 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244476 T2800 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244476 T2800 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\delLiveColl_shard1_replica1\'
   [junit4]   2> 1244479 T2768 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244479 T2768 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\delLiveColl_shard2_replica2\'
   [junit4]   2> 1244503 T2734 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1244503 T2734 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244527 T2734 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244527 T2734 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244534 T2734 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244534 T2734 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\delLiveColl_shard2_replica1\'
   [junit4]   2> 1244596 T2784 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244597 T2800 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244642 T2768 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244669 T2800 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244701 T2768 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244703 T2800 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244711 T2784 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244715 T2800 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1244719 T2768 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244729 T2768 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1244745 T2734 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244804 T2734 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244805 T2784 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244815 T2784 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1245175 T2734 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1245187 T2734 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1245934 T2784 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1245939 T2800 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1245949 T2784 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1245954 T2800 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1245958 T2784 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1245964 T2800 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1246072 T2784 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1246083 T2784 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1246105 T2768 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1246112 T2768 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1246115 T2768 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1246124 T2784 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1246136 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1246136 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1246137 T2784 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using configuration from collection delLiveColl
   [junit4]   2> 1246137 T2784 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1246138 T2784 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\
   [junit4]   2> 1246138 T2784 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10163f6
   [junit4]   2> 1246140 T2784 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\
   [junit4]   2> 1246141 T2784 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index/
   [junit4]   2> 1246141 T2784 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index' doesn't exist. Creating new index...
   [junit4]   2> 1246142 T2784 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index
   [junit4]   2> 1246143 T2784 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=14.45703125, floorSegmentMB=1.69140625, forceMergeDeletesPctAllowed=8.899551040512542, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3486525566445853
   [junit4]   2> 1246143 T2784 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@14d9ffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7eaddf),segFN=segments_1,generation=1}
   [junit4]   2> 1246143 T2784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1246149 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1246149 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1246151 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1246152 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1246152 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1246152 T2784 oasc.SolrCore.loadUpdateProcessorChains no upd

[...truncated too long message...]

unit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:55)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:315)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 4338042 T8472 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 4338062 T7724 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/be,null}
   [junit4]   2> 4344312 T7724 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 4344313 T7724 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55825 55825
   [junit4]   2> 4347137 T7725 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55825 55825
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=40015FF1CB47EE66 -Dtests.slow=true -Dtests.locale=zh -Dtests.timezone=Africa/Porto-Novo -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    803s | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4347162 T7724 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-40015FF1CB47EE66-001
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=108), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=108)), sim=DefaultSimilarity, locale=zh, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=117008136,total=264413184
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, TestBinaryResponseWriter, QueryElevationComponentTest, TestManagedStopFilterFactory, LukeRequestHandlerTest, NotRequiredUniqueKeyTest, BJQParserTest, SuggesterFSTTest, AssignTest, HdfsChaosMonkeySafeLeaderTest, LoggingHandlerTest, TestIndexSearcher, TestComponentsName, TestBM25SimilarityFactory, TestSolrXml, TestRangeQuery, HdfsCollectionsAPIDistributedZkTest, TestSolrXmlPersistor, SolrCoreCheckLockOnStartupTest, TestCodecSupport, SchemaVersionSpecificBehaviorTest, RequestHandlersTest, BadIndexSchemaTest, SimpleFacetsTest, QueryParsingTest, TestReplicationHandler, TestAnalyzeInfixSuggestions, TestNRTOpen, WordBreakSolrSpellCheckerTest, TestPivotHelperCode, TestRecoveryHdfs, TestLazyCores, DisMaxRequestHandlerTest, TestCSVLoader, SignatureUpdateProcessorFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestDocSet, SOLR749Test, StatelessScriptUpdateProcessorFactoryTest, TestCloudPivotFacet, TestJoin, DistributedIntervalFacetingTest, TestHighFrequencyDictionaryFactory, CurrencyFieldXmlFileTest, TestSurroundQueryParser, MergeStrategyTest, CoreAdminCreateDiscoverTest, FastVectorHighlighterTest, TestRandomDVFaceting, QueryEqualityTest, TestUtils, TestFunctionQuery, XsltUpdateRequestHandlerTest, TestFoldingMultitermQuery, TestFastOutputStream, TestFiltering, HttpPartitionTest, ZkControllerTest, TestManagedSchemaDynamicFieldResource, ReplicationFactorTest, RollingRestartTest, TestGroupingSearch, CollectionsAPIAsyncDistributedZkTest, TestCollationField, TestShortCircuitedRequests, AnalyticsMergeStrategyTest, TestWriterPerf, TestFastWriter, CursorMarkTest, DebugComponentTest, TestCopyFieldCollectionResource, RangeFacetTest, TestMissingGroups, TimeZoneUtilsTest, DistributedSpellCheckComponentTest, SystemInfoHandlerTest, TestFieldCollectionResource, TestUpdate, AliasIntegrationTest, AbstractAnalyticsFacetTest, ConnectionManagerTest, TestConfig, TestQueryTypes, TestMergePolicyConfig, UUIDFieldTest, DeleteReplicaTest, HdfsBasicDistributedZk2Test, TestLRUCache, RegexBoostProcessorTest, TestDynamicFieldCollectionResource, TestInitQParser, TestStressReorder, ResponseHeaderTest, TestDocBasedVersionConstraints, TestJmxMonitoredMap, TestMaxScoreQueryParser, QueryResultKeyTest, BasicFunctionalityTest, CacheHeaderTest, SSLMigrationTest, HdfsWriteToMultipleCollectionsTest, TriLevelCompositeIdRoutingTest, MigrateRouteKeyTest, TestQueryUtils, MultiTermTest, TestDistributedSearch, TestCoreDiscovery, HdfsUnloadDistributedZkTest, ParsingFieldUpdateProcessorsTest, InfoHandlerTest, TestTolerantSearch, AnalysisErrorHandlingTest, TestBadConfig, IndexSchemaTest, PluginInfoTest, DocExpirationUpdateProcessorFactoryTest, TestReRankQParserPlugin, RankQueryTest, TestClassNameShortening, DateFieldTest, CustomCollectionTest, RecoveryZkTest, TestFaceting, TestPostingsSolrHighlighter, TestAddFieldRealTimeGet, PathHierarchyTokenizerFactoryTest, CoreMergeIndexesAdminHandlerTest, TestExceedMaxTermLength, PolyFieldTest, HighlighterTest, TestArbitraryIndexDir, DirectSolrConnectionTest, AbstractAnalyticsStatsTest, DocValuesMissingTest, SliceStateTest, SharedFSAutoReplicaFailoverUtilsTest, TestSimpleQParserPlugin, TestCloudSchemaless, TestCloudInspectUtil, TestFieldSortValues, TestRandomFaceting, TestRandomMergePolicy, SolrCoreTest, ChaosMonkeySafeLeaderTest, TestNoOpRegenerator, UnloadDistributedZkTest, BlockCacheTest, SynonymTokenizerTest, SolrPluginUtilsTest, HdfsDirectoryTest, SyncSliceTest, SearchHandlerTest, TestHighlightDedupGrouping, ResourceLoaderTest, TestManagedSynonymFilterFactory, TestDefaultSimilarityFactory, TestAnalyzedSuggestions, TestQuerySenderListener, TestReplicationHandlerBackup, BasicDistributedZkTest, TestFreeTextSuggestions, TestAtomicUpdateErrorCases, TestSolrXmlPersistence, TestParamSet, PrimUtilsTest, ModifyConfFileTest, SolrIndexSplitterTest, DOMUtilTest, DateMathParserTest, SuggesterTest, CircularListTest, LegacyHTMLStripCharFilterTest, SuggestComponentTest, TestMiniSolrCloudCluster, HighlighterConfigTest, TestStandardQParsers, SolrInfoMBeanTest, TestSchemaResource, TestValueSourceCache, TestSolrJ, TestChildDocTransformer, PreAnalyzedFieldTest, TestPseudoReturnFields, OutOfBoxZkACLAndCredentialsProvidersTest, TestManagedSchemaFieldResource, TestSolr4Spatial, DistanceFunctionTest, TestManagedResourceStorage, SpellingQueryConverterTest, SpellCheckCollatorTest, SpellPossibilityIteratorTest, PeerSyncTest, TestInfoStreamLogging, HdfsBasicDistributedZkTest, TestFileDictionaryLookup, LeaderElectionTest, SharedFSAutoReplicaFailoverTest, SoftAutoCommitTest, AtomicUpdatesTest, ZkCLITest, TestShardHandlerFactory, TestLuceneMatchVersion, TestDistributedMissingSort, ScriptEngineTest, DeleteLastCustomShardedReplicaTest, TestCollapseQParserPlugin, TestManagedSchemaFieldTypeResource, SpellCheckComponentTest, DeleteShardTest, OverseerTest, DistributedDebugComponentTest, HdfsLockFactoryTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 803.55s, 1 test, 1 error <<< FAILURES!

[...truncated 699 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 errors, 54 ignored (32 assumptions)

Total time: 181 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_67) - Build # 4312 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4312/
Java: 32bit/jdk1.7.0_67 -client -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6460, name=coreLoadExecutor-3852-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6460, name=coreLoadExecutor-3852-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@12e9f4d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13487fd)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6480, name=coreLoadExecutor-3859-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6480, name=coreLoadExecutor-3859-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@1801d44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cfd92)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6498, name=coreLoadExecutor-3866-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6498, name=coreLoadExecutor-3866-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@274403 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10ba15b)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "\fake\path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "\fake\path" "write")
	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD:5380019E9591BF05]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10866 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.CachingDirectoryFactoryTest-DBD43E443B6DD2FD-001\init-core-data-001
   [junit4]   2> 1790292 T3912 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1790296 T3912 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 1790662 T3913 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1ba49d1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9053a assuming 'simple'
   [junit4]   2> 1790671 T3917 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1de2296 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f8a6d assuming 'simple'
   [junit4]   2> 1790672 T3919 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@4939bc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1d63ff6 assuming 'simple'
   [junit4]   2> 1790673 T3922 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@cda7b3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1dc5f83 assuming 'simple'
   [junit4]   2> 1790673 T3918 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1029ba4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19fc18f assuming 'simple'
   [junit4]   2> 1790674 T3916 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@15cc247 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb7183 assuming 'simple'
   [junit4]   2> 1790676 T3914 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@16971c7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10cd827 assuming 'simple'
   [junit4]   2> 1790677 T3920 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@3529b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fd3d8 assuming 'simple'
   [junit4]   2> 385 T3917 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1569,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790679 T3915 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@139ba97 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f1fbc assuming 'simple'
   [junit4]   2> 385 T3922 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1574,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3913 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1565,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3916 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1568,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790679 T3921 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@ecae3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c2b49c assuming 'simple'
   [junit4]   2> 388 T3921 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1573,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790680 T3923 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@160f31c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c710ea assuming 'simple'
   [junit4]   2> 386 T3919 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1571,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3920 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1572,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 390 T3923 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1575,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 388 T3914 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1566,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 387 T3918 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1570,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 387 T3915 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1567,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1799517 T3930 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1799603 T3912 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=DBD43E443B6DD2FD -Dtests.slow=true -Dtests.locale=ar_SD -Dtests.timezone=America/Iqaluit -Dtests.file.encoding=Cp1252
   [junit4] ERROR   9.36s | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3914, name=Thread-1566, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3916, name=Thread-1568, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3917, name=Thread-1569, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3918, name=Thread-1570, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3922, name=Thread-1574, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3920, name=Thread-1572, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3919, name=Thread-1571, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3915, name=Thread-1567, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3921, name=Thread-1573, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3923, name=Thread-1575, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 1799653 T3912 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_SD, timezone=America/Iqaluit
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=185532568,total=373710848
   [junit4]   2> NOTE: All tests run in this JVM: [TestReplicationHandler, TestFreeTextSuggestions, SolrInfoMBeanTest, MinimalSchemaTest, URLClassifyProcessorTest, FunctionTest, TestArbitraryIndexDir, FieldFacetExtrasTest, TestSystemIdResolver, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, CurrencyFieldXmlFileTest, QueryParsingTest, TestRawResponseWriter, TestDynamicFieldCollectionResource, TestFastOutputStream, TestSolrXmlPersistence, IndexSchemaTest, BJQParserTest, UUIDFieldTest, TestDynamicFieldResource, TestCSVLoader, TestDistributedSearch, IndexBasedSpellCheckerTest, TestCloudManagedSchema, SolrTestCaseJ4Test, TestStressLucene, TestAnalyzedSuggestions, TestSolrDeletionPolicy1, SuggesterTSTTest, HdfsSyncSliceTest, TestSearcherReuse, NumericFieldsTest, TestOmitPositions, TestSolrXmlPersistor, TestLFUCache, TestDefaultSimilarityFactory, QueryFacetTest, TestFieldTypeResource, ShardSplitTest, RequestHandlersTest, TestManagedResourceStorage, TestFunctionQuery, AnalyticsQueryTest, DateFieldTest, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, SpellPossibilityIteratorTest, MigrateRouteKeyTest, BadComponentTest, JsonLoaderTest, MoreLikeThisHandlerTest, TestFastWriter, ClusterStateTest, AnalysisErrorHandlingTest, CursorPagingTest, TriLevelCompositeIdRoutingTest, ShardRoutingTest, ZkCLITest, SharedFSAutoReplicaFailoverUtilsTest, TestQuerySenderListener, TestCSVResponseWriter, TestNonNRTOpen, TestMultiCoreConfBootstrap, TestQueryTypes, StatelessScriptUpdateProcessorFactoryTest, BasicDistributedZkTest, TestStressVersions, TestTrie, TestFiltering, TestUpdate, TestModifyConfFiles, LeaderElectionIntegrationTest, ConvertedLegacyTest, SpellingQueryConverterTest, TestDocumentBuilder, AssignTest, TestJmxIntegration, TestStressUserVersions, TestFaceting, MBeansHandlerTest, TestRecoveryHdfs, TestPivotHelperCode, StandardRequestHandlerTest, OpenExchangeRatesOrgProviderTest, DisMaxRequestHandlerTest, TestCoreContainer, TestExtendedDismaxParser, TestValueSourceCache, TestJoin, EchoParamsTest, SortByFunctionTest, DocValuesMultiTest, TestShortCircuitedRequests, MergeStrategyTest, TestCoreDiscovery, TestLMDirichletSimilarityFactory, ExternalCollectionsTest, HardAutoCommitTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, ScriptEngineTest, TestMaxScoreQueryParser, CachingDirectoryFactoryTest]
   [junit4] Completed in 9.43s, 1 test, 1 error <<< FAILURES!

[...truncated 260 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestZkChroot-DBD43E443B6DD2FD-001\init-core-data-001
   [junit4]   2> 3106275 T6438 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3106284 T6438 oas.SolrTestCaseJ4.setUp ###Starting testNoBootstrapConf
   [junit4]   2> 3106286 T6438 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3106287 T6439 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3106374 T6438 oasc.ZkTestServer.run start zk server on port:52273
   [junit4]   2> 3106375 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3106376 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106380 T6445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64beb0 name:ZooKeeperConnection Watcher:127.0.0.1:52273 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106381 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106381 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3106382 T6438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\'
   [junit4]   2> 3106413 T6438 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\solr.xml
   [junit4]   2> 3106446 T6438 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3106447 T6438 oasc.CoreContainer.<init> New CoreContainer 6371933
   [junit4]   2> 3106447 T6438 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\]
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3106451 T6438 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 3106451 T6438 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3106452 T6438 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3106452 T6438 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3106453 T6438 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52273/foo/bar2
   [junit4]   2> 3106453 T6438 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3106453 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3106455 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106457 T6447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ed3c4 name:ZooKeeperConnection Watcher:127.0.0.1:52273 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106458 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106458 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3106462 T6440 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148766a75440001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3106465 T6438 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52273 52273
   [junit4]   2> 3108288 T6439 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52273 52273
   [junit4]   2> 3108289 T6438 oas.SolrTestCaseJ4.tearDown ###Ending testNoBootstrapConf
   [junit4]   2> 3108294 T6438 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 3108296 T6438 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3108297 T6448 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3108375 T6438 oasc.ZkTestServer.run start zk server on port:52282
   [junit4]   2> 3108375 T6438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\'
   [junit4]   2> 3108394 T6438 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\solr.xml
   [junit4]   2> 3108426 T6438 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108426 T6438 oasc.CoreContainer.<init> New CoreContainer 13594553
   [junit4]   2> 3108426 T6438 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\]
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3108428 T6438 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 3108430 T6438 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3108430 T6438 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3108430 T6438 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3108430 T6438 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52282/foo/bar
   [junit4]   2> 3108430 T6438 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3108430 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3108433 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108436 T6454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b00f6a name:ZooKeeperConnection Watcher:127.0.0.1:52282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108437 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108437 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3108438 T6438 oascc.SolrZkClient.makePath makePath: /foo/bar
   [junit4]   2> 3108446 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108447 T6456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1232469 name:ZooKeeperConnection Watcher:127.0.0.1:52282/foo/bar got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108447 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108449 T6438 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3108453 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3108457 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3108462 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3108465 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3108470 T6438 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3108472 T6438 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/192.168.57.16:8983_solr
   [junit4]   2> 3108473 T6438 oascc.SolrZkClient.makePath makePath: /live_nodes/192.168.57.16:8983_solr
   [junit4]   2> 3108476 T6438 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 3108479 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3108481 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3108484 T6438 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3108486 T6438 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 192.168.57.16:8983_solr
   [junit4]   2> 3108486 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3108488 T6438 oasc.Overseer.start Overseer (id=92453992098758657-192.168.57.16:8983_solr-n_0000000000) starting
   [junit4]   2> 3108492 T6438 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3108503 T6438 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3108505 T6458 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3108506 T6438 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3108510 T6438 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3108513 T6438 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3108516 T6438 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108518 T6457 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3108528 T6438 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\
   [junit4]   2> 3108528 T6438 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3108528 T6438 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\
   [junit4]   2> 3108528 T6438 oasc.ZkController.bootstrapConf Uploading directory C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\conf with name collection1 for SolrCore collection1
   [junit4]   2> 3108531 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 3108539 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 3108543 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 3108550 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 3108557 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 3108563 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 3108568 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 3108572 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 3108576 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 3108581 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 3108586 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 3108590 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 3108595 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 3108598 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 3108602 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 3108607 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 3108611 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 3108616 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 3108620 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 3108626 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 3108630 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 3108634 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 3108637 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 3108642 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 3108646 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 3108650 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 3108655 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 3108658 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 3108662 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 3108668 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 3108671 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 3108675 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 3108680 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 3108684 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 3108689 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 3108693 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 3108698 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 3108703 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 3108707 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 3108712 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 3108716 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 3108720 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 3108725 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 3108730 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 3108734 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 3108739 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 3108743 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 3108753 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 3108759 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3108762 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 3108767 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 3108771 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 3108775 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 3108780 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 3108783 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 3108787 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 3108791 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 3108795 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 3108800 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 3108804 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 3108808 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 3108811 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 3108816 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 3108820 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 3108824 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 3108829 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 3108833 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 3108838 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 3108842 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 3108847 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 3108851 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 3108856 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 3108860 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 3108865 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 3108870 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 3108873 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 3108879 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 3108883 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 3108887 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 3108892 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 3108895 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 3108900 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 3108904 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 3108909 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 3108913 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 3108917 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 3108920 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 3108925 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 3108930 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 3108934 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 3108939 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 3108943 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 3108949 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 3108954 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 3108960 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 3108963 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 3108967 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 3108972 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 3108976 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 3108980 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 3108984 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 3108993 T6438 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108996 T6438 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\
   [junit4]   2> 3108996 T6438 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3108998 T6460 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3108998 T6460 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3108999 T6460 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108999 T6461 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3109003 T6457 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://192.168.57.16:8983/solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"192.168.57.16:8983_solr"}
   [junit4]   2> 3109003 T6457 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3109008 T6461 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3109777 T6460 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3109777 T6460 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3109779 T6460 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 3109779 T6460 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3109782 T6460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3109783 T6460 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 3109783 T6460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\'
   [junit4]   2> 3109811 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=6371933
   [junit4]   2> 3109844 T6460 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 3109846 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\extraction\lib).
   [junit4]   2> 3109846 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\clustering\lib).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\langid\lib).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\velocity\lib).
   [junit4]   2> 3109851 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109885 T6460 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 3109890 T6460 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 3109963 T6460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3109966 T6460 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3109976 T6460 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 3110433 T6460 o

[...truncated too long message...]

.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@274403 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10ba15b)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed in 14.64s, 4 tests, 3 errors <<< FAILURES!

[...truncated 13 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 3265593 T6986 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@14b73d2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@167ece2 assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=DBD43E443B6DD2FD -Dtests.slow=true -Dtests.locale=fr_CA -Dtests.timezone=Africa/Bissau -Dtests.file.encoding=Cp1252
   [junit4] ERROR   0.01s | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "\fake\path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD:5380019E9591BF05]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr_CA, timezone=Africa/Bissau
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=231459928,total=523501568
   [junit4]   2> NOTE: All tests run in this JVM: [TestReplicationHandler, TestFreeTextSuggestions, SolrInfoMBeanTest, MinimalSchemaTest, URLClassifyProcessorTest, FunctionTest, TestArbitraryIndexDir, FieldFacetExtrasTest, TestSystemIdResolver, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, CurrencyFieldXmlFileTest, QueryParsingTest, TestRawResponseWriter, TestDynamicFieldCollectionResource, TestFastOutputStream, TestSolrXmlPersistence, IndexSchemaTest, BJQParserTest, UUIDFieldTest, TestDynamicFieldResource, TestCSVLoader, TestDistributedSearch, IndexBasedSpellCheckerTest, TestCloudManagedSchema, SolrTestCaseJ4Test, TestStressLucene, TestAnalyzedSuggestions, TestSolrDeletionPolicy1, SuggesterTSTTest, HdfsSyncSliceTest, TestSearcherReuse, NumericFieldsTest, TestOmitPositions, TestSolrXmlPersistor, TestLFUCache, TestDefaultSimilarityFactory, QueryFacetTest, TestFieldTypeResource, ShardSplitTest, RequestHandlersTest, TestManagedResourceStorage, TestFunctionQuery, AnalyticsQueryTest, DateFieldTest, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, SpellPossibilityIteratorTest, MigrateRouteKeyTest, BadComponentTest, JsonLoaderTest, MoreLikeThisHandlerTest, TestFastWriter, ClusterStateTest, AnalysisErrorHandlingTest, CursorPagingTest, TriLevelCompositeIdRoutingTest, ShardRoutingTest, ZkCLITest, SharedFSAutoReplicaFailoverUtilsTest, TestQuerySenderListener, TestCSVResponseWriter, TestNonNRTOpen, TestMultiCoreConfBootstrap, TestQueryTypes, StatelessScriptUpdateProcessorFactoryTest, BasicDistributedZkTest, TestStressVersions, TestTrie, TestFiltering, TestUpdate, TestModifyConfFiles, LeaderElectionIntegrationTest, ConvertedLegacyTest, SpellingQueryConverterTest, TestDocumentBuilder, AssignTest, TestJmxIntegration, TestStressUserVersions, TestFaceting, MBeansHandlerTest, TestRecoveryHdfs, TestPivotHelperCode, StandardRequestHandlerTest, OpenExchangeRatesOrgProviderTest, DisMaxRequestHandlerTest, TestCoreContainer, TestExtendedDismaxParser, TestValueSourceCache, TestJoin, EchoParamsTest, SortByFunctionTest, DocValuesMultiTest, TestShortCircuitedRequests, MergeStrategyTest, TestCoreDiscovery, TestLMDirichletSimilarityFactory, ExternalCollectionsTest, HardAutoCommitTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, ScriptEngineTest, TestMaxScoreQueryParser, CachingDirectoryFactoryTest, RegexBoostProcessorTest, TestRandomFaceting, DeleteInactiveReplicaTest, TestStressRecovery, TestFastLRUCache, TestExceedMaxTermLength, DateRangeFieldTest, AddBlockUpdateTest, SSLMigrationTest, ResourceLoaderTest, EnumFieldTest, UUIDUpdateProcessorFallbackTest, FieldMutatingUpdateProcessorTest, HighlighterConfigTest, BasicZkTest, TestCharFilters, TestAtomicUpdateErrorCases, SuggesterTest, TestCloudPivotFacet, TestImplicitCoreProperties, TestOverriddenPrefixQueryForCustomFieldType, StatsComponentTest, TestDistributedGrouping, TestRTGBase, TestCollationFieldDocValues, TestFoldingMultitermQuery, PolyFieldTest, CopyFieldTest, DocValuesMissingTest, TestOrdValues, TestQueryUtils, TestRangeQuery, TestParamSet, TestSolr4Spatial2, OverriddenZkACLAndCredentialsProvidersTest, DistribCursorPagingTest, BadIndexSchemaTest, AtomicUpdatesTest, DeleteReplicaTest, SearchHandlerTest, ReplicationFactorTest, CacheHeaderTest, ShowFileRequestHandlerTest, WordBreakSolrSpellCheckerTest, HdfsLockFactoryTest, BasicFunctionalityTest, TestSolrXml, FileBasedSpellCheckerTest, ChangedSchemaMergeTest, TestPerFieldSimilarity, TestNoOpRegenerator, TestLazyCores, AliasIntegrationTest, TestHighFrequencyDictionaryFactory, SimpleFacetsTest, AbstractAnalyticsStatsTest, TestIndexingPerformance, SolrCoreTest, TestSolrCoreProperties, TestBinaryField, TestBinaryResponseWriter, TestStandardQParsers, AutoCommitTest, CoreMergeIndexesAdminHandlerTest, TestCloudSchemaless, TestHashPartitioner, MultiThreadedOCPTest, SuggesterWFSTTest, NoFacetTest, DefaultValueUpdateProcessorTest, HdfsBasicDistributedZkTest, CoreAdminCreateDiscoverTest, TestSolrDeletionPolicy2, CoreAdminRequestStatusTest, FileUtilsTest, TestReloadAndDeleteDocs, CursorMarkTest, NotRequiredUniqueKeyTest, TestTolerantSearch, TestSearchPerf, TestZkChroot, TestSolrQueryParser, DistributedSpellCheckComponentTest, OverseerTest, ExpressionTest, RAMDirectoryFactoryTest]
   [junit4] Completed in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 792 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 54 ignored (32 assumptions)

Total time: 173 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_20) - Build # 4311 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4311/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTasksParse

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001

	at __randomizedtesting.SeedInfo.seed([EBB0DBD1FB2CCFE9]:0)
	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:126)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 7125 lines...]
   [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> Spatial Strategy: RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:SpatialContext{geo=true, calculator=Haversine, worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ro_RO, timezone=Europe/Minsk
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_20 (32-bit)/cpus=2,threads=1,free=21303776,total=32571392
   [junit4]   2> NOTE: All tests run in this JVM: [TestPerfTasksParse]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse -Dtests.seed=EBB0DBD1FB2CCFE9 -Dtests.slow=true -Dtests.locale=ro_RO -Dtests.timezone=Europe/Minsk -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
   [junit4]    > Throwable #1: java.io.IOException: Could not remove the following files (in the order of attempts):
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EBB0DBD1FB2CCFE9]:0)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed in 3.21s, 3 tests, 1 error <<< FAILURES!

[...truncated 60 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:462: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2140: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 18 suites, 93 tests, 1 suite-level error

Total time: 30 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Robert Muir <rc...@gmail.com>.
thank you for debugging this! I didn't know about this crazy finalizer
at all: I tried to figure it out last night, but i could not see the
problem and without a windows machine it was hopeless for me.

On Sun, Sep 14, 2014 at 7:29 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> I know why it fails consequently with the new Java 7 APIs:
> FileOutputStream since version 1.0 of Java has a custom finalizer() method, so the file gets closed on GC, kept until now for backwards compatibility. But if you the stream with Java 7's NIO.2, the created OutputStream instance (some non-public internal platform specific impl class) no longer has a finalizer(), so the file handle is never closed when the OutputStream is GCed. This is great (f*ck finalizers)!
>
> So by using NIO.2 we have a very nice "detector" for unclosed file, if people do broken stuff like "new Properties(new FileInputStream(...))" as seen in Solr quite often!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Sunday, September 14, 2014 1:16 PM
>> To: dev@lucene.apache.org
>> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> Build # 4310 - Still Failing!
>>
>> The problem here is incomplete cleanup:
>>
>> This test parses the whole algorithm, it then creates an Algorithm instance
>> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated,
>> which in fact opens the output stream for the line docs file in its ctor. But as
>> the tasks are never executed, so the Task's close() is never called.
>>
>> I fixed the whole thing by making Algorithm implement AutoCloseable (not
>> Closeable, because it throws not only IOException). I also backported the
>> whole stuff to Lucene 4, because this is unrelated to Robert's changes.
>> Roberts changes just triggered other code execution paths, so GC did not
>> clean up algorithm.
>>
>> TODO: verify other uses of Algorithm and check that they are correctly
>> closed. In fact I have seen this bug sometimes when running tests locally, so
>> this was existent since long time, just in most cases GC was faster!
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> > Sent: Sunday, September 14, 2014 12:21 PM
>> > To: dev@lucene.apache.org
>> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> > Build # 4310 - Still Failing!
>> >
>> > This fails reproducible.
>> >
>> > I will dig!
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> > > -----Original Message-----
>> > > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
>> > > Sent: Sunday, September 14, 2014 2:48 AM
>> > > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
>> > > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
>> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> > > Build #
>> > > 4310 - Still Failing!
>> > >
>> > > Build:
>> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
>> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
>> > > XX:+UseConcMarkSweepGC
>> > >
>> > > 1 tests failed.
>> > > FAILED:
>> > >
>> >
>> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
>> > > a
>> > > sksParse
>> > >
>> > > Error Message:
>> > > Could not remove the following files (in the order of attempts):
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >
>> > > Stack Trace:
>> > > java.io.IOException: Could not remove the following files (in the
>> > > order of
>> > > attempts):
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >
>> > >   at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>> > >   at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>> > >   at
>> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes
>> > > tR
>> > > ul
>> > > eTemporaryFilesCleanup.java:126)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa
>> > > ys
>> > > (T
>> > > estRuleAdapter.java:31)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:43)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:36)
>> > >   at
>> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul
>> > > eA
>> > > ss
>> > > ertionsRequired.java:43)
>> > >   at
>> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa
>> > > il
>> > > ure
>> > > .java:48)
>> > >   at
>> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes
>> > > tR
>> > > ule
>> > > IgnoreAfterMaxFailures.java:65)
>> > >   at
>> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI
>> > > gn
>> > > ore
>> > > TestSuites.java:55)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:36)
>> > >   at
>> > >
>> >
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
>> > > run(ThreadLeakControl.java:365)
>> > >   at java.lang.Thread.run(Thread.java:745)
>> > >
>> > >
>> > >
>> > >
>> > > Build Log:
>> > > [...truncated 7622 lines...]
>> > >    [junit4] Suite:
>> org.apache.lucene.benchmark.byTask.TestPerfTasksParse
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> Spatial Strategy:
>> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa
>> > > ti al Context{geo=true, calculator=Haversine,
>> > > worldBounds=Rect(minX=-
>> > > 180.0,maxX=180.0,minY=-
>> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
>> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
>> > >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67
>> (64-
>> > > bit)/cpus=2,threads=1,free=120281832,total=140083200
>> > >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
>> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
>> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
>> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
>> > >    [junit4]   2> NOTE: reproduce with: ant test  -
>> > Dtestcase=TestPerfTasksParse
>> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL
>> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
>> > >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
>> > >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
>> > > following files (in the order of attempts):
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >    [junit4]    >  at
>> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>> > >    [junit4]    >  at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>> > >    [junit4]    >  at java.lang.Thread.run(Thread.java:745)
>> > >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
>> > >
>> > > [...truncated 33 lines...]
>> > > BUILD FAILED
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\build.xml:491: The following error occurred while executing
>> > > this
>> > > line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\build.xml:471: The following error occurred while executing
>> > > this
>> > > line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > Windows\build.xml:60:
>> > > The following error occurred while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
>> > > targets.xml:39: The following error occurred while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\build.xml:462: The following error occurred while
>> > > executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:2140: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\module-build.xml:58: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:1358: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:961: There were test failures: 18
>> > > suites,
>> > > 93 tests, 1 suite-level error
>> > >
>> > > Total time: 31 minutes 50 seconds
>> > > Build step 'Invoke Ant' marked build as failure [description-setter]
>> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
>> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
>> > > Email was triggered for: Failure - Any Sending email for trigger:
>> > > Failure - Any
>> > >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> > additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

I know why it fails consequently with the new Java 7 APIs:
FileOutputStream since version 1.0 of Java has a custom finalizer() method, so the file gets closed on GC, kept until now for backwards compatibility. But if you the stream with Java 7's NIO.2, the created OutputStream instance (some non-public internal platform specific impl class) no longer has a finalizer(), so the file handle is never closed when the OutputStream is GCed. This is great (f*ck finalizers)!

So by using NIO.2 we have a very nice "detector" for unclosed file, if people do broken stuff like "new Properties(new FileInputStream(...))" as seen in Solr quite often!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, September 14, 2014 1:16 PM
> To: dev@lucene.apache.org
> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> Build # 4310 - Still Failing!
> 
> The problem here is incomplete cleanup:
> 
> This test parses the whole algorithm, it then creates an Algorithm instance
> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated,
> which in fact opens the output stream for the line docs file in its ctor. But as
> the tasks are never executed, so the Task's close() is never called.
> 
> I fixed the whole thing by making Algorithm implement AutoCloseable (not
> Closeable, because it throws not only IOException). I also backported the
> whole stuff to Lucene 4, because this is unrelated to Robert's changes.
> Roberts changes just triggered other code execution paths, so GC did not
> clean up algorithm.
> 
> TODO: verify other uses of Algorithm and check that they are correctly
> closed. In fact I have seen this bug sometimes when running tests locally, so
> this was existent since long time, just in most cases GC was faster!
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Sunday, September 14, 2014 12:21 PM
> > To: dev@lucene.apache.org
> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > Build # 4310 - Still Failing!
> >
> > This fails reproducible.
> >
> > I will dig!
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> > > Sent: Sunday, September 14, 2014 2:48 AM
> > > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> > > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > > Build #
> > > 4310 - Still Failing!
> > >
> > > Build:
> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC
> > >
> > > 1 tests failed.
> > > FAILED:
> > >
> >
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
> > > a
> > > sksParse
> > >
> > > Error Message:
> > > Could not remove the following files (in the order of attempts):
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > Stack Trace:
> > > java.io.IOException: Could not remove the following files (in the
> > > order of
> > > attempts):
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > > 	at
> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes
> > > tR
> > > ul
> > > eTemporaryFilesCleanup.java:126)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa
> > > ys
> > > (T
> > > estRuleAdapter.java:31)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:43)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul
> > > eA
> > > ss
> > > ertionsRequired.java:43)
> > > 	at
> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa
> > > il
> > > ure
> > > .java:48)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes
> > > tR
> > > ule
> > > IgnoreAfterMaxFailures.java:65)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI
> > > gn
> > > ore
> > > TestSuites.java:55)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > >
> >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > > run(ThreadLeakControl.java:365)
> > > 	at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > >
> > >
> > > Build Log:
> > > [...truncated 7622 lines...]
> > >    [junit4] Suite:
> org.apache.lucene.benchmark.byTask.TestPerfTasksParse
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> Spatial Strategy:
> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa
> > > ti al Context{geo=true, calculator=Haversine,
> > > worldBounds=Rect(minX=-
> > > 180.0,maxX=180.0,minY=-
> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
> > >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67
> (64-
> > > bit)/cpus=2,threads=1,free=120281832,total=140083200
> > >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
> > >    [junit4]   2> NOTE: reproduce with: ant test  -
> > Dtestcase=TestPerfTasksParse
> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL
> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
> > >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
> > >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> > > following files (in the order of attempts):
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >    [junit4]    > 	at
> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > >    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > >    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
> > >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> > >
> > > [...truncated 33 lines...]
> > > BUILD FAILED
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:491: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:471: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:60:
> > > The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> > > targets.xml:39: The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\build.xml:462: The following error occurred while
> > > executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:2140: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\module-build.xml:58: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:1358: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:961: There were test failures: 18
> > > suites,
> > > 93 tests, 1 suite-level error
> > >
> > > Total time: 31 minutes 50 seconds
> > > Build step 'Invoke Ant' marked build as failure [description-setter]
> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
> > > Email was triggered for: Failure - Any Sending email for trigger:
> > > Failure - Any
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
The problem here is incomplete cleanup:

This test parses the whole algorithm, it then creates an Algorithm instance containing all Tasks to execute. The "WriteLineDocsTask" is instantiated, which in fact opens the output stream for the line docs file in its ctor. But as the tasks are never executed, so the Task's close() is never called.

I fixed the whole thing by making Algorithm implement AutoCloseable (not Closeable, because it throws not only IOException). I also backported the whole stuff to Lucene 4, because this is unrelated to Robert's changes. Roberts changes just triggered other code execution paths, so GC did not clean up algorithm.

TODO: verify other uses of Algorithm and check that they are correctly closed. In fact I have seen this bug sometimes when running tests locally, so this was existent since long time, just in most cases GC was faster!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, September 14, 2014 12:21 PM
> To: dev@lucene.apache.org
> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> Build # 4310 - Still Failing!
> 
> This fails reproducible.
> 
> I will dig!
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> > -----Original Message-----
> > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> > Sent: Sunday, September 14, 2014 2:48 AM
> > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > Build #
> > 4310 - Still Failing!
> >
> > Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > XX:+UseConcMarkSweepGC
> >
> > 1 tests failed.
> > FAILED:
> >
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
> > a
> > sksParse
> >
> > Error Message:
> > Could not remove the following files (in the order of attempts):
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >
> > Stack Trace:
> > java.io.IOException: Could not remove the following files (in the
> > order of
> > attempts):
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >
> > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > 	at
> > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestR
> > ul
> > eTemporaryFilesCleanup.java:126)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways
> > (T
> > estRuleAdapter.java:31)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:43)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:36)
> > 	at
> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleA
> > ss
> > ertionsRequired.java:43)
> > 	at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail
> > ure
> > .java:48)
> > 	at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR
> > ule
> > IgnoreAfterMaxFailures.java:65)
> > 	at
> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgn
> > ore
> > TestSuites.java:55)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:36)
> > 	at
> >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> > 	at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > Build Log:
> > [...truncated 7622 lines...]
> >    [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> Spatial Strategy:
> > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spati
> > al Context{geo=true, calculator=Haversine, worldBounds=Rect(minX=-
> > 180.0,maxX=180.0,minY=-
> > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
> >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-
> > bit)/cpus=2,threads=1,free=120281832,total=140083200
> >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
> > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
> >    [junit4]   2> NOTE: reproduce with: ant test  -
> Dtestcase=TestPerfTasksParse
> > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -
> > Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
> >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
> >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> > following files (in the order of attempts):
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >    [junit4]    > 	at
> > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> >    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> >    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
> >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> >
> > [...truncated 33 lines...]
> > BUILD FAILED
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:491: The following error occurred while executing
> > this
> > line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:471: The following error occurred while executing
> > this
> > line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:60:
> > The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> > targets.xml:39: The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\build.xml:462: The following error occurred while
> > executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:2140: The following error occurred
> > while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\module-build.xml:58: The following error occurred while
> > executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:1358: The following error occurred
> > while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:961: There were test failures: 18
> > suites,
> > 93 tests, 1 suite-level error
> >
> > Total time: 31 minutes 50 seconds
> > Build step 'Invoke Ant' marked build as failure [description-setter]
> > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
> > Email was triggered for: Failure - Any Sending email for trigger:
> > Failure - Any
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
This fails reproducible.

I will dig!

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Sunday, September 14, 2014 2:48 AM
> To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build #
> 4310 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> XX:+UseConcMarkSweepGC
> 
> 1 tests failed.
> FAILED:
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTa
> sksParse
> 
> Error Message:
> Could not remove the following files (in the order of attempts):
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
> 
> Stack Trace:
> java.io.IOException: Could not remove the following files (in the order of
> attempts):
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
> 
> 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> 	at
> org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRul
> eTemporaryFilesCleanup.java:126)
> 	at
> com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(T
> estRuleAdapter.java:31)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:43)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> ertionsRequired.java:43)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> TestSuites.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 
> Build Log:
> [...truncated 7622 lines...]
>    [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> Spatial Strategy:
> RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spatial
> Context{geo=true, calculator=Haversine, worldBounds=Rect(minX=-
> 180.0,maxX=180.0,minY=-
> 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
>    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-
> bit)/cpus=2,threads=1,free=120281832,total=140083200
>    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> TrecContentSourceTest, TestQualityRun, PerfTaskTest, LineDocSourceTest,
> TestPerfTasksLogic, TestPerfTasksParse]
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse
> -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -
> Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
>    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
>    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> following files (in the order of attempts):
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
>    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> 
> [...truncated 33 lines...]
> BUILD FAILED
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:491: The following error occurred while executing this
> line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:471: The following error occurred while executing this
> line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60:
> The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> targets.xml:39: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build.xml:462: The following error occurred while executing
> this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:2140: The following error occurred
> while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\module-build.xml:58: The following error occurred while
> executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:1358: The following error occurred
> while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:961: There were test failures: 18 suites,
> 93 tests, 1 suite-level error
> 
> Total time: 31 minutes 50 seconds
> Build step 'Invoke Ant' marked build as failure [description-setter]
> Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> XX:+UseConcMarkSweepGC Archiving artifacts Recording test results Email
> was triggered for: Failure - Any Sending email for trigger: Failure - Any
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTasksParse

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001

	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:126)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 7622 lines...]
   [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> Spatial Strategy: RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:SpatialContext{geo=true, calculator=Haversine, worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=2,threads=1,free=120281832,total=140083200
   [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest, WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest, TrecContentSourceTest, TestQualityRun, PerfTaskTest, LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
   [junit4]    > Throwable #1: java.io.IOException: Could not remove the following files (in the order of attempts):
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!

[...truncated 33 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:462: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2140: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 18 suites, 93 tests, 1 suite-level error

Total time: 31 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_20) - Build # 4309 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4309/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 31708 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.3
[forbidden-apis] Reading API signatures: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\tools\forbiddenApis\base.txt
[forbidden-apis] Reading API signatures: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\tools\forbiddenApis\servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#<init>(byte[]) [Uses default charset]
[forbidden-apis]   in org.apache.solr.cloud.Overseer$ClusterStateUpdater (Overseer.java:316)
[forbidden-apis] Scanned 2357 (and 1560 related) class file(s) for forbidden API invocations (in 3.45s), 1 error(s).

BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:85: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:271: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:479: Check for forbidden API calls failed, see log.

Total time: 177 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any