You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/12 14:24:40 UTC

[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_20) - Build # 4307 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4307/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseG1GC

2 tests failed.
REGRESSION:  org.apache.solr.handler.TestReplicationHandlerBackup.doTestBackup

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([D9185B50208A695A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([D9185B50208A695A]:0)




Build Log:
[...truncated 11210 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandlerBackup
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\init-core-data-001
   [junit4]   2> 2369644 T6566 oas.SolrTestCaseJ4.setUp ###Starting doTestBackup
   [junit4]   2> 2369659 T6566 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2369662 T6566 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54815
   [junit4]   2> 2369664 T6566 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2369664 T6566 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2369664 T6566 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001
   [junit4]   2> 2369664 T6566 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\'
   [junit4]   2> 2369679 T6566 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\solr.xml
   [junit4]   2> 2369684 T6566 oasc.CoreContainer.<init> New CoreContainer 8705765
   [junit4]   2> 2369684 T6566 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\]
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2369686 T6566 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2369687 T6566 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2369687 T6566 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2369687 T6566 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2369687 T6566 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2369691 T6576 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\'
   [junit4]   2> 2369711 T6576 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2369715 T6576 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2369716 T6576 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2369718 T6576 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2369720 T6576 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2369720 T6576 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\
   [junit4]   2> 2369720 T6576 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2369720 T6576 oasc.SolrCore.<init> [collection1] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\
   [junit4]   2> 2369720 T6576 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2369721 T6576 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\
   [junit4]   2> 2369721 T6576 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index/
   [junit4]   2> 2369721 T6576 oasc.SolrCore.initIndex WARN [collection1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 2369722 T6576 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index
   [junit4]   2> 2369722 T6576 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2369722 T6576 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b1fef7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1722856),segFN=segments_1,generation=1}
   [junit4]   2> 2369723 T6576 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2369724 T6576 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2369724 T6576 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2369724 T6576 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2369724 T6576 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2369724 T6576 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2369725 T6576 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2369725 T6576 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2369725 T6576 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2369725 T6576 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2369725 T6576 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2369727 T6576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2369727 T6576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2369729 T6576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2369729 T6576 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2369731 T6576 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2369731 T6576 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2369731 T6576 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=13, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2369731 T6576 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b1fef7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1722856),segFN=segments_1,generation=1}
   [junit4]   2> 2369731 T6576 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2369732 T6576 oass.SolrIndexSearcher.<init> Opening Searcher@150303e[collection1] main
   [junit4]   2> 2369732 T6576 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\conf
   [junit4]   2> 2369732 T6576 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\conf}
   [junit4]   2> 2369733 T6576 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\conf
   [junit4]   2> 2369733 T6576 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2369734 T6576 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\conf
   [junit4]   2> 2369735 T6576 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2369735 T6576 oash.ReplicationHandler.inform Replication enabled for following config files: schema-replication2.xml:schema.xml
   [junit4]   2> 2369735 T6576 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 2369735 T6577 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@150303e[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2369735 T6576 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2369736 T6566 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2369736 T6566 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C1032 name=collection1 org.apache.solr.core.SolrCore@13dcfda
   [junit4]   2> 2369738 T6569 C1032 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 2369738 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 0
   [junit4]   2> 2369741 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 1
   [junit4]   2> 2369743 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 0
   [junit4]   2> 2369745 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 0
   [junit4]   2> 2369747 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 0
   [junit4]   2> 2369750 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 0
   [junit4]   2> 2369753 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 0
   [junit4]   2> 2369758 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 0
   [junit4]   2> 2369761 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 0
   [junit4]   2> 2369763 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 0
   [junit4]   2> 2369765 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 0
   [junit4]   2> 2369767 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 0
   [junit4]   2> 2369769 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 0
   [junit4]   2> 2369771 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 0
   [junit4]   2> 2369772 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 0
   [junit4]   2> 2369774 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 0
   [junit4]   2> 2369776 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 0
   [junit4]   2> 2369778 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 0
   [junit4]   2> 2369780 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 0
   [junit4]   2> 2369782 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 0
   [junit4]   2> 2369784 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 0
   [junit4]   2> 2369786 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 0
   [junit4]   2> 2369788 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 0
   [junit4]   2> 2369790 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 0
   [junit4]   2> 2369794 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 0
   [junit4]   2> 2369796 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 0
   [junit4]   2> 2369800 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 0
   [junit4]   2> 2369803 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26]} 0 0
   [junit4]   2> 2369807 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27]} 0 0
   [junit4]   2> 2369810 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[28]} 0 1
   [junit4]   2> 2369813 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[29]} 0 0
   [junit4]   2> 2369817 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[30]} 0 0
   [junit4]   2> 2369820 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[31]} 0 0
   [junit4]   2> 2369822 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[32]} 0 0
   [junit4]   2> 2369825 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[33]} 0 0
   [junit4]   2> 2369829 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[34]} 0 0
   [junit4]   2> 2369832 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[35]} 0 0
   [junit4]   2> 2369835 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36]} 0 0
   [junit4]   2> 2369839 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[37]} 0 1
   [junit4]   2> 2369841 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[38]} 0 0
   [junit4]   2> 2369844 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39]} 0 0
   [junit4]   2> 2369847 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[40]} 0 0
   [junit4]   2> 2369850 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41]} 0 0
   [junit4]   2> 2369853 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[42]} 0 0
   [junit4]   2> 2369856 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[43]} 0 0
   [junit4]   2> 2369859 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44]} 0 0
   [junit4]   2> 2369863 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45]} 0 0
   [junit4]   2> 2369865 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[46]} 0 0
   [junit4]   2> 2369868 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47]} 0 0
   [junit4]   2> 2369870 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48]} 0 0
   [junit4]   2> 2369874 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[49]} 0 0
   [junit4]   2> 2369877 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[50]} 0 0
   [junit4]   2> 2369879 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[51]} 0 0
   [junit4]   2> 2369881 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[52]} 0 0
   [junit4]   2> 2369885 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[53]} 0 0
   [junit4]   2> 2369888 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54]} 0 0
   [junit4]   2> 2369891 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[55]} 0 0
   [junit4]   2> 2369894 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56]} 0 0
   [junit4]   2> 2369897 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[57]} 0 0
   [junit4]   2> 2369899 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[58]} 0 0
   [junit4]   2> 2369903 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[59]} 0 0
   [junit4]   2> 2369908 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[60]} 0 2
   [junit4]   2> 2369916 T6572 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[61]} 0 4
   [junit4]   2> 2369923 T6573 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[62]} 0 5
   [junit4]   2> 2369926 T6574 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63]} 0 0
   [junit4]   2> 2369935 T6575 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64]} 0 0
   [junit4]   2> 2369944 T6571 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[65]} 0 6
   [junit4]   2> 2369948 T6569 C1032 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 2369956 T6569 C1032 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b1fef7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1722856),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1b1fef7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1722856),segFN=segments_2,generation=2}
   [junit4]   2> 2369958 T6569 C1032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 2369958 T6569 C1032 oass.SolrIndexSearcher.<init> Opening Searcher@71583[collection1] main
   [junit4]   2> 2369958 T6569 C1032 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 2369959 T6577 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71583[collection1] main{UninvertingDirectoryReader(Uninverting(_0(5.0.0):C66))}
   [junit4]   2> 2369959 T6569 C1032 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} {commit=} 0 11
   [junit4]   2> 2369970 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2369984 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=backup&numberToKeep=1} status=0 QTime=7 
   [junit4]   2> 2369995 T6581 oash.SnapShooter.createSnapshot Creating backup snapshot...
   [junit4]   2> 2370026 T6581 oash.SnapShooter.createSnapshot Done creating backup snapshot: <not named>
   [junit4]   2> 2370105 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2370800 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2370806 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=backup&numberToKeep=1} status=0 QTime=3 
   [junit4]   2> 2370811 T6583 oash.SnapShooter.createSnapshot Creating backup snapshot...
   [junit4]   2> 2370828 T6583 oash.SnapShooter.createSnapshot Done creating backup snapshot: <not named>
   [junit4]   2> 2370947 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371093 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371245 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371391 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371542 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371690 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371837 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2371983 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2372130 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2372277 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2372423 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2372570 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2372717 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2372865 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373012 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373158 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373305 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373452 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373599 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373745 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2373892 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374039 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374186 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374333 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374479 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374626 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374772 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2374919 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375066 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375213 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375360 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375507 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375654 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2375803 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2375950 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376097 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2376245 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376392 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376539 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376686 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376833 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2376983 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2377131 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2377278 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2377425 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2377572 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2377718 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2377878 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2378040 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2378202 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2378364 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2378526 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2378687 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2378849 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379011 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379174 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2379335 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379496 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379658 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379820 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2379982 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2380144 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2380305 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2380466 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2380628 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2380789 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2380950 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2381112 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2381277 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2381438 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2381599 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2381761 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2381923 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382085 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382247 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382408 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382569 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382730 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2382892 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2383053 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2383214 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2383376 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2383538 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2383700 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2383862 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2384024 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2384186 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2384347 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2384508 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2384672 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2384833 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2384995 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2385157 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2385319 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2385480 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2385642 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2385812 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2385990 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2386173 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2386351 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2386528 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2386705 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2386883 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2387060 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2387239 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2387418 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2387596 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2387773 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2387950 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2388128 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2388305 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2388483 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2388660 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2388839 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2389016 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2389195 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2389373 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2389550 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2389728 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2389906 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2390084 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2390262 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2390441 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2390619 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2390797 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2390974 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2391153 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2391330 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2391507 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2391685 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2391862 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392040 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392217 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392394 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392572 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392749 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2392926 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393104 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393281 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393459 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393636 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393813 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2393991 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2394168 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2394346 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2394528 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2394726 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2394920 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2395115 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2395310 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2395505 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2395700 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2395895 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2396091 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2396286 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2396481 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2396677 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2396872 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2397067 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2397262 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2397458 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2397653 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2397848 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2398044 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2398239 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2398434 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2398630 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2398827 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399023 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399217 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399412 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399608 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399803 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2399999 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2400194 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2400389 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2400584 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2400779 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2400974 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2401169 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2401364 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2401559 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2401755 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2401950 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2402145 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2402340 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2402535 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2402730 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2402926 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2403121 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2403317 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2403512 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2403707 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2403902 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2404097 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2404307 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2404521 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2404736 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2404951 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2405165 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2405380 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2405595 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2405809 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2406024 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2406238 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2406453 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2406668 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2406882 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2407097 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2407312 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2407526 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2407741 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2407964 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2408178 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2408393 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2408608 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2408822 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2409037 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2409253 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2409467 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2409683 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2409898 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2410113 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2410328 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2410543 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2410758 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2410973 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2411188 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2411402 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2411617 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2411831 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2412046 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2412262 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2412477 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2412692 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2412909 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2413123 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2413339 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2413554 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2413768 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2413983 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2414199 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2414413 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2414629 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2414852 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2415088 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2415324 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2415561 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2415797 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2416034 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2416272 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2416508 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2416747 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2416986 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2417225 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2417461 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2417699 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2417935 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2418171 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2418407 T6571 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2418643 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2418880 T6572 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2419116 T6574 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2419352 T6575 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2419589 T6569 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2419825 T6573 C1032 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> ASYNC  NEW_CORE C1033 name=collection1 org.apache.solr.core.SolrCore@13dcfda
   [junit4]   2> 2420093 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2420329 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2420565 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2420801 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2421037 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2421274 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2421510 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2421747 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2421983 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2422219 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2422455 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2422691 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2422927 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2423163 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2423399 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2423635 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2423872 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2424109 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2424346 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2424582 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2424818 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2425055 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2425293 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2425529 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2425765 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2426001 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2426237 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2426476 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2426736 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2426996 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2427265 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2427525 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2427785 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2428046 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2428307 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2428566 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2428827 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2429087 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2429347 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2429606 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2429866 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2430126 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2430386 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2430645 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2430905 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2431165 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2431424 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2431684 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2431944 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2432203 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2432463 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2432723 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2432983 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2433242 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2433503 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2433763 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2434023 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2434282 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2434543 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2434803 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2435063 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2435322 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2435582 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2435842 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2436102 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2436361 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2436621 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2436881 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2437140 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2437400 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2437660 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2437921 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2438180 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2438440 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2438700 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2438959 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2439220 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2439499 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2439789 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2440075 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2440360 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2440647 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2440933 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2441219 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2441510 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2441798 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2442084 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2442372 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2442658 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2442944 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2443229 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2443515 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2443801 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2444087 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2444372 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2444658 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2444944 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2445229 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2445515 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2445801 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2446086 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2446372 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2446658 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2446943 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2447230 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2447519 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2447805 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2448090 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2448377 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2448663 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2448950 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2449237 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2449523 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2449808 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2450094 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2450380 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2450667 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2450953 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2451244 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2451529 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2451816 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2452102 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2452388 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2452674 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2452958 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2453244 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2453536 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2453839 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2454142 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2454445 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2454749 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2455052 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2455356 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2455660 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=2 
   [junit4]   2> 2455963 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2456268 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2456572 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2456875 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2457180 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2457483 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2457787 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2458090 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2458393 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2458696 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2458999 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2459303 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2459606 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2459912 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2460215 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2460520 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2460823 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2461126 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2461431 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2461734 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2462037 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2462341 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2462645 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2462954 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2463259 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2463565 T6572 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2463868 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=1 
   [junit4]   2> 2464172 T6571 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2464475 T6569 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2464778 T6573 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2465081 T6574 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2465384 T6575 C1033 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=details} status=0 QTime=0 
   [junit4]   2> 2465689 T6569 C1033 oasc.SolrCore.ex

[...truncated too long message...]

essionScavenger-196, state=TIMED_WAITING, group=TGRP-TestReplicationHandlerBackup]
   [junit4]   2> 	  10) Thread[id=6572, name=qtp3349864-6572, state=TIMED_WAITING, group=TGRP-TestReplicationHandlerBackup]
   [junit4]   2> 	  11) Thread[id=6577, name=searcherExecutor-4131-thread-1, state=WAITING, group=TGRP-TestReplicationHandlerBackup]
   [junit4]   2> 	  12) Thread[id=6569, name=qtp3349864-6569, state=TIMED_WAITING, group=TGRP-TestReplicationHandlerBackup]
   [junit4]   2> 9658715 T6566 oas.SolrTestCaseJ4.tearDown ###Ending doTestBackup
   [junit4]   2> 9658729 T6566 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=8705765
   [junit4]   2> 9658730 T6566 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@13dcfda
   [junit4]   2> 9658730 T6566 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=66,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 9658730 T6566 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 9658730 T6566 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 9658730 T6566 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 9658735 T6566 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 9658735 T6566 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 9658735 T6566 oasc.CachingDirectoryFactory.closeCacheValue looking to close C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index [CachedDir<<refCount=0;path=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index;done=false>>]
   [junit4]   2> 9658735 T6566 oasc.CachingDirectoryFactory.close Closing directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\index
   [junit4]   2> 9658735 T6566 oasc.CachingDirectoryFactory.closeCacheValue looking to close C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\ [CachedDir<<refCount=0;path=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\;done=false>>]
   [junit4]   2> 9658737 T6566 oasc.CachingDirectoryFactory.close Closing directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001\solr-instance-001\collection1\data\
   [junit4]   2> 9658737 T6566 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 9658769 T6566 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.handler.TestReplicationHandlerBackup-D9185B50208A695A-001
   [junit4]   2> 7289129 T6565 ccr.ThreadLeakControl.tryToInterruptAll All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Lucene410: {name=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=TestBloomFilteredLucene41Postings(BloomFilteringPostingsFormat(Lucene41(blocksize=128)))}, docValues:{}, sim=DefaultSimilarity, locale=ar_AE, timezone=Asia/Jayapura
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_20 (32-bit)/cpus=2,threads=1,free=197613448,total=313524224
   [junit4]   2> NOTE: All tests run in this JVM: [SchemaVersionSpecificBehaviorTest, CacheHeaderTest, TestMultiCoreConfBootstrap, TestSolrXml, SSLMigrationTest, TestReload, TestSolr4Spatial, TestCoreContainer, DocValuesTest, FullSolrCloudDistribCmdsTest, SpellPossibilityIteratorTest, TestFaceting, TestStressUserVersions, TestSolrDeletionPolicy1, DisMaxRequestHandlerTest, AnalyticsMergeStrategyTest, TestFiltering, TestShortCircuitedRequests, RangeFacetTest, DateMathParserTest, SimplePostToolTest, TestFoldingMultitermQuery, SoftAutoCommitTest, AtomicUpdatesTest, SpellCheckCollatorTest, SolrIndexConfigTest, TestStressVersions, OutOfBoxZkACLAndCredentialsProvidersTest, HighlighterTest, TestNoOpRegenerator, TestManagedSchemaDynamicFieldResource, ParsingFieldUpdateProcessorsTest, CustomCollectionTest, AbstractAnalyticsStatsTest, TestQueryTypes, TestRandomDVFaceting, TestSolrIndexConfig, SearchHandlerTest, TestSchemaNameResource, TestRemoteStreaming, OverseerCollectionProcessorTest, TestMaxScoreQueryParser, SystemInfoHandlerTest, TestLFUCache, SignatureUpdateProcessorFactoryTest, TestSolrDeletionPolicy2, ZkNodePropsTest, NoCacheHeaderTest, FieldFacetTest, DeleteLastCustomShardedReplicaTest, IndexBasedSpellCheckerTest, TestFastLRUCache, TestReloadAndDeleteDocs, TestHighFrequencyDictionaryFactory, TestCoreDiscovery, UpdateRequestProcessorFactoryTest, RemoteQueryErrorTest, TestSolrCoreProperties, AsyncMigrateRouteKeyTest, NumericFieldsTest, HdfsWriteToMultipleCollectionsTest, PluginInfoTest, CircularListTest, QueryResultKeyTest, SortByFunctionTest, FieldAnalysisRequestHandlerTest, HdfsSyncSliceTest, TestCollapseQParserPlugin, PrimUtilsTest, DistributedSpellCheckComponentTest, StandardRequestHandlerTest, TestInfoStreamLogging, SyncSliceTest, TestRestManager, TestOmitPositions, BasicZkTest, TestReplicationHandler, DistributedQueryElevationComponentTest, BlockDirectoryTest, TermVectorComponentDistributedTest, ExternalFileFieldSortTest, BadCopyFieldTest, DistanceFunctionTest, DirectSolrConnectionTest, RegexBoostProcessorTest, RecoveryZkTest, TestImplicitCoreProperties, TestQuerySenderNoQuery, HdfsCollectionsAPIDistributedZkTest, TestDistributedMissingSort, TestClassNameShortening, TestAnalyzeInfixSuggestions, TestSuggestSpellingConverter, TestRTGBase, TestPostingsSolrHighlighter, TestCSVLoader, TestOverriddenPrefixQueryForCustomFieldType, LegacyHTMLStripCharFilterTest, TestLeaderElectionZkExpiry, TestDefaultSearchFieldResource, PreAnalyzedFieldTest, BJQParserTest, SolrCmdDistributorTest, ChaosMonkeySafeLeaderTest, DirectUpdateHandlerTest, TestReversedWildcardFilterFactory, CollectionsAPIDistributedZkTest, LeaderElectionTest, TestRecoveryHdfs, TestLMDirichletSimilarityFactory, TestCopyFieldCollectionResource, BasicDistributedZkTest, TestSolrJ, JsonLoaderTest, SharedFSAutoReplicaFailoverTest, TestLazyCores, ReturnFieldsTest, LukeRequestHandlerTest, TestModifyConfFiles, TestAddFieldRealTimeGet, DateRangeFieldTest, TestWriterPerf, PeerSyncTest, DocValuesMultiTest, OverseerRolesTest, TestFieldTypeResource, TestStressLucene, TestSolr4Spatial2, SolrCoreCheckLockOnStartupTest, TestDefaultSimilarityFactory, OverseerStatusTest, CursorMarkTest, ReplicationFactorTest, MultiThreadedOCPTest, TestFastOutputStream, FastVectorHighlighterTest, HttpPartitionTest, PrimitiveFieldTypeTest, AnalyticsQueryTest, WordBreakSolrSpellCheckerTest, TestComplexPhraseQParserPlugin, ClusterStateTest, DistributedQueryComponentOptimizationTest, TestPhraseSuggestions, DistributedSuggestComponentTest, ScriptEngineTest, TestValueSourceCache, HardAutoCommitTest, TestNRTOpen, DistributedDebugComponentTest, BadComponentTest, ChangedSchemaMergeTest, TestIBSimilarityFactory, TestReplicationHandlerBackup]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandlerBackup -Dtests.seed=D9185B50208A695A -Dtests.slow=true -Dtests.locale=ar_AE -Dtests.timezone=Asia/Jayapura -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | TestReplicationHandlerBackup (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D9185B50208A695A]:0)
   [junit4] Completed in 7289.15s, 1 test, 2 errors <<< FAILURES!

[...truncated 880 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 437 suites, 1829 tests, 1 suite-level error, 1 error, 54 ignored (32 assumptions)

Total time: 262 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_67) - Build # 4313 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4313/
Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:55043/rpswv/k, http://127.0.0.1:55019/rpswv/k, http://127.0.0.1:55034/rpswv/k]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55043/rpswv/k, http://127.0.0.1:55019/rpswv/k, http://127.0.0.1:55034/rpswv/k]
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 45 more


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:64255, http://127.0.0.1:64273, http://127.0.0.1:64282, http://127.0.0.1:64240, http://127.0.0.1:64264]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:64255, http://127.0.0.1:64273, http://127.0.0.1:64282, http://127.0.0.1:64240, http://127.0.0.1:64264]
	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:139)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:123)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:87)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard2/core_node4
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 47 more




Build Log:
[...truncated 10877 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\init-core-data-001
   [junit4]   2> 1226895 T2716 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1226895 T2716 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1226899 T2716 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1226900 T2716 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1226901 T2717 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1226981 T2716 oasc.ZkTestServer.run start zk server on port:64233
   [junit4]   2> 1226981 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1226985 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1226991 T2723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83fdac name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1226992 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1226992 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1226992 T2716 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1226999 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1227001 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1227003 T2725 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69416a name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1227004 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1227004 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1227004 T2716 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1227011 T2716 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1227015 T2716 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1227021 T2716 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1227026 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1227026 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1227032 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1227032 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1227038 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1227038 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1227043 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1227043 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1227048 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1227048 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1227054 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1227054 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1227059 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1227059 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1227064 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1227064 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1227068 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1227068 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1227074 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1227074 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1227080 T2716 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1227081 T2716 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1228507 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1228515 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64240
   [junit4]   2> 1228515 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1228515 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\'
   [junit4]   2> 1228545 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\solr.xml
   [junit4]   2> 1228585 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1228585 T2716 oasc.CoreContainer.<init> New CoreContainer 33362955
   [junit4]   2> 1228585 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\]
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1228587 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1228590 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1228590 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1228590 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1228590 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1228590 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1228590 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1228590 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1228593 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1228594 T2736 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f1383 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1228595 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1228595 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1228599 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1228600 T2738 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fdabab name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1228600 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1228602 T2716 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1228607 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1228611 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1228614 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1228619 T2716 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1228624 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1228627 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64240_
   [junit4]   2> 1228628 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64240_
   [junit4]   2> 1228633 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1228636 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1228639 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1228642 T2716 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64240_
   [junit4]   2> 1228642 T2716 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1228645 T2716 oasc.Overseer.start Overseer (id=92456901443452931-127.0.0.1:64240_-n_0000000000) starting
   [junit4]   2> 1228649 T2716 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1228663 T2716 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1228665 T2740 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1228666 T2716 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1228669 T2716 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1228672 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1228679 T2739 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1228681 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002
   [junit4]   2> 1229263 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\conf\
   [junit4]   2> 1229364 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1229366 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1229366 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1229368 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1229370 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1229372 T2744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d1035 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1229372 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1229373 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1229375 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1229382 T2716 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1229382 T2716 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1229382 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1229384 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1229385 T2746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8635b1 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1229386 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1229386 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1229391 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1229396 T2739 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1229402 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1229406 T2747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1229412 T2748 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1230791 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1230791 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1230795 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64255
   [junit4]   2> 1230797 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1230797 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1230797 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1230798 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\'
   [junit4]   2> 1230826 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\solr.xml
   [junit4]   2> 1230881 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1230881 T2716 oasc.CoreContainer.<init> New CoreContainer 22393952
   [junit4]   2> 1230882 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\]
   [junit4]   2> 1230883 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1230883 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1230884 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1230885 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1230885 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1230886 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1230886 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1230886 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1230887 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1230887 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1230887 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1230887 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1230890 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1230892 T2759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b520da name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1230893 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1230894 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1230900 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1230902 T2761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bda455 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1230902 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1230911 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1231713 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64255_
   [junit4]   2> 1231714 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64255_
   [junit4]   2> 1231722 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1231728 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003
   [junit4]   2> 1232714 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-003\conf\
   [junit4]   2> 1232897 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1232899 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1232899 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1234253 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1234254 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1234259 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64264
   [junit4]   2> 1234259 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1234261 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\'
   [junit4]   2> 1234291 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\solr.xml
   [junit4]   2> 1234331 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1234331 T2716 oasc.CoreContainer.<init> New CoreContainer 13801278
   [junit4]   2> 1234332 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\]
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1234334 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1234335 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1234335 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1234335 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1234335 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1234335 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1234337 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1234337 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1234337 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1234339 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1234340 T2774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c78390 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1234341 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1234342 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1234344 T2718 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148790fdf920008, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1234345 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1234346 T2776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130e843 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1234347 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1234355 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1235227 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64264_
   [junit4]   2> 1235230 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64264_
   [junit4]   2> 1235242 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1235249 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004
   [junit4]   2> 1235904 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\conf\
   [junit4]   2> 1236010 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1236012 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1236012 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1237639 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1237640 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1237649 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64273
   [junit4]   2> 1237650 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1237650 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\'
   [junit4]   2> 1237685 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\solr.xml
   [junit4]   2> 1237732 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1237733 T2716 oasc.CoreContainer.<init> New CoreContainer 12121304
   [junit4]   2> 1237733 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\]
   [junit4]   2> 1237734 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1237734 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1237735 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1237736 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1237736 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1237737 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1237738 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1237738 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1237738 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1237738 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1237739 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1237739 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1237739 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1237740 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1237744 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1237749 T2789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2b948 name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1237750 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1237750 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1237756 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1237758 T2791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f2539e name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1237758 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1237773 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1238656 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64273_
   [junit4]   2> 1238658 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64273_
   [junit4]   2> 1238672 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1238679 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005
   [junit4]   2> 1239291 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\conf\
   [junit4]   2> 1239414 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1239415 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1239415 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1240998 T2716 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1240998 T2716 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1241004 T2716 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:64282
   [junit4]   2> 1241005 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1241005 T2716 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1241005 T2716 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1241006 T2716 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\'
   [junit4]   2> 1241035 T2716 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\solr.xml
   [junit4]   2> 1241108 T2716 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1241108 T2716 oasc.CoreContainer.<init> New CoreContainer 4794936
   [junit4]   2> 1241108 T2716 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\]
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1241112 T2716 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1241114 T2716 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1241114 T2716 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1241114 T2716 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1241116 T2716 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1241116 T2716 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1241116 T2716 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1241118 T2716 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64233/solr
   [junit4]   2> 1241118 T2716 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1241118 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1241121 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241123 T2804 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@172eaaf name:ZooKeeperConnection Watcher:127.0.0.1:64233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241124 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241125 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1241128 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1241130 T2806 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1582550 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1241131 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1241143 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1242026 T2716 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64282_
   [junit4]   2> 1242028 T2716 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64282_
   [junit4]   2> 1242042 T2716 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1242051 T2716 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006
   [junit4]   2> 1243114 T2716 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\conf\
   [junit4]   2> 1243304 T2716 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1243307 T2716 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 1243307 T2716 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1243312 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1243313 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243314 T2810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1564a75 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243315 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243315 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1243318 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1243325 T2716 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
   [junit4]   2> 1243325 T2716 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1243327 T2716 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1243328 T2812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b88624 name:ZooKeeperConnection Watcher:127.0.0.1:64233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1243328 T2716 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1243329 T2716 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1243331 T2716 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1243344 T2733 oasha.CollectionsHandler.handleCreateAction Creating Collection : stateFormat=2&action=CREATE&version=2&maxShardsPerNode=2&wt=javabin&name=delLiveColl&replicationFactor=2&numShards=2
   [junit4]   2> 1243347 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243349 T2740 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1243351 T2813 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 1243355 T2813 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1243355 T2813 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl 
   [junit4]   2> 1243356 T2813 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 1243364 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243372 T2739 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delLiveColl
   [junit4]   2> 1243373 T2739 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delLiveColl with shards [shard1, shard2]
   [junit4]   2> 1243373 T2739 oasc.Overseer$ClusterStateUpdater.createCollection state version delLiveColl 2
   [junit4]   2> 1243377 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243380 T2748 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243381 T2747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243382 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2762 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243383 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243384 T2815 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243384 T2814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1243461 T2813 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1243461 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:64282_
   [junit4]   2> 1243463 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:64273_
   [junit4]   2> 1243465 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:64240_
   [junit4]   2> 1243468 T2800 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard1&collection.configName=conf1&wt=javabin&name=delLiveColl_shard1_replica1&collection=delLiveColl&numShards=2
   [junit4]   2> 1243468 T2784 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard1&collection.configName=conf1&wt=javabin&name=delLiveColl_shard1_replica2&collection=delLiveColl&numShards=2
   [junit4]   2> 1243471 T2813 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:64264_
   [junit4]   2> 1243475 T2768 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard2&collection.configName=conf1&wt=javabin&name=delLiveColl_shard2_replica2&collection=delLiveColl&numShards=2
   [junit4]   2> 1243481 T2734 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&action=CREATE&version=2&shard=shard2&collection.configName=conf1&wt=javabin&name=delLiveColl_shard2_replica1&collection=delLiveColl&numShards=2
   [junit4]   2> 1243482 T2784 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
   [junit4]   2> 1243483 T2800 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
   [junit4]   2> 1243485 T2768 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
   [junit4]   2> 1243487 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243488 T2734 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
   [junit4]   2> 1243494 T2784 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243494 T2784 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243494 T2800 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243494 T2800 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243496 T2784 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243497 T2768 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243497 T2768 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243498 T2800 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243500 T2768 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243504 T2784 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243504 T2784 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243505 T2734 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 1243507 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64273",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64273_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243507 T2800 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243508 T2800 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243509 T2768 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 1243509 T2768 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243526 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243527 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243529 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243529 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243530 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243535 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243535 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243537 T2734 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 1243538 T2734 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 1243539 T2734 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243539 T2734 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1243550 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 1243551 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64282",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64282_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243554 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243555 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243556 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243556 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243558 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243558 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243559 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243559 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243560 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243563 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243571 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 1243572 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64264",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64264_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243575 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243576 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243578 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243578 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243579 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243580 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243579 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243580 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243581 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243584 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243591 T2739 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 3 
   [junit4]   2> 1243591 T2739 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64240",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "node_name":"127.0.0.1:64240_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 1243595 T2747 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1243595 T2739 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/delLiveColl/state.json
   [junit4]   2> 1243597 T2792 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243597 T2777 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243598 T2807 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243599 T2747 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 1243599 T2792 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243599 T2777 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243600 T2807 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1243603 T2747 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 4 
   [junit4]   2> 1244465 T2784 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 1244465 T2784 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244467 T2784 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244467 T2784 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244469 T2800 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 1244469 T2800 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244470 T2784 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244470 T2784 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\'
   [junit4]   2> 1244470 T2768 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 1244471 T2768 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244471 T2800 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244472 T2800 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244474 T2768 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244474 T2768 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244476 T2800 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244476 T2800 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-006\delLiveColl_shard1_replica1\'
   [junit4]   2> 1244479 T2768 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244479 T2768 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-004\delLiveColl_shard2_replica2\'
   [junit4]   2> 1244503 T2734 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 1244503 T2734 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 1244527 T2734 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1244527 T2734 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 1244534 T2734 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1244534 T2734 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-002\delLiveColl_shard2_replica1\'
   [junit4]   2> 1244596 T2784 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244597 T2800 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244642 T2768 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244669 T2800 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244701 T2768 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244703 T2800 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244711 T2784 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244715 T2800 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 1244719 T2768 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244729 T2768 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 1244745 T2734 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1244804 T2734 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1244805 T2784 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1244815 T2784 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 1245175 T2734 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1245187 T2734 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 1245934 T2784 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1245939 T2800 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1245949 T2784 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1245954 T2800 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1245958 T2784 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1245964 T2800 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1246072 T2784 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1246083 T2784 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1246105 T2768 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1246112 T2768 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1246115 T2768 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1246124 T2784 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1246129 T2784 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1246136 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1246136 T2784 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1246137 T2784 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using configuration from collection delLiveColl
   [junit4]   2> 1246137 T2784 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1246138 T2784 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\
   [junit4]   2> 1246138 T2784 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10163f6
   [junit4]   2> 1246140 T2784 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\
   [junit4]   2> 1246141 T2784 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index/
   [junit4]   2> 1246141 T2784 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index' doesn't exist. Creating new index...
   [junit4]   2> 1246142 T2784 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.DeleteReplicaTest-40015FF1CB47EE66-001\tempDir-005\delLiveColl_shard1_replica2\data\index
   [junit4]   2> 1246143 T2784 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=14.45703125, floorSegmentMB=1.69140625, forceMergeDeletesPctAllowed=8.899551040512542, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3486525566445853
   [junit4]   2> 1246143 T2784 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@14d9ffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7eaddf),segFN=segments_1,generation=1}
   [junit4]   2> 1246143 T2784 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1246149 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1246149 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1246150 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1246151 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1246152 T2784 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1246152 T2784 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1246152 T2784 oasc.SolrCore.loadUpdateProcessorChains no upd

[...truncated too long message...]

unit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:55)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:315)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 4338042 T8472 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 4338062 T7724 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/be,null}
   [junit4]   2> 4344312 T7724 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 4344313 T7724 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55825 55825
   [junit4]   2> 4347137 T7725 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55825 55825
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=40015FF1CB47EE66 -Dtests.slow=true -Dtests.locale=zh -Dtests.timezone=Africa/Porto-Novo -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    803s | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([40015FF1CB47EE66:C1E7D1E9BC188E5A]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4347162 T7724 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.CollectionsAPIDistributedZkTest-40015FF1CB47EE66-001
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=108), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=108)), sim=DefaultSimilarity, locale=zh, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=117008136,total=264413184
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, TestBinaryResponseWriter, QueryElevationComponentTest, TestManagedStopFilterFactory, LukeRequestHandlerTest, NotRequiredUniqueKeyTest, BJQParserTest, SuggesterFSTTest, AssignTest, HdfsChaosMonkeySafeLeaderTest, LoggingHandlerTest, TestIndexSearcher, TestComponentsName, TestBM25SimilarityFactory, TestSolrXml, TestRangeQuery, HdfsCollectionsAPIDistributedZkTest, TestSolrXmlPersistor, SolrCoreCheckLockOnStartupTest, TestCodecSupport, SchemaVersionSpecificBehaviorTest, RequestHandlersTest, BadIndexSchemaTest, SimpleFacetsTest, QueryParsingTest, TestReplicationHandler, TestAnalyzeInfixSuggestions, TestNRTOpen, WordBreakSolrSpellCheckerTest, TestPivotHelperCode, TestRecoveryHdfs, TestLazyCores, DisMaxRequestHandlerTest, TestCSVLoader, SignatureUpdateProcessorFactoryTest, UniqFieldsUpdateProcessorFactoryTest, TestDocSet, SOLR749Test, StatelessScriptUpdateProcessorFactoryTest, TestCloudPivotFacet, TestJoin, DistributedIntervalFacetingTest, TestHighFrequencyDictionaryFactory, CurrencyFieldXmlFileTest, TestSurroundQueryParser, MergeStrategyTest, CoreAdminCreateDiscoverTest, FastVectorHighlighterTest, TestRandomDVFaceting, QueryEqualityTest, TestUtils, TestFunctionQuery, XsltUpdateRequestHandlerTest, TestFoldingMultitermQuery, TestFastOutputStream, TestFiltering, HttpPartitionTest, ZkControllerTest, TestManagedSchemaDynamicFieldResource, ReplicationFactorTest, RollingRestartTest, TestGroupingSearch, CollectionsAPIAsyncDistributedZkTest, TestCollationField, TestShortCircuitedRequests, AnalyticsMergeStrategyTest, TestWriterPerf, TestFastWriter, CursorMarkTest, DebugComponentTest, TestCopyFieldCollectionResource, RangeFacetTest, TestMissingGroups, TimeZoneUtilsTest, DistributedSpellCheckComponentTest, SystemInfoHandlerTest, TestFieldCollectionResource, TestUpdate, AliasIntegrationTest, AbstractAnalyticsFacetTest, ConnectionManagerTest, TestConfig, TestQueryTypes, TestMergePolicyConfig, UUIDFieldTest, DeleteReplicaTest, HdfsBasicDistributedZk2Test, TestLRUCache, RegexBoostProcessorTest, TestDynamicFieldCollectionResource, TestInitQParser, TestStressReorder, ResponseHeaderTest, TestDocBasedVersionConstraints, TestJmxMonitoredMap, TestMaxScoreQueryParser, QueryResultKeyTest, BasicFunctionalityTest, CacheHeaderTest, SSLMigrationTest, HdfsWriteToMultipleCollectionsTest, TriLevelCompositeIdRoutingTest, MigrateRouteKeyTest, TestQueryUtils, MultiTermTest, TestDistributedSearch, TestCoreDiscovery, HdfsUnloadDistributedZkTest, ParsingFieldUpdateProcessorsTest, InfoHandlerTest, TestTolerantSearch, AnalysisErrorHandlingTest, TestBadConfig, IndexSchemaTest, PluginInfoTest, DocExpirationUpdateProcessorFactoryTest, TestReRankQParserPlugin, RankQueryTest, TestClassNameShortening, DateFieldTest, CustomCollectionTest, RecoveryZkTest, TestFaceting, TestPostingsSolrHighlighter, TestAddFieldRealTimeGet, PathHierarchyTokenizerFactoryTest, CoreMergeIndexesAdminHandlerTest, TestExceedMaxTermLength, PolyFieldTest, HighlighterTest, TestArbitraryIndexDir, DirectSolrConnectionTest, AbstractAnalyticsStatsTest, DocValuesMissingTest, SliceStateTest, SharedFSAutoReplicaFailoverUtilsTest, TestSimpleQParserPlugin, TestCloudSchemaless, TestCloudInspectUtil, TestFieldSortValues, TestRandomFaceting, TestRandomMergePolicy, SolrCoreTest, ChaosMonkeySafeLeaderTest, TestNoOpRegenerator, UnloadDistributedZkTest, BlockCacheTest, SynonymTokenizerTest, SolrPluginUtilsTest, HdfsDirectoryTest, SyncSliceTest, SearchHandlerTest, TestHighlightDedupGrouping, ResourceLoaderTest, TestManagedSynonymFilterFactory, TestDefaultSimilarityFactory, TestAnalyzedSuggestions, TestQuerySenderListener, TestReplicationHandlerBackup, BasicDistributedZkTest, TestFreeTextSuggestions, TestAtomicUpdateErrorCases, TestSolrXmlPersistence, TestParamSet, PrimUtilsTest, ModifyConfFileTest, SolrIndexSplitterTest, DOMUtilTest, DateMathParserTest, SuggesterTest, CircularListTest, LegacyHTMLStripCharFilterTest, SuggestComponentTest, TestMiniSolrCloudCluster, HighlighterConfigTest, TestStandardQParsers, SolrInfoMBeanTest, TestSchemaResource, TestValueSourceCache, TestSolrJ, TestChildDocTransformer, PreAnalyzedFieldTest, TestPseudoReturnFields, OutOfBoxZkACLAndCredentialsProvidersTest, TestManagedSchemaFieldResource, TestSolr4Spatial, DistanceFunctionTest, TestManagedResourceStorage, SpellingQueryConverterTest, SpellCheckCollatorTest, SpellPossibilityIteratorTest, PeerSyncTest, TestInfoStreamLogging, HdfsBasicDistributedZkTest, TestFileDictionaryLookup, LeaderElectionTest, SharedFSAutoReplicaFailoverTest, SoftAutoCommitTest, AtomicUpdatesTest, ZkCLITest, TestShardHandlerFactory, TestLuceneMatchVersion, TestDistributedMissingSort, ScriptEngineTest, DeleteLastCustomShardedReplicaTest, TestCollapseQParserPlugin, TestManagedSchemaFieldTypeResource, SpellCheckComponentTest, DeleteShardTest, OverseerTest, DistributedDebugComponentTest, HdfsLockFactoryTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed in 803.55s, 1 test, 1 error <<< FAILURES!

[...truncated 699 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 errors, 54 ignored (32 assumptions)

Total time: 181 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.7.0_67) - Build # 4312 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4312/
Java: 32bit/jdk1.7.0_67 -client -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6460, name=coreLoadExecutor-3852-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6460, name=coreLoadExecutor-3852-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@12e9f4d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13487fd)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6480, name=coreLoadExecutor-3859-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6480, name=coreLoadExecutor-3859-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@1801d44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cfd92)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestZkChroot

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6498, name=coreLoadExecutor-3866-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=6498, name=coreLoadExecutor-3866-thread-1, state=RUNNABLE, group=TGRP-TestZkChroot], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1589)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:731)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1274)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1165)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1157)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:40)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@274403 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10ba15b)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	... 1 more


FAILED:  org.apache.solr.core.CachingDirectoryFactoryTest.stressTest

Error Message:
Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)


FAILED:  org.apache.solr.core.RAMDirectoryFactoryTest.test

Error Message:
access denied ("java.io.FilePermission" "\fake\path" "write")

Stack Trace:
java.security.AccessControlException: access denied ("java.io.FilePermission" "\fake\path" "write")
	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD:5380019E9591BF05]:0)
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
	at java.nio.file.Files.createDirectory(Files.java:630)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
	at java.nio.file.Files.createDirectories(Files.java:683)
	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10866 lines...]
   [junit4] Suite: org.apache.solr.core.CachingDirectoryFactoryTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.CachingDirectoryFactoryTest-DBD43E443B6DD2FD-001\init-core-data-001
   [junit4]   2> 1790292 T3912 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1790296 T3912 oas.SolrTestCaseJ4.setUp ###Starting stressTest
   [junit4]   2> 1790662 T3913 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1ba49d1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9053a assuming 'simple'
   [junit4]   2> 1790671 T3917 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1de2296 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f8a6d assuming 'simple'
   [junit4]   2> 1790672 T3919 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@4939bc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1d63ff6 assuming 'simple'
   [junit4]   2> 1790673 T3922 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@cda7b3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1dc5f83 assuming 'simple'
   [junit4]   2> 1790673 T3918 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@1029ba4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19fc18f assuming 'simple'
   [junit4]   2> 1790674 T3916 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@15cc247 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb7183 assuming 'simple'
   [junit4]   2> 1790676 T3914 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@16971c7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10cd827 assuming 'simple'
   [junit4]   2> 1790677 T3920 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@3529b0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fd3d8 assuming 'simple'
   [junit4]   2> 385 T3917 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1569,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790679 T3915 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@139ba97 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@f1fbc assuming 'simple'
   [junit4]   2> 385 T3922 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1574,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3913 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1565,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3916 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1568,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790679 T3921 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@ecae3e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c2b49c assuming 'simple'
   [junit4]   2> 388 T3921 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1573,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1790680 T3923 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@160f31c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c710ea assuming 'simple'
   [junit4]   2> 386 T3919 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1571,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 386 T3920 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1572,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 390 T3923 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1575,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 388 T3914 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1566,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 387 T3918 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1570,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 387 T3915 ccr.RandomizedRunner$QueueUncaughtExceptionsHandler.uncaughtException WARNING Uncaught exception in thread: Thread[Thread-1567,5,TGRP-CachingDirectoryFactoryTest] java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]   2> 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]   2> 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]   2> 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]   2> 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]   2> 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]   2> 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]   2> 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]   2> 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]   2> 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]   2> 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 
   [junit4]   2> 1799517 T3930 oasc.CachingDirectoryFactory.close Closing RAMDirectoryFactory - 0 directories currently being tracked
   [junit4]   2> 1799603 T3912 oas.SolrTestCaseJ4.tearDown ###Ending stressTest
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CachingDirectoryFactoryTest -Dtests.method=stressTest -Dtests.seed=DBD43E443B6DD2FD -Dtests.slow=true -Dtests.locale=ar_SD -Dtests.timezone=America/Iqaluit -Dtests.file.encoding=Cp1252
   [junit4] ERROR   9.36s | CachingDirectoryFactoryTest.stressTest <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3913, name=Thread-1565, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3914, name=Thread-1566, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3916, name=Thread-1568, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3917, name=Thread-1569, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3918, name=Thread-1570, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3922, name=Thread-1574, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3920, name=Thread-1572, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3919, name=Thread-1571, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3915, name=Thread-1567, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3921, name=Thread-1573, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3923, name=Thread-1575, state=RUNNABLE, group=TGRP-CachingDirectoryFactoryTest]
   [junit4]    > Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "path1\8\12" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactoryTest$GetDirThread.run(CachingDirectoryFactoryTest.java:190)
   [junit4]   2> 1799653 T3912 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_SD, timezone=America/Iqaluit
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=185532568,total=373710848
   [junit4]   2> NOTE: All tests run in this JVM: [TestReplicationHandler, TestFreeTextSuggestions, SolrInfoMBeanTest, MinimalSchemaTest, URLClassifyProcessorTest, FunctionTest, TestArbitraryIndexDir, FieldFacetExtrasTest, TestSystemIdResolver, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, CurrencyFieldXmlFileTest, QueryParsingTest, TestRawResponseWriter, TestDynamicFieldCollectionResource, TestFastOutputStream, TestSolrXmlPersistence, IndexSchemaTest, BJQParserTest, UUIDFieldTest, TestDynamicFieldResource, TestCSVLoader, TestDistributedSearch, IndexBasedSpellCheckerTest, TestCloudManagedSchema, SolrTestCaseJ4Test, TestStressLucene, TestAnalyzedSuggestions, TestSolrDeletionPolicy1, SuggesterTSTTest, HdfsSyncSliceTest, TestSearcherReuse, NumericFieldsTest, TestOmitPositions, TestSolrXmlPersistor, TestLFUCache, TestDefaultSimilarityFactory, QueryFacetTest, TestFieldTypeResource, ShardSplitTest, RequestHandlersTest, TestManagedResourceStorage, TestFunctionQuery, AnalyticsQueryTest, DateFieldTest, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, SpellPossibilityIteratorTest, MigrateRouteKeyTest, BadComponentTest, JsonLoaderTest, MoreLikeThisHandlerTest, TestFastWriter, ClusterStateTest, AnalysisErrorHandlingTest, CursorPagingTest, TriLevelCompositeIdRoutingTest, ShardRoutingTest, ZkCLITest, SharedFSAutoReplicaFailoverUtilsTest, TestQuerySenderListener, TestCSVResponseWriter, TestNonNRTOpen, TestMultiCoreConfBootstrap, TestQueryTypes, StatelessScriptUpdateProcessorFactoryTest, BasicDistributedZkTest, TestStressVersions, TestTrie, TestFiltering, TestUpdate, TestModifyConfFiles, LeaderElectionIntegrationTest, ConvertedLegacyTest, SpellingQueryConverterTest, TestDocumentBuilder, AssignTest, TestJmxIntegration, TestStressUserVersions, TestFaceting, MBeansHandlerTest, TestRecoveryHdfs, TestPivotHelperCode, StandardRequestHandlerTest, OpenExchangeRatesOrgProviderTest, DisMaxRequestHandlerTest, TestCoreContainer, TestExtendedDismaxParser, TestValueSourceCache, TestJoin, EchoParamsTest, SortByFunctionTest, DocValuesMultiTest, TestShortCircuitedRequests, MergeStrategyTest, TestCoreDiscovery, TestLMDirichletSimilarityFactory, ExternalCollectionsTest, HardAutoCommitTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, ScriptEngineTest, TestMaxScoreQueryParser, CachingDirectoryFactoryTest]
   [junit4] Completed in 9.43s, 1 test, 1 error <<< FAILURES!

[...truncated 260 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestZkChroot
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestZkChroot-DBD43E443B6DD2FD-001\init-core-data-001
   [junit4]   2> 3106275 T6438 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3106284 T6438 oas.SolrTestCaseJ4.setUp ###Starting testNoBootstrapConf
   [junit4]   2> 3106286 T6438 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3106287 T6439 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3106374 T6438 oasc.ZkTestServer.run start zk server on port:52273
   [junit4]   2> 3106375 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3106376 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106380 T6445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64beb0 name:ZooKeeperConnection Watcher:127.0.0.1:52273 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106381 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106381 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3106382 T6438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\'
   [junit4]   2> 3106413 T6438 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\solr.xml
   [junit4]   2> 3106446 T6438 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3106447 T6438 oasc.CoreContainer.<init> New CoreContainer 6371933
   [junit4]   2> 3106447 T6438 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\]
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3106449 T6438 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3106450 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3106451 T6438 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3106451 T6438 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 3106451 T6438 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3106452 T6438 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3106452 T6438 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3106453 T6438 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52273/foo/bar2
   [junit4]   2> 3106453 T6438 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3106453 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3106455 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3106457 T6447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ed3c4 name:ZooKeeperConnection Watcher:127.0.0.1:52273 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3106458 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3106458 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3106462 T6440 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x148766a75440001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 3106465 T6438 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52273 52273
   [junit4]   2> 3108288 T6439 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52273 52273
   [junit4]   2> 3108289 T6438 oas.SolrTestCaseJ4.tearDown ###Ending testNoBootstrapConf
   [junit4]   2> 3108294 T6438 oas.SolrTestCaseJ4.setUp ###Starting testChrootBootstrap
   [junit4]   2> 3108296 T6438 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3108297 T6448 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3108375 T6438 oasc.ZkTestServer.run start zk server on port:52282
   [junit4]   2> 3108375 T6438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\'
   [junit4]   2> 3108394 T6438 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\solr.xml
   [junit4]   2> 3108426 T6438 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108426 T6438 oasc.CoreContainer.<init> New CoreContainer 13594553
   [junit4]   2> 3108426 T6438 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\]
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3108428 T6438 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3108428 T6438 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
   [junit4]   2> 3108430 T6438 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3108430 T6438 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3108430 T6438 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3108430 T6438 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52282/foo/bar
   [junit4]   2> 3108430 T6438 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3108430 T6438 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 3108433 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108436 T6454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b00f6a name:ZooKeeperConnection Watcher:127.0.0.1:52282 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108437 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108437 T6438 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 3108438 T6438 oascc.SolrZkClient.makePath makePath: /foo/bar
   [junit4]   2> 3108446 T6438 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3108447 T6456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1232469 name:ZooKeeperConnection Watcher:127.0.0.1:52282/foo/bar got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3108447 T6438 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3108449 T6438 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3108453 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3108457 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3108462 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3108465 T6438 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3108470 T6438 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3108472 T6438 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/192.168.57.16:8983_solr
   [junit4]   2> 3108473 T6438 oascc.SolrZkClient.makePath makePath: /live_nodes/192.168.57.16:8983_solr
   [junit4]   2> 3108476 T6438 oascc.SolrZkClient.makePath makePath: /collections
   [junit4]   2> 3108479 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3108481 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3108484 T6438 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 3108486 T6438 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 192.168.57.16:8983_solr
   [junit4]   2> 3108486 T6438 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3108488 T6438 oasc.Overseer.start Overseer (id=92453992098758657-192.168.57.16:8983_solr-n_0000000000) starting
   [junit4]   2> 3108492 T6438 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3108503 T6438 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3108505 T6458 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3108506 T6438 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3108510 T6438 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3108513 T6438 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3108516 T6438 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108518 T6457 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3108528 T6438 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\
   [junit4]   2> 3108528 T6438 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3108528 T6438 oasc.ZkController.bootstrapConf bootstrapping config for 1 cores into ZooKeeper using solr.xml from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\
   [junit4]   2> 3108528 T6438 oasc.ZkController.bootstrapConf Uploading directory C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\conf with name collection1 for SolrCore collection1
   [junit4]   2> 3108531 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.html
   [junit4]   2> 3108539 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-bottom.html
   [junit4]   2> 3108543 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/admin-extra.menu-top.html
   [junit4]   2> 3108550 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/kmeans-attributes.xml
   [junit4]   2> 3108557 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/lingo-attributes.xml
   [junit4]   2> 3108563 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/clustering/carrot2/stc-attributes.xml
   [junit4]   2> 3108568 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/currency.xml
   [junit4]   2> 3108572 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/elevate.xml
   [junit4]   2> 3108576 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ca.txt
   [junit4]   2> 3108581 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_fr.txt
   [junit4]   2> 3108586 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_ga.txt
   [junit4]   2> 3108590 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/contractions_it.txt
   [junit4]   2> 3108595 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/hyphenations_ga.txt
   [junit4]   2> 3108598 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stemdict_nl.txt
   [junit4]   2> 3108602 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stoptags_ja.txt
   [junit4]   2> 3108607 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ar.txt
   [junit4]   2> 3108611 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_bg.txt
   [junit4]   2> 3108616 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ca.txt
   [junit4]   2> 3108620 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ckb.txt
   [junit4]   2> 3108626 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_cz.txt
   [junit4]   2> 3108630 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_da.txt
   [junit4]   2> 3108634 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_de.txt
   [junit4]   2> 3108637 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_el.txt
   [junit4]   2> 3108642 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_en.txt
   [junit4]   2> 3108646 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_es.txt
   [junit4]   2> 3108650 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_eu.txt
   [junit4]   2> 3108655 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fa.txt
   [junit4]   2> 3108658 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fi.txt
   [junit4]   2> 3108662 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_fr.txt
   [junit4]   2> 3108668 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ga.txt
   [junit4]   2> 3108671 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_gl.txt
   [junit4]   2> 3108675 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hi.txt
   [junit4]   2> 3108680 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hu.txt
   [junit4]   2> 3108684 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_hy.txt
   [junit4]   2> 3108689 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_id.txt
   [junit4]   2> 3108693 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_it.txt
   [junit4]   2> 3108698 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ja.txt
   [junit4]   2> 3108703 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_lv.txt
   [junit4]   2> 3108707 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_nl.txt
   [junit4]   2> 3108712 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_no.txt
   [junit4]   2> 3108716 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_pt.txt
   [junit4]   2> 3108720 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ro.txt
   [junit4]   2> 3108725 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_ru.txt
   [junit4]   2> 3108730 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_sv.txt
   [junit4]   2> 3108734 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_th.txt
   [junit4]   2> 3108739 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/stopwords_tr.txt
   [junit4]   2> 3108743 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/lang/userdict_ja.txt
   [junit4]   2> 3108753 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-FoldToASCII.txt
   [junit4]   2> 3108759 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3108762 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/protwords.txt
   [junit4]   2> 3108767 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/schema.xml
   [junit4]   2> 3108771 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/scripts.conf
   [junit4]   2> 3108775 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/solrconfig.xml
   [junit4]   2> 3108780 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/spellings.txt
   [junit4]   2> 3108783 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/stopwords.txt
   [junit4]   2> 3108787 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/synonyms.txt
   [junit4]   2> 3108791 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/update-script.js
   [junit4]   2> 3108795 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/browse.vm
   [junit4]   2> 3108800 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster.vm
   [junit4]   2> 3108804 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/cluster_results.vm
   [junit4]   2> 3108808 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/debug.vm
   [junit4]   2> 3108811 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/did_you_mean.vm
   [junit4]   2> 3108816 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/error.vm
   [junit4]   2> 3108820 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facets.vm
   [junit4]   2> 3108824 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_fields.vm
   [junit4]   2> 3108829 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_pivot.vm
   [junit4]   2> 3108833 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_queries.vm
   [junit4]   2> 3108838 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/facet_ranges.vm
   [junit4]   2> 3108842 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/footer.vm
   [junit4]   2> 3108847 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/head.vm
   [junit4]   2> 3108851 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/header.vm
   [junit4]   2> 3108856 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit.vm
   [junit4]   2> 3108860 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_grouped.vm
   [junit4]   2> 3108865 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/hit_plain.vm
   [junit4]   2> 3108870 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/join_doc.vm
   [junit4]   2> 3108873 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.css
   [junit4]   2> 3108879 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/jquery.autocomplete.js
   [junit4]   2> 3108883 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/layout.vm
   [junit4]   2> 3108887 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/main.css
   [junit4]   2> 3108892 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/mime_type_lists.vm
   [junit4]   2> 3108895 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_bottom.vm
   [junit4]   2> 3108900 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/pagination_top.vm
   [junit4]   2> 3108904 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/product_doc.vm
   [junit4]   2> 3108909 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query.vm
   [junit4]   2> 3108913 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_form.vm
   [junit4]   2> 3108917 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_group.vm
   [junit4]   2> 3108920 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/query_spatial.vm
   [junit4]   2> 3108925 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/README.txt
   [junit4]   2> 3108930 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/results_list.vm
   [junit4]   2> 3108934 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/richtext_doc.vm
   [junit4]   2> 3108939 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/suggest.vm
   [junit4]   2> 3108943 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/tabs.vm
   [junit4]   2> 3108949 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/velocity/VM_global_library.vm
   [junit4]   2> 3108954 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example.xsl
   [junit4]   2> 3108960 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_atom.xsl
   [junit4]   2> 3108963 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/example_rss.xsl
   [junit4]   2> 3108967 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/luke.xsl
   [junit4]   2> 3108972 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/xslt/updateXml.xsl
   [junit4]   2> 3108976 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_rest_managed.json
   [junit4]   2> 3108980 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_stopwords_english.json
   [junit4]   2> 3108984 T6438 oascc.SolrZkClient.makePath makePath: /configs/collection1/_schema_analysis_synonyms_english.json
   [junit4]   2> 3108993 T6438 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr
   [junit4]   2> 3108996 T6438 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\
   [junit4]   2> 3108996 T6438 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 3108998 T6460 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3108998 T6460 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3108999 T6460 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3108999 T6461 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3109003 T6457 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://192.168.57.16:8983/solr",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"192.168.57.16:8983_solr"}
   [junit4]   2> 3109003 T6457 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3109008 T6461 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3109777 T6460 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3109777 T6460 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3109779 T6460 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:collection1
   [junit4]   2> 3109779 T6460 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3109782 T6460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3109783 T6460 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=collection1 specified config exists in ZooKeeper
   [junit4]   2> 3109783 T6460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\'
   [junit4]   2> 3109811 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not close prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=6371933
   [junit4]   2> 3109844 T6460 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 3109846 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/extraction/lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\extraction\lib).
   [junit4]   2> 3109846 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/clustering/lib/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\clustering\lib).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/langid/lib/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\langid\lib).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109849 T6460 oasc.SolrResourceLoader.addToClassLoader WARN No files added to classloader from lib: ../../../contrib/velocity/lib (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\contrib\velocity\lib).
   [junit4]   2> 3109851 T6460 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\example\solr\collection1\..\..\..\dist).
   [junit4]   2> 3109885 T6460 oasu.SolrIndexConfig.<init> IndexWriter infoStream solr logging is enabled
   [junit4]   2> 3109890 T6460 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 3109963 T6460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3109966 T6460 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3109976 T6460 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 3110433 T6460 o

[...truncated too long message...]

.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:276)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:488)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:794)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:652)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(RAMDirectory@274403 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10ba15b)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183)
   [junit4]    > 	... 1 more
   [junit4] Completed in 14.64s, 4 tests, 3 errors <<< FAILURES!

[...truncated 13 lines...]
   [junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
   [junit4]   2> 3265593 T6986 oasc.CachingDirectoryFactory.injectLockFactory WARN No lockType configured for RAMDirectory@14b73d2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@167ece2 assuming 'simple'
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RAMDirectoryFactoryTest -Dtests.method=test -Dtests.seed=DBD43E443B6DD2FD -Dtests.slow=true -Dtests.locale=fr_CA -Dtests.timezone=Africa/Bissau -Dtests.file.encoding=Cp1252
   [junit4] ERROR   0.01s | RAMDirectoryFactoryTest.test <<<
   [junit4]    > Throwable #1: java.security.AccessControlException: access denied ("java.io.FilePermission" "\fake\path" "write")
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DBD43E443B6DD2FD:5380019E9591BF05]:0)
   [junit4]    > 	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:559)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]    > 	at java.lang.SecurityManager.checkWrite(SecurityManager.java:979)
   [junit4]    > 	at sun.nio.fs.WindowsPath.checkWrite(WindowsPath.java:799)
   [junit4]    > 	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:491)
   [junit4]    > 	at java.nio.file.Files.createDirectory(Files.java:630)
   [junit4]    > 	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
   [junit4]    > 	at java.nio.file.Files.createDirectories(Files.java:683)
   [junit4]    > 	at org.apache.lucene.store.FSLockFactory.setLockDir(FSLockFactory.java:46)
   [junit4]    > 	at org.apache.lucene.store.SimpleFSLockFactory.<init>(SimpleFSLockFactory.java:81)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.injectLockFactory(CachingDirectoryFactory.java:500)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:357)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.dotestOpenReturnsTheSameForSamePath(RAMDirectoryFactoryTest.java:46)
   [junit4]    > 	at org.apache.solr.core.RAMDirectoryFactoryTest.test(RAMDirectoryFactoryTest.java:33)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr_CA, timezone=Africa/Bissau
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.7.0_67 (32-bit)/cpus=2,threads=1,free=231459928,total=523501568
   [junit4]   2> NOTE: All tests run in this JVM: [TestReplicationHandler, TestFreeTextSuggestions, SolrInfoMBeanTest, MinimalSchemaTest, URLClassifyProcessorTest, FunctionTest, TestArbitraryIndexDir, FieldFacetExtrasTest, TestSystemIdResolver, TestCopyFieldCollectionResource, DocumentAnalysisRequestHandlerTest, CurrencyFieldXmlFileTest, QueryParsingTest, TestRawResponseWriter, TestDynamicFieldCollectionResource, TestFastOutputStream, TestSolrXmlPersistence, IndexSchemaTest, BJQParserTest, UUIDFieldTest, TestDynamicFieldResource, TestCSVLoader, TestDistributedSearch, IndexBasedSpellCheckerTest, TestCloudManagedSchema, SolrTestCaseJ4Test, TestStressLucene, TestAnalyzedSuggestions, TestSolrDeletionPolicy1, SuggesterTSTTest, HdfsSyncSliceTest, TestSearcherReuse, NumericFieldsTest, TestOmitPositions, TestSolrXmlPersistor, TestLFUCache, TestDefaultSimilarityFactory, QueryFacetTest, TestFieldTypeResource, ShardSplitTest, RequestHandlersTest, TestManagedResourceStorage, TestFunctionQuery, AnalyticsQueryTest, DateFieldTest, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, SpellPossibilityIteratorTest, MigrateRouteKeyTest, BadComponentTest, JsonLoaderTest, MoreLikeThisHandlerTest, TestFastWriter, ClusterStateTest, AnalysisErrorHandlingTest, CursorPagingTest, TriLevelCompositeIdRoutingTest, ShardRoutingTest, ZkCLITest, SharedFSAutoReplicaFailoverUtilsTest, TestQuerySenderListener, TestCSVResponseWriter, TestNonNRTOpen, TestMultiCoreConfBootstrap, TestQueryTypes, StatelessScriptUpdateProcessorFactoryTest, BasicDistributedZkTest, TestStressVersions, TestTrie, TestFiltering, TestUpdate, TestModifyConfFiles, LeaderElectionIntegrationTest, ConvertedLegacyTest, SpellingQueryConverterTest, TestDocumentBuilder, AssignTest, TestJmxIntegration, TestStressUserVersions, TestFaceting, MBeansHandlerTest, TestRecoveryHdfs, TestPivotHelperCode, StandardRequestHandlerTest, OpenExchangeRatesOrgProviderTest, DisMaxRequestHandlerTest, TestCoreContainer, TestExtendedDismaxParser, TestValueSourceCache, TestJoin, EchoParamsTest, SortByFunctionTest, DocValuesMultiTest, TestShortCircuitedRequests, MergeStrategyTest, TestCoreDiscovery, TestLMDirichletSimilarityFactory, ExternalCollectionsTest, HardAutoCommitTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, ScriptEngineTest, TestMaxScoreQueryParser, CachingDirectoryFactoryTest, RegexBoostProcessorTest, TestRandomFaceting, DeleteInactiveReplicaTest, TestStressRecovery, TestFastLRUCache, TestExceedMaxTermLength, DateRangeFieldTest, AddBlockUpdateTest, SSLMigrationTest, ResourceLoaderTest, EnumFieldTest, UUIDUpdateProcessorFallbackTest, FieldMutatingUpdateProcessorTest, HighlighterConfigTest, BasicZkTest, TestCharFilters, TestAtomicUpdateErrorCases, SuggesterTest, TestCloudPivotFacet, TestImplicitCoreProperties, TestOverriddenPrefixQueryForCustomFieldType, StatsComponentTest, TestDistributedGrouping, TestRTGBase, TestCollationFieldDocValues, TestFoldingMultitermQuery, PolyFieldTest, CopyFieldTest, DocValuesMissingTest, TestOrdValues, TestQueryUtils, TestRangeQuery, TestParamSet, TestSolr4Spatial2, OverriddenZkACLAndCredentialsProvidersTest, DistribCursorPagingTest, BadIndexSchemaTest, AtomicUpdatesTest, DeleteReplicaTest, SearchHandlerTest, ReplicationFactorTest, CacheHeaderTest, ShowFileRequestHandlerTest, WordBreakSolrSpellCheckerTest, HdfsLockFactoryTest, BasicFunctionalityTest, TestSolrXml, FileBasedSpellCheckerTest, ChangedSchemaMergeTest, TestPerFieldSimilarity, TestNoOpRegenerator, TestLazyCores, AliasIntegrationTest, TestHighFrequencyDictionaryFactory, SimpleFacetsTest, AbstractAnalyticsStatsTest, TestIndexingPerformance, SolrCoreTest, TestSolrCoreProperties, TestBinaryField, TestBinaryResponseWriter, TestStandardQParsers, AutoCommitTest, CoreMergeIndexesAdminHandlerTest, TestCloudSchemaless, TestHashPartitioner, MultiThreadedOCPTest, SuggesterWFSTTest, NoFacetTest, DefaultValueUpdateProcessorTest, HdfsBasicDistributedZkTest, CoreAdminCreateDiscoverTest, TestSolrDeletionPolicy2, CoreAdminRequestStatusTest, FileUtilsTest, TestReloadAndDeleteDocs, CursorMarkTest, NotRequiredUniqueKeyTest, TestTolerantSearch, TestSearchPerf, TestZkChroot, TestSolrQueryParser, DistributedSpellCheckComponentTest, OverseerTest, ExpressionTest, RAMDirectoryFactoryTest]
   [junit4] Completed in 0.03s, 1 test, 1 error <<< FAILURES!

[...truncated 792 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 3 suite-level errors, 2 errors, 54 ignored (32 assumptions)

Total time: 173 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_20) - Build # 4311 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4311/
Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTasksParse

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001

	at __randomizedtesting.SeedInfo.seed([EBB0DBD1FB2CCFE9]:0)
	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:126)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 7125 lines...]
   [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> Spatial Strategy: RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:SpatialContext{geo=true, calculator=Haversine, worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ro_RO, timezone=Europe/Minsk
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_20 (32-bit)/cpus=2,threads=1,free=21303776,total=32571392
   [junit4]   2> NOTE: All tests run in this JVM: [TestPerfTasksParse]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse -Dtests.seed=EBB0DBD1FB2CCFE9 -Dtests.slow=true -Dtests.locale=ro_RO -Dtests.timezone=Europe/Minsk -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
   [junit4]    > Throwable #1: java.io.IOException: Could not remove the following files (in the order of attempts):
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-001.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001\linefile-002.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-EBB0DBD1FB2CCFE9-001
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EBB0DBD1FB2CCFE9]:0)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed in 3.21s, 3 tests, 1 error <<< FAILURES!

[...truncated 60 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:462: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2140: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 18 suites, 93 tests, 1 suite-level error

Total time: 30 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Robert Muir <rc...@gmail.com>.
thank you for debugging this! I didn't know about this crazy finalizer
at all: I tried to figure it out last night, but i could not see the
problem and without a windows machine it was hopeless for me.

On Sun, Sep 14, 2014 at 7:29 AM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi,
>
> I know why it fails consequently with the new Java 7 APIs:
> FileOutputStream since version 1.0 of Java has a custom finalizer() method, so the file gets closed on GC, kept until now for backwards compatibility. But if you the stream with Java 7's NIO.2, the created OutputStream instance (some non-public internal platform specific impl class) no longer has a finalizer(), so the file handle is never closed when the OutputStream is GCed. This is great (f*ck finalizers)!
>
> So by using NIO.2 we have a very nice "detector" for unclosed file, if people do broken stuff like "new Properties(new FileInputStream(...))" as seen in Solr quite often!
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Sunday, September 14, 2014 1:16 PM
>> To: dev@lucene.apache.org
>> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> Build # 4310 - Still Failing!
>>
>> The problem here is incomplete cleanup:
>>
>> This test parses the whole algorithm, it then creates an Algorithm instance
>> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated,
>> which in fact opens the output stream for the line docs file in its ctor. But as
>> the tasks are never executed, so the Task's close() is never called.
>>
>> I fixed the whole thing by making Algorithm implement AutoCloseable (not
>> Closeable, because it throws not only IOException). I also backported the
>> whole stuff to Lucene 4, because this is unrelated to Robert's changes.
>> Roberts changes just triggered other code execution paths, so GC did not
>> clean up algorithm.
>>
>> TODO: verify other uses of Algorithm and check that they are correctly
>> closed. In fact I have seen this bug sometimes when running tests locally, so
>> this was existent since long time, just in most cases GC was faster!
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> > Sent: Sunday, September 14, 2014 12:21 PM
>> > To: dev@lucene.apache.org
>> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> > Build # 4310 - Still Failing!
>> >
>> > This fails reproducible.
>> >
>> > I will dig!
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> > > -----Original Message-----
>> > > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
>> > > Sent: Sunday, September 14, 2014 2:48 AM
>> > > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
>> > > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
>> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
>> > > Build #
>> > > 4310 - Still Failing!
>> > >
>> > > Build:
>> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
>> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
>> > > XX:+UseConcMarkSweepGC
>> > >
>> > > 1 tests failed.
>> > > FAILED:
>> > >
>> >
>> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
>> > > a
>> > > sksParse
>> > >
>> > > Error Message:
>> > > Could not remove the following files (in the order of attempts):
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >
>> > > Stack Trace:
>> > > java.io.IOException: Could not remove the following files (in the
>> > > order of
>> > > attempts):
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >
>> > >   at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>> > >   at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>> > >   at
>> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes
>> > > tR
>> > > ul
>> > > eTemporaryFilesCleanup.java:126)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa
>> > > ys
>> > > (T
>> > > estRuleAdapter.java:31)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:43)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:36)
>> > >   at
>> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul
>> > > eA
>> > > ss
>> > > ertionsRequired.java:43)
>> > >   at
>> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa
>> > > il
>> > > ure
>> > > .java:48)
>> > >   at
>> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes
>> > > tR
>> > > ule
>> > > IgnoreAfterMaxFailures.java:65)
>> > >   at
>> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI
>> > > gn
>> > > ore
>> > > TestSuites.java:55)
>> > >   at
>> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
>> > > ta
>> > > t
>> > > ementAdapter.java:36)
>> > >   at
>> > >
>> >
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
>> > > run(ThreadLeakControl.java:365)
>> > >   at java.lang.Thread.run(Thread.java:745)
>> > >
>> > >
>> > >
>> > >
>> > > Build Log:
>> > > [...truncated 7622 lines...]
>> > >    [junit4] Suite:
>> org.apache.lucene.benchmark.byTask.TestPerfTasksParse
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> Spatial Strategy:
>> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa
>> > > ti al Context{geo=true, calculator=Haversine,
>> > > worldBounds=Rect(minX=-
>> > > 180.0,maxX=180.0,minY=-
>> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   1> ------------> queries:
>> > >    [junit4]   1>
>> > >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
>> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
>> > >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67
>> (64-
>> > > bit)/cpus=2,threads=1,free=120281832,total=140083200
>> > >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
>> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
>> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
>> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
>> > >    [junit4]   2> NOTE: reproduce with: ant test  -
>> > Dtestcase=TestPerfTasksParse
>> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL
>> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
>> > >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
>> > >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
>> > > following files (in the order of attempts):
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
>> > > java.nio.file.AccessDeniedException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001:
>> > > java.nio.file.DirectoryNotEmptyException:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > >
>> >
>> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
>> > > .TestPerfTasksParse-2D34950549928D3F-001
>> > >    [junit4]    >  at
>> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>> > >    [junit4]    >  at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>> > >    [junit4]    >  at java.lang.Thread.run(Thread.java:745)
>> > >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
>> > >
>> > > [...truncated 33 lines...]
>> > > BUILD FAILED
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\build.xml:491: The following error occurred while executing
>> > > this
>> > > line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\build.xml:471: The following error occurred while executing
>> > > this
>> > > line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > Windows\build.xml:60:
>> > > The following error occurred while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
>> > > targets.xml:39: The following error occurred while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\build.xml:462: The following error occurred while
>> > > executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:2140: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\module-build.xml:58: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:1358: The following error occurred
>> > > while executing this line:
>> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
>> > > Windows\lucene\common-build.xml:961: There were test failures: 18
>> > > suites,
>> > > 93 tests, 1 suite-level error
>> > >
>> > > Total time: 31 minutes 50 seconds
>> > > Build step 'Invoke Ant' marked build as failure [description-setter]
>> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
>> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
>> > > Email was triggered for: Failure - Any Sending email for trigger:
>> > > Failure - Any
>> > >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> > additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,

I know why it fails consequently with the new Java 7 APIs:
FileOutputStream since version 1.0 of Java has a custom finalizer() method, so the file gets closed on GC, kept until now for backwards compatibility. But if you the stream with Java 7's NIO.2, the created OutputStream instance (some non-public internal platform specific impl class) no longer has a finalizer(), so the file handle is never closed when the OutputStream is GCed. This is great (f*ck finalizers)!

So by using NIO.2 we have a very nice "detector" for unclosed file, if people do broken stuff like "new Properties(new FileInputStream(...))" as seen in Solr quite often!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, September 14, 2014 1:16 PM
> To: dev@lucene.apache.org
> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> Build # 4310 - Still Failing!
> 
> The problem here is incomplete cleanup:
> 
> This test parses the whole algorithm, it then creates an Algorithm instance
> containing all Tasks to execute. The "WriteLineDocsTask" is instantiated,
> which in fact opens the output stream for the line docs file in its ctor. But as
> the tasks are never executed, so the Task's close() is never called.
> 
> I fixed the whole thing by making Algorithm implement AutoCloseable (not
> Closeable, because it throws not only IOException). I also backported the
> whole stuff to Lucene 4, because this is unrelated to Robert's changes.
> Roberts changes just triggered other code execution paths, so GC did not
> clean up algorithm.
> 
> TODO: verify other uses of Algorithm and check that they are correctly
> closed. In fact I have seen this bug sometimes when running tests locally, so
> this was existent since long time, just in most cases GC was faster!
> 
> Uwe
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> 
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Sunday, September 14, 2014 12:21 PM
> > To: dev@lucene.apache.org
> > Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > Build # 4310 - Still Failing!
> >
> > This fails reproducible.
> >
> > I will dig!
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> > > Sent: Sunday, September 14, 2014 2:48 AM
> > > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> > > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> > > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > > Build #
> > > 4310 - Still Failing!
> > >
> > > Build:
> > > http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> > > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC
> > >
> > > 1 tests failed.
> > > FAILED:
> > >
> >
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
> > > a
> > > sksParse
> > >
> > > Error Message:
> > > Could not remove the following files (in the order of attempts):
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > Stack Trace:
> > > java.io.IOException: Could not remove the following files (in the
> > > order of
> > > attempts):
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >
> > > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > > 	at
> > > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(Tes
> > > tR
> > > ul
> > > eTemporaryFilesCleanup.java:126)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlwa
> > > ys
> > > (T
> > > estRuleAdapter.java:31)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:43)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRul
> > > eA
> > > ss
> > > ertionsRequired.java:43)
> > > 	at
> > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFa
> > > il
> > > ure
> > > .java:48)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Tes
> > > tR
> > > ule
> > > IgnoreAfterMaxFailures.java:65)
> > > 	at
> > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleI
> > > gn
> > > ore
> > > TestSuites.java:55)
> > > 	at
> > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(S
> > > ta
> > > t
> > > ementAdapter.java:36)
> > > 	at
> > >
> >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > > run(ThreadLeakControl.java:365)
> > > 	at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > >
> > >
> > > Build Log:
> > > [...truncated 7622 lines...]
> > >    [junit4] Suite:
> org.apache.lucene.benchmark.byTask.TestPerfTasksParse
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> Spatial Strategy:
> > > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spa
> > > ti al Context{geo=true, calculator=Haversine,
> > > worldBounds=Rect(minX=-
> > > 180.0,maxX=180.0,minY=-
> > > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   1> ------------> queries:
> > >    [junit4]   1>
> > >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> > > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
> > >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67
> (64-
> > > bit)/cpus=2,threads=1,free=120281832,total=140083200
> > >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> > > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> > > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
> > > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
> > >    [junit4]   2> NOTE: reproduce with: ant test  -
> > Dtestcase=TestPerfTasksParse
> > > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL
> > > - Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
> > >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
> > >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> > > following files (in the order of attempts):
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > > java.nio.file.AccessDeniedException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001:
> > > java.nio.file.DirectoryNotEmptyException:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > >
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > > .TestPerfTasksParse-2D34950549928D3F-001
> > >    [junit4]    > 	at
> > > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > >    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > >    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
> > >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> > >
> > > [...truncated 33 lines...]
> > > BUILD FAILED
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:491: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\build.xml:471: The following error occurred while executing
> > > this
> > > line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:60:
> > > The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> > > targets.xml:39: The following error occurred while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\build.xml:462: The following error occurred while
> > > executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:2140: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\module-build.xml:58: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:1358: The following error occurred
> > > while executing this line:
> > > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > > Windows\lucene\common-build.xml:961: There were test failures: 18
> > > suites,
> > > 93 tests, 1 suite-level error
> > >
> > > Total time: 31 minutes 50 seconds
> > > Build step 'Invoke Ant' marked build as failure [description-setter]
> > > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
> > > Email was triggered for: Failure - Any Sending email for trigger:
> > > Failure - Any
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
The problem here is incomplete cleanup:

This test parses the whole algorithm, it then creates an Algorithm instance containing all Tasks to execute. The "WriteLineDocsTask" is instantiated, which in fact opens the output stream for the line docs file in its ctor. But as the tasks are never executed, so the Task's close() is never called.

I fixed the whole thing by making Algorithm implement AutoCloseable (not Closeable, because it throws not only IOException). I also backported the whole stuff to Lucene 4, because this is unrelated to Robert's changes. Roberts changes just triggered other code execution paths, so GC did not clean up algorithm.

TODO: verify other uses of Algorithm and check that they are correctly closed. In fact I have seen this bug sometimes when running tests locally, so this was existent since long time, just in most cases GC was faster!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, September 14, 2014 12:21 PM
> To: dev@lucene.apache.org
> Subject: RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> Build # 4310 - Still Failing!
> 
> This fails reproducible.
> 
> I will dig!
> 
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
> 
> > -----Original Message-----
> > From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> > Sent: Sunday, September 14, 2014 2:48 AM
> > To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> > uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> > Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) -
> > Build #
> > 4310 - Still Failing!
> >
> > Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> > Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > XX:+UseConcMarkSweepGC
> >
> > 1 tests failed.
> > FAILED:
> >
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfT
> > a
> > sksParse
> >
> > Error Message:
> > Could not remove the following files (in the order of attempts):
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >
> > Stack Trace:
> > java.io.IOException: Could not remove the following files (in the
> > order of
> > attempts):
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >
> > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> > 	at
> > org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestR
> > ul
> > eTemporaryFilesCleanup.java:126)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways
> > (T
> > estRuleAdapter.java:31)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:43)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:36)
> > 	at
> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleA
> > ss
> > ertionsRequired.java:43)
> > 	at
> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail
> > ure
> > .java:48)
> > 	at
> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR
> > ule
> > IgnoreAfterMaxFailures.java:65)
> > 	at
> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgn
> > ore
> > TestSuites.java:55)
> > 	at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Sta
> > t
> > ementAdapter.java:36)
> > 	at
> >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> > run(ThreadLeakControl.java:365)
> > 	at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > Build Log:
> > [...truncated 7622 lines...]
> >    [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> Spatial Strategy:
> > RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spati
> > al Context{geo=true, calculator=Haversine, worldBounds=Rect(minX=-
> > 180.0,maxX=180.0,minY=-
> > 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   1> ------------> queries:
> >    [junit4]   1>
> >    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> > sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
> >    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-
> > bit)/cpus=2,threads=1,free=120281832,total=140083200
> >    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> > WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> > TrecContentSourceTest, TestQualityRun, PerfTaskTest,
> > LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
> >    [junit4]   2> NOTE: reproduce with: ant test  -
> Dtestcase=TestPerfTasksParse
> > -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -
> > Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
> >    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
> >    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> > following files (in the order of attempts):
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> > java.nio.file.AccessDeniedException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> >    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001:
> > java.nio.file.DirectoryNotEmptyException:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> >
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> > .TestPerfTasksParse-2D34950549928D3F-001
> >    [junit4]    > 	at
> > __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> >    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> >    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
> >    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> >
> > [...truncated 33 lines...]
> > BUILD FAILED
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:491: The following error occurred while executing
> > this
> > line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\build.xml:471: The following error occurred while executing
> > this
> > line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:60:
> > The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> > targets.xml:39: The following error occurred while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\build.xml:462: The following error occurred while
> > executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:2140: The following error occurred
> > while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\module-build.xml:58: The following error occurred while
> > executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:1358: The following error occurred
> > while executing this line:
> > C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> > Windows\lucene\common-build.xml:961: There were test failures: 18
> > suites,
> > 93 tests, 1 suite-level error
> >
> > Total time: 31 minutes 50 seconds
> > Build step 'Invoke Ant' marked build as failure [description-setter]
> > Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> > XX:+UseConcMarkSweepGC Archiving artifacts Recording test results
> > Email was triggered for: Failure - Any Sending email for trigger:
> > Failure - Any
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


RE: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
This fails reproducible.

I will dig!

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Sunday, September 14, 2014 2:48 AM
> To: noble@apache.org; sarowe@gmail.com; rmuir@apache.org;
> uwe@thetaphi.de; mikemccand@apache.org; dev@lucene.apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build #
> 4310 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
> Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> XX:+UseConcMarkSweepGC
> 
> 1 tests failed.
> FAILED:
> junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTa
> sksParse
> 
> Error Message:
> Could not remove the following files (in the order of attempts):
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
> 
> Stack Trace:
> java.io.IOException: Could not remove the following files (in the order of
> attempts):
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
> 
> 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
> 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
> 	at
> org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRul
> eTemporaryFilesCleanup.java:126)
> 	at
> com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(T
> estRuleAdapter.java:31)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:43)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAss
> ertionsRequired.java:43)
> 	at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure
> .java:48)
> 	at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRule
> IgnoreAfterMaxFailures.java:65)
> 	at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnore
> TestSuites.java:55)
> 	at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stat
> ementAdapter.java:36)
> 	at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.
> run(ThreadLeakControl.java:365)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 
> Build Log:
> [...truncated 7622 lines...]
>    [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> Spatial Strategy:
> RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:Spatial
> Context{geo=true, calculator=Haversine, worldBounds=Rect(minX=-
> 180.0,maxX=180.0,minY=-
> 90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   1> ------------> queries:
>    [junit4]   1>
>    [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{},
> sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
>    [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-
> bit)/cpus=2,threads=1,free=120281832,total=140083200
>    [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest,
> WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest,
> TrecContentSourceTest, TestQualityRun, PerfTaskTest, LineDocSourceTest,
> TestPerfTasksLogic, TestPerfTasksParse]
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse
> -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -
> Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
>    [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
>    [junit4]    > Throwable #1: java.io.IOException: Could not remove the
> following files (in the order of attempts):
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt:
> java.nio.file.AccessDeniedException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
>    [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001:
> java.nio.file.DirectoryNotEmptyException:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask
> .TestPerfTasksParse-2D34950549928D3F-001
>    [junit4]    > 	at
> __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
>    [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
>    [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!
> 
> [...truncated 33 lines...]
> BUILD FAILED
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:491: The following error occurred while executing this
> line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\build.xml:471: The following error occurred while executing this
> line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60:
> The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-
> targets.xml:39: The following error occurred while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\build.xml:462: The following error occurred while executing
> this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:2140: The following error occurred
> while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\module-build.xml:58: The following error occurred while
> executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:1358: The following error occurred
> while executing this line:
> C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-
> Windows\lucene\common-build.xml:961: There were test failures: 18 suites,
> 93 tests, 1 suite-level error
> 
> Total time: 31 minutes 50 seconds
> Build step 'Invoke Ant' marked build as failure [description-setter]
> Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -
> XX:+UseConcMarkSweepGC Archiving artifacts Recording test results Email
> was triggered for: Failure - Any Sending email for trigger: Failure - Any
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.7.0_67) - Build # 4310 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4310/
Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.benchmark.byTask.TestPerfTasksParse

Error Message:
Could not remove the following files (in the order of attempts):    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
   C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001

	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:126)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 7622 lines...]
   [junit4] Suite: org.apache.lucene.benchmark.byTask.TestPerfTasksParse
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> Spatial Strategy: RecursivePrefixTreeStrategy(SPG:(QuadPrefixTree(maxLevels:26,ctx:SpatialContext{geo=true, calculator=Haversine, worldBounds=Rect(minX=-180.0,maxX=180.0,minY=-90.0,maxY=90.0)})),pointsOnly,pruneLeafyBranches)
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   1> ------------> queries:
   [junit4]   1> 
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=sq_AL, timezone=America/Yakutat
   [junit4]   2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=2,threads=1,free=120281832,total=140083200
   [junit4]   2> NOTE: All tests run in this JVM: [DocMakerTest, WriteLineDocTaskTest, AddIndexesTaskTest, StreamUtilsTest, TrecContentSourceTest, TestQualityRun, PerfTaskTest, LineDocSourceTest, TestPerfTasksLogic, TestPerfTasksParse]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestPerfTasksParse -Dtests.seed=2D34950549928D3F -Dtests.slow=true -Dtests.locale=sq_AL -Dtests.timezone=America/Yakutat -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestPerfTasksParse (suite) <<<
   [junit4]    > Throwable #1: java.io.IOException: Could not remove the following files (in the order of attempts):
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-001.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt: java.nio.file.AccessDeniedException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001\linefile-002.txt
   [junit4]    >    C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001: java.nio.file.DirectoryNotEmptyException: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build\benchmark\test\J0\temp\lucene.benchmark.byTask.TestPerfTasksParse-2D34950549928D3F-001
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2D34950549928D3F]:0)
   [junit4]    > 	at org.apache.lucene.util.IOUtils.rm(IOUtils.java:288)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed in 3.00s, 3 tests, 1 error <<< FAILURES!

[...truncated 33 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\build.xml:462: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:2140: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\module-build.xml:58: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 18 suites, 93 tests, 1 suite-level error

Total time: 31 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_20) - Build # 4309 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4309/
Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 31708 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.3
[forbidden-apis] Reading API signatures: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\tools\forbiddenApis\base.txt
[forbidden-apis] Reading API signatures: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\tools\forbiddenApis\servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#<init>(byte[]) [Uses default charset]
[forbidden-apis]   in org.apache.solr.cloud.Overseer$ClusterStateUpdater (Overseer.java:316)
[forbidden-apis] Scanned 2357 (and 1560 related) class file(s) for forbidden API invocations (in 3.45s), 1 error(s).

BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:85: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:271: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:479: Check for forbidden API calls failed, see log.

Total time: 177 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_40-ea-b04) - Build # 4308 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4308/
Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]
	at __randomizedtesting.SeedInfo.seed([EE65860654A833E4:6F83081E23F753D8]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	... 46 more




Build Log:
[...truncated 11187 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\init-core-data-001
   [junit4]   2> 2489168 T5111 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2489168 T5111 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2489180 T5111 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2489184 T5111 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2489186 T5112 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2489281 T5111 oasc.ZkTestServer.run start zk server on port:56184
   [junit4]   2> 2489283 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2489287 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2489297 T5118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e04284 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2489298 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2489299 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2489300 T5111 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2489314 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2489320 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2489323 T5120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a1b7aa name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2489323 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2489323 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2489324 T5111 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2489328 T5111 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2489334 T5111 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2489339 T5111 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2489346 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2489347 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2489357 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2489357 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2489366 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2489366 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2489374 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2489374 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2489383 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2489383 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2489395 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2489395 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2489402 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2489403 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2489412 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2489413 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2489418 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2489419 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2489426 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2489426 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2489434 T5111 oasc.AbstractZkTestCase.putConfig put C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2489434 T5111 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2491840 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2491871 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56191
   [junit4]   2> 2491871 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2491871 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\'
   [junit4]   2> 2491913 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\solr.xml
   [junit4]   2> 2491932 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2491934 T5111 oasc.CoreContainer.<init> New CoreContainer 27308953
   [junit4]   2> 2491934 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\]
   [junit4]   2> 2491935 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2491936 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2491936 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2491937 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2491937 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2491938 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2491938 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2491939 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2491940 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2491940 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2491941 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2491941 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2491942 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2491942 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2491943 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2491944 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2491946 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2491947 T5131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27bd5f name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2491948 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2491949 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2491954 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2491957 T5133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d6ac57 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2491958 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2491961 T5111 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2491969 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2491977 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2491983 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2491990 T5111 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2491996 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2492000 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56191_
   [junit4]   2> 2492001 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56191_
   [junit4]   2> 2492007 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2492011 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2492016 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2492019 T5111 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:56191_
   [junit4]   2> 2492020 T5111 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2492023 T5111 oasc.Overseer.start Overseer (id=92443007120441347-127.0.0.1:56191_-n_0000000000) starting
   [junit4]   2> 2492029 T5111 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2492050 T5111 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2492052 T5135 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2492054 T5111 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2492059 T5111 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2492063 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2492072 T5134 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2492074 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002
   [junit4]   2> 2493317 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\conf\
   [junit4]   2> 2493540 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2493544 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2493544 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2493545 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2493548 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2493552 T5139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12b21ea name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2493552 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2493552 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2493553 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2493561 T5111 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2493561 T5111 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2493561 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2493565 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2493567 T5141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d7672d name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2493568 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2493568 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2493572 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2493583 T5134 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 2493589 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2493596 T5142 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2493605 T5143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2495646 T5111 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2495648 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2495653 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56206
   [junit4]   2> 2495655 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2495655 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\'
   [junit4]   2> 2495694 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\solr.xml
   [junit4]   2> 2495714 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2495715 T5111 oasc.CoreContainer.<init> New CoreContainer 19705578
   [junit4]   2> 2495715 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\]
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2495717 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2495719 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2495720 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2495720 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2495720 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2495720 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2495721 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2495722 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2495722 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2495724 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2495727 T5154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@463656 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2495728 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2495728 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2495733 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2495737 T5156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d49c1a name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2495737 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2495753 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2496629 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56206_
   [junit4]   2> 2496631 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56206_
   [junit4]   2> 2496644 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2496649 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003
   [junit4]   2> 2497874 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\conf\
   [junit4]   2> 2498069 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2498074 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2498074 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2500365 T5111 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2500366 T5111 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2500372 T5111 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56215
   [junit4]   2> 2500374 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2500374 T5111 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2500374 T5111 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2500375 T5111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\'
   [junit4]   2> 2500423 T5111 oasc.ConfigSolr.fromFile Loading container configuration from C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\solr.xml
   [junit4]   2> 2500440 T5111 oasc.CorePropertiesLocator.<init> Config-defined core root directory: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2500441 T5111 oasc.CoreContainer.<init> New CoreContainer 10850303
   [junit4]   2> 2500441 T5111 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\]
   [junit4]   2> 2500442 T5111 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2500442 T5111 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2500443 T5111 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2500443 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2500444 T5111 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2500445 T5111 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2500446 T5111 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2500446 T5111 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2500447 T5111 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2500447 T5111 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56184/solr
   [junit4]   2> 2500448 T5111 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2500448 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2500450 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2500453 T5169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@804a27 name:ZooKeeperConnection Watcher:127.0.0.1:56184 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2500454 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2500454 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2500461 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2500463 T5171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8d66 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2500464 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2500478 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2501354 T5111 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56215_
   [junit4]   2> 2501356 T5111 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56215_
   [junit4]   2> 2501370 T5111 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2501382 T5111 oasc.CorePropertiesLocator.discover Looking for core definitions underneath C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004
   [junit4]   2> 2502839 T5111 oasc.CorePropertiesLocator.discoverUnder Found core conf in C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-004\conf\
   [junit4]   2> 2503112 T5111 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2503115 T5111 oass.SolrDispatchFilter.init user.dir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0
   [junit4]   2> 2503115 T5111 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2503122 T5111 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 2503122 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2503124 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2503128 T5175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9622b name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2503128 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2503129 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2503132 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2503147 T5152 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 2503151 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503154 T5135 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2503157 T5176 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"customcollreplicadeletion",
   [junit4]   2> 	  "replicationFactor":"1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "maxShardsPerNode":"5",
   [junit4]   2> 	  "shards":"a,b",
   [junit4]   2> 	  "stateFormat":"2",
   [junit4]   2> 	  "router.name":"implicit"}
   [junit4]   2> 2503162 T5176 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2503163 T5176 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 2503164 T5176 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion
   [junit4]   2> 2503178 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: customcollreplicadeletion
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.createCollection Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 2503192 T5134 oasc.Overseer$ClusterStateUpdater.createCollection state version customcollreplicadeletion 2
   [junit4]   2> 2503204 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503210 T5143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5172 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503210 T5142 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5177 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503211 T5157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2503278 T5176 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, b] , replicationFactor : 1
   [junit4]   2> 2503278 T5176 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_a_replica1 as part of slice a of collection customcollreplicadeletion on 127.0.0.1:56191_
   [junit4]   2> 2503281 T5176 oasc.OverseerCollectionProcessor.createCollection Creating shard customcollreplicadeletion_b_replica1 as part of slice b of collection customcollreplicadeletion on 127.0.0.1:56206_
   [junit4]   2> 2503291 T5128 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2503292 T5148 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 2503312 T5128 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2503314 T5148 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=down collection=customcollreplicadeletion
   [junit4]   2> 2503324 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503327 T5128 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2503327 T5128 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2503328 T5148 oasc.ZkController.preRegister Registering watch for external collection customcollreplicadeletion
   [junit4]   2> 2503332 T5128 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2503333 T5148 oascc.ZkStateReader.addZkWatch addZkWatch customcollreplicadeletion
   [junit4]   2> 2503336 T5148 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 2503338 T5128 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503340 T5128 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2503345 T5148 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503346 T5148 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2503356 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 2503357 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56191",
   [junit4]   2> 	  "node_name":"127.0.0.1:56191_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2503363 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503365 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503366 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503368 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503374 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503385 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 2503386 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56206",
   [junit4]   2> 	  "node_name":"127.0.0.1:56206_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2503392 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2503394 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2503398 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503398 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2503401 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2503406 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2504341 T5128 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 2504342 T5128 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2504344 T5128 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2504346 T5148 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 2504346 T5128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2504346 T5148 oasc.ZkController.createCollectionZkNode Check for collection zkNode:customcollreplicadeletion
   [junit4]   2> 2504350 T5148 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2504351 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2504351 T5128 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2504352 T5128 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\'
   [junit4]   2> 2504357 T5148 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2504357 T5148 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\'
   [junit4]   2> 2504519 T5128 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2504539 T5148 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 2504565 T5148 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2504574 T5128 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2504576 T5148 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2504585 T5128 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2504646 T5128 oass.IndexSchema.readSchema [customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 2504677 T5148 oass.IndexSchema.readSchema [customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 2504958 T5128 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2504968 T5148 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2504987 T5148 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2504992 T5148 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2504996 T5128 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2505005 T5128 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2505092 T5128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505095 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505107 T5148 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505107 T5128 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2505115 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505119 T5128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505122 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505125 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505126 T5148 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505128 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505129 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505131 T5128 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2505132 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505135 T5148 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505135 T5148 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_b_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2505136 T5148 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2505137 T5148 oasc.SolrCore.<init> [customcollreplicadeletion_b_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\
   [junit4]   2> 2505138 T5148 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a10908
   [junit4]   2> 2505138 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2505140 T5148 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\
   [junit4]   2> 2505141 T5148 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index/
   [junit4]   2> 2505142 T5148 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_b_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 2505140 T5128 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2505144 T5128 oasc.CoreContainer.create Creating SolrCore 'customcollreplicadeletion_a_replica1' using configuration from collection customcollreplicadeletion
   [junit4]   2> 2505144 T5128 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2505143 T5148 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-003\customcollreplicadeletion_b_replica1\data\index
   [junit4]   2> 2505148 T5128 oasc.SolrCore.<init> [customcollreplicadeletion_a_replica1] Opening new SolrCore at ..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\, dataDir=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\
   [junit4]   2> 2505150 T5128 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a10908
   [junit4]   2> 2505149 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=75.3212890625, floorSegmentMB=2.1005859375, forceMergeDeletesPctAllowed=3.1211462431648207, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44228484152719927
   [junit4]   2> 2505152 T5128 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\
   [junit4]   2> 2505152 T5128 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index/
   [junit4]   2> 2505152 T5128 oasc.SolrCore.initIndex WARN [customcollreplicadeletion_a_replica1] Solr index directory 'C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index' doesn't exist. Creating new index...
   [junit4]   2> 2505153 T5128 oasc.CachingDirectoryFactory.get return new directory for C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001\tempDir-002\customcollreplicadeletion_a_replica1\data\index
   [junit4]   2> 2505155 T5128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=42, maxMergedSegmentMB=75.3212890625, floorSegmentMB=2.1005859375, forceMergeDeletesPctAllowed=3.1211462431648207, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.44228484152719927
   [junit4]   2> 2505157 T5148 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5caee6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bd4bc0),segFN=segments_1,generation=1}
   [junit4]   2> 2505157 T5128 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dbbc23 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@159860e),segFN=segments_1,generation=1}
   [junit4]   2> 2505159 T5128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505158 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505186 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2505188 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505189 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505190 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505190 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505191 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2505191 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2505193 T5148 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505193 T5148 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505194 T5148 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2505193 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505196 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2505197 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505198 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2505199 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2505196 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505201 T5128 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505203 T5128 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2505203 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505204 T5128 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2505207 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505210 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505212 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505215 T5128 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2505216 T5128 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2505216 T5128 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2505218 T5128 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2505220 T5128 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2505205 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2505222 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2505224 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2505225 T5148 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2505226 T5128 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2505226 T5148 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2505230 T5148 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2505231 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2505233 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2505235 T5148 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2505238 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505240 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505244 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505244 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505249 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505250 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505254 T5128 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505254 T5148 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2505262 T5128 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2505262 T5148 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2505266 T5148 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2505265 T5128 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2505268 T5148 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1604624151325485]
   [junit4]   2> 2505269 T5128 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=11, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1604624151325485]
   [junit4]   2> 2505271 T5148 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5caee6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bd4bc0),segFN=segments_1,generation=1}
   [junit4]   2> 2505271 T5128 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dbbc23 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@159860e),segFN=segments_1,generation=1}
   [junit4]   2> 2505274 T5128 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505275 T5128 oass.SolrIndexSearcher.<init> Opening Searcher@a56adc[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 2505276 T5128 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2505273 T5148 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2505277 T5148 oass.SolrIndexSearcher.<init> Opening Searcher@19fb027[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 2505279 T5148 oascc.ZkStateReader.readConfigName Load collection config from:/collections/customcollreplicadeletion
   [junit4]   2> 2505280 T5128 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2505281 T5128 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2505283 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2505285 T5128 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2505286 T5128 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505287 T5148 oascc.ZkStateReader.readConfigName path=/collections/customcollreplicadeletion configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2505288 T5148 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2505290 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505291 T5128 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505291 T5148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2505294 T5148 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2505295 T5128 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2505297 T5148 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505302 T5148 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505302 T5148 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505307 T5148 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2505318 T5128 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2505319 T5128 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505320 T5128 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2505319 T5148 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2505323 T5148 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2505343 T5148 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_b_replica1
   [junit4]   2> 2505343 T5148 oasc.ZkController.register Register replica - core:customcollreplicadeletion_b_replica1 address:http://127.0.0.1:56206 collection:customcollreplicadeletion shard:b
   [junit4]   2> 2505346 T5181 oasc.SolrCore.registerSearcher [customcollreplicadeletion_b_replica1] Registered new searcher Searcher@19fb027[customcollreplicadeletion_b_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2505354 T5180 oasc.SolrCore.registerSearcher [customcollreplicadeletion_a_replica1] Registered new searcher Searcher@a56adc[customcollreplicadeletion_a_replica1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2505361 T5148 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 2505363 T5128 oasc.CoreContainer.registerCore registering core: customcollreplicadeletion_a_replica1
   [junit4]   2> 2505364 T5128 oasc.ZkController.register Register replica - core:customcollreplicadeletion_a_replica1 address:http://127.0.0.1:56191 collection:customcollreplicadeletion shard:a
   [junit4]   2> 2505370 T5128 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 2505404 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard b
   [junit4]   2> 2505409 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard a
   [junit4]   2> 2505412 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505415 T5148 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2505416 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3139 name=customcollreplicadeletion_b_replica1 org.apache.solr.core.SolrCore@61f90e url=http://127.0.0.1:56206/customcollreplicadeletion_b_replica1 node=127.0.0.1:56206_ C3139_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_b_replica1 props:{core=customcollreplicadeletion_b_replica1, base_url=http://127.0.0.1:56206, node_name=127.0.0.1:56206_, state=down}
   [junit4]   2> 2505418 T5148 C3139 P56206 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2505419 T5148 C3139 P56206 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2505422 T5148 C3139 P56206 oasc.SyncStrategy.syncToMe http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 2505423 T5148 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 2505424 T5148 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 2505424 T5128 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2505427 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3140 name=customcollreplicadeletion_a_replica1 org.apache.solr.core.SolrCore@1ebd0d9 url=http://127.0.0.1:56191/customcollreplicadeletion_a_replica1 node=127.0.0.1:56191_ C3140_STATE=coll:customcollreplicadeletion core:customcollreplicadeletion_a_replica1 props:{core=customcollreplicadeletion_a_replica1, base_url=http://127.0.0.1:56191, node_name=127.0.0.1:56191_, state=down}
   [junit4]   2> 2505428 T5128 C3140 P56191 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2505429 T5128 C3140 P56191 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2505430 T5128 C3140 P56191 oasc.SyncStrategy.syncToMe http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 2505431 T5128 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 2505432 T5128 oascc.SolrZkClient.makePath makePath: /collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 2505443 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 2505458 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505464 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505466 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505468 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505475 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505490 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 2505497 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505500 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505506 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505507 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505512 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505524 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505535 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 2505544 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505546 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505548 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505550 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505552 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505558 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505563 T5148 oasc.ZkController.register We are http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/ and leader is http://127.0.0.1:56206/customcollreplicadeletion_b_replica1/
   [junit4]   2> 2505564 T5148 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_b_replica1 baseURL=http://127.0.0.1:56206
   [junit4]   2> 2505564 T5148 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2505564 T5148 oasc.ZkController.publish publishing core=customcollreplicadeletion_b_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2505571 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505577 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505586 T5148 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 2505588 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505593 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505593 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505593 T5148 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2} status=0 QTime=2301 
   [junit4]   2> 2505597 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505605 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505617 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 2505618 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56206",
   [junit4]   2> 	  "node_name":"127.0.0.1:56206_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"b",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2505623 T5128 oasc.ZkController.register We are http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/ and leader is http://127.0.0.1:56191/customcollreplicadeletion_a_replica1/
   [junit4]   2> 2505624 T5128 oasc.ZkController.register No LogReplay needed for core=customcollreplicadeletion_a_replica1 baseURL=http://127.0.0.1:56191
   [junit4]   2> 2505624 T5128 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2505627 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505628 T5128 oasc.ZkController.publish publishing core=customcollreplicadeletion_a_replica1 state=active collection=customcollreplicadeletion
   [junit4]   2> 2505628 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505633 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505634 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505636 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505652 T5128 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505657 T5128 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2} status=0 QTime=2366 
   [junit4]   2> 2505659 T5176 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 2505665 T5176 oasc.OverseerCollectionProcessor$Runner.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=2301},core=customcollreplicadeletion_b_replica1},null={responseHeader={status=0,QTime=2366},core=customcollreplicadeletion_a_replica1}}}
   [junit4]   2> 2505666 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505669 T5157 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
   [junit4]   2> 2505675 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505678 T5152 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2} status=0 QTime=2531 
   [junit4]   2> 2505683 T5111 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: customcollreplicadeletion failOnTimeout:true timeout (sec):330
   [junit4]   2> 2505690 T5134 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 2505691 T5134 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56191",
   [junit4]   2> 	  "node_name":"127.0.0.1:56191_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2505698 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2505699 T5134 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 2505701 T5157 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505701 T5142 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/customcollreplicadeletion/state.json, has occurred - updating... 
   [junit4]   2> 2505703 T5157 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2505708 T5142 oascc.ZkStateReader.updateWatchedCollection Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 2506702 T5111 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: customcollreplicadeletion
   [junit4]   2> 2506703 T5111 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2506706 T5111 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2506710 T5185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c27170 name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2506710 T5111 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2506711 T5111 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2506715 T5111 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2506735 T5150 oasha.CollectionsHandler.handleRemoveReplica Remove replica: replica=core_node1&action=deletereplica&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 2506738 T5142 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2506743 T5135 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
   [junit4]   2> 	  "operation":"deletereplica",
   [junit4]   2> 	  "collection":"customcollreplicadeletion",
   [junit4]   2> 	  "shard":"a",
   [junit4]   2> 	  "replica":"core_node1"}
   [jun

[...truncated too long message...]

bstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2631993 T5175 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9622b name:ZooKeeperConnection Watcher:127.0.0.1:56184/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 2631993 T5175 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 2632013 T5175 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@182a1ac rejected from java.util.concurrent.ThreadPoolExecutor@1ad5ee3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteLastCustomShardedReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=EE65860654A833E4 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=Atlantic/Reykjavik -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    143s | DeleteLastCustomShardedReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:56215, http://127.0.0.1:56191, http://127.0.0.1:56206]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([EE65860654A833E4:6F83081E23F753D8]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:890)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:672)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:615)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:116)
   [junit4]    > 	at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:106)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : customcollreplicadeletion/a/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
   [junit4]    > 	... 46 more
   [junit4]   2> 2632021 T5111 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\.\temp\solr.cloud.DeleteLastCustomShardedReplicaTest-EE65860654A833E4-001
   [junit4]   2> 142861 T5110 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene410: {}, docValues:{}, sim=DefaultSimilarity, locale=is, timezone=Atlantic/Reykjavik
   [junit4]   2> NOTE: Windows 7 6.1 x86/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=2,threads=1,free=124461504,total=345505792
   [junit4]   2> NOTE: All tests run in this JVM: [ShardRoutingCustomTest, BadComponentTest, EnumFieldTest, DOMUtilTest, SharedFSAutoReplicaFailoverUtilsTest, RangeFacetTest, DistributedFacetPivotLongTailTest, ReturnFieldsTest, TestQueryTypes, TestManagedSchemaFieldTypeResource, TestCloudSchemaless, ShowFileRequestHandlerTest, OverseerRolesTest, StatelessScriptUpdateProcessorFactoryTest, DistributedIntervalFacetingTest, TestReload, TestDocBasedVersionConstraints, TestQueryUtils, TestSolrDeletionPolicy2, RAMDirectoryFactoryTest, BlockCacheTest, HighlighterTest, TestDFRSimilarityFactory, AlternateDirectoryTest, TestFaceting, ReplicationFactorTest, TestCursorMarkWithoutUniqueKey, PathHierarchyTokenizerFactoryTest, TestArbitraryIndexDir, TestManagedResource, TestHighlightDedupGrouping, TestSolr4Spatial, PluginInfoTest, TestFieldTypeResource, UpdateParamsTest, TestOverriddenPrefixQueryForCustomFieldType, SpellingQueryConverterTest, TestIBSimilarityFactory, TestBadConfig, CoreAdminCreateDiscoverTest, BadIndexSchemaTest, BasicFunctionalityTest, TermVectorComponentDistributedTest, SpatialFilterTest, HdfsRecoveryZkTest, TermVectorComponentTest, RequestHandlersTest, TestCloudInspectUtil, TestValueSourceCache, IndexSchemaTest, TestImplicitCoreProperties, TestTolerantSearch, TestRandomMergePolicy, TestCustomSort, TestExceedMaxTermLength, DeleteShardTest, QueryFacetTest, TestLRUCache, SoftAutoCommitTest, SuggesterFSTTest, ChaosMonkeySafeLeaderTest, TestRTGBase, NumericFieldsTest, PrimUtilsTest, LoggingHandlerTest, TestRestManager, QueryResultKeyTest, TestManagedSynonymFilterFactory, TestFreeTextSuggestions, TestJmxIntegration, TestClassNameShortening, TestSuggestSpellingConverter, TestCoreDiscovery, RankQueryTest, SolrIndexSplitterTest, CollectionsAPIDistributedZkTest, TestExtendedDismaxParser, DebugComponentTest, SimpleFacetsTest, HardAutoCommitTest, TestOrdValues, TestCloudPivotFacet, TestNonNRTOpen, AbstractAnalyticsStatsTest, SuggesterTest, TestSolrIndexConfig, MinimalSchemaTest, CursorMarkTest, TestStressLucene, UUIDFieldTest, TestConfigSets, CurrencyFieldXmlFileTest, TestShortCircuitedRequests, HighlighterConfigTest, SpellCheckCollatorTest, TestNRTOpen, TestCollationField, TestShardHandlerFactory, DistribDocExpirationUpdateProcessorTest, DeleteInactiveReplicaTest, UniqFieldsUpdateProcessorFactoryTest, OverriddenZkACLAndCredentialsProvidersTest, TestStressUserVersions, BlockDirectoryTest, OutputWriterTest, TestFieldTypeCollectionResource, DistributedFacetPivotSmallTest, TestCloudManagedSchemaConcurrent, DocExpirationUpdateProcessorFactoryTest, TestReversedWildcardFilterFactory, CopyFieldTest, TimeZoneUtilsTest, MigrateRouteKeyTest, TestJoin, SuggesterWFSTTest, PingRequestHandlerTest, TestDefaultSimilarityFactory, CursorPagingTest, TestRawResponseWriter, ExternalCollectionsTest, PrimitiveFieldTypeTest, URLClassifyProcessorTest, IndexBasedSpellCheckerTest, TestAddFieldRealTimeGet, PolyFieldTest, TestBlendedInfixSuggestions, TestReplicationHandlerBackup, SolrCoreCheckLockOnStartupTest, ResponseLogComponentTest, TestManagedSchemaDynamicFieldResource, ScriptEngineTest, FunctionTest, TestSchemaVersionResource, BadCopyFieldTest, TestPseudoReturnFields, FullSolrCloudDistribCmdsTest, ModifyConfFileTest, DateRangeFieldTest, TestJmxMonitoredMap, TestMaxScoreQueryParser, TestRangeQuery, TestExpandComponent, FileBasedSpellCheckerTest, ChangedSchemaMergeTest, TestChildDocTransformer, TestDynamicFieldResource, DisMaxRequestHandlerTest, TestSimpleQParserPlugin, TestDocSet, SharedFSAutoReplicaFailoverTest, TestManagedResourceStorage, TestReplicationHandler, DeleteLastCustomShardedReplicaTest]
   [junit4] Completed in 143.76s, 1 test, 1 error <<< FAILURES!

[...truncated 887 lines...]
BUILD FAILED
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:491: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:471: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\build.xml:60: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\extra-targets.xml:39: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build.xml:189: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\common-build.xml:496: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:1358: The following error occurred while executing this line:
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\lucene\common-build.xml:961: There were test failures: 438 suites, 1830 tests, 1 error, 54 ignored (32 assumptions)

Total time: 155 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any