You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/02/26 01:15:39 UTC

[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1911 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1911/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([4D75887B35EB759B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([4D75887B35EB759B]:0)


REGRESSION:  org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest.testDistribSearch

Error Message:
this codec cannot write docvalues

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: this codec cannot write docvalues
	at __randomizedtesting.SeedInfo.seed([4D75887B35EB759B:CC93066342B415A7]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:497)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146)
	at org.apache.solr.BaseDistributedSearchTestCase.commit(BaseDistributedSearchTestCase.java:505)
	at org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest.doTest(DistributedQueryComponentOptimizationTest.java:66)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 10577 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest
   [junit4]   2> 4988487 T16025 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 4988488 T16025 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /v_fzy/e
   [junit4]   2> 4988491 T16025 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395
   [junit4]   2> 4988492 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4988493 T16025 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4988494 T16025 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4988495 T16025 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4988643 T16025 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4988741 T16025 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4988743 T16025 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4988750 T16025 oass.IndexSchema.readSchema [null] Schema name=test-custom-field-sort
   [junit4]   2> 4988793 T16025 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4988797 T16025 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4988799 T16025 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4988799 T16025 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr
   [junit4]   2> 4988800 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/'
   [junit4]   2> 4988894 T16025 oasc.CoreContainer.<init> New CoreContainer 1514486332
   [junit4]   2> 4988895 T16025 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/]
   [junit4]   2> 4988897 T16025 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 4988897 T16025 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 4988898 T16025 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 4988898 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4988899 T16025 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4988899 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4988900 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4988900 T16025 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4988901 T16025 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4988917 T16025 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4988917 T16025 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4988917 T16025 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 4988924 T16026 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1
   [junit4]   2> 4988925 T16026 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4988926 T16026 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4988927 T16026 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4988927 T16026 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4989010 T16026 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4989107 T16026 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4989108 T16026 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4989115 T16026 oass.IndexSchema.readSchema [collection1] Schema name=test-custom-field-sort
   [junit4]   2> 4989155 T16026 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4989158 T16026 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4989160 T16026 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4989161 T16026 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395/
   [junit4]   2> 4989161 T16026 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5738ff7c
   [junit4]   2> 4989162 T16026 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395
   [junit4]   2> 4989163 T16026 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395/index/
   [junit4]   2> 4989163 T16026 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395/index' doesn't exist. Creating new index...
   [junit4]   2> 4989164 T16026 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-DistributedQueryComponentOptimizationTest-1393371199395/index
   [junit4]   2> 4989164 T16026 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24571304246398445]
   [junit4]   2> 4989165 T16026 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@37fa5b99 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d6d88ea)),segFN=segments_1,generation=1}
   [junit4]   2> 4989166 T16026 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4989167 T16026 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 4989171 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989172 T16026 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989172 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4989173 T16026 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4989173 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989173 T16026 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989174 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4989174 T16026 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 4989175 T16026 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 4989175 T16026 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4989176 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4989176 T16026 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989177 T16026 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989177 T16026 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4989178 T16026 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 4989178 T16026 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 4989179 T16026 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4989180 T16026 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 4989180 T16026 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 4989181 T16026 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4989182 T16026 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 4989182 T16026 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4989183 T16026 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 4989183 T16026 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4989184 T16026 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 4989184 T16026 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4989185 T16026 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989186 T16026 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989186 T16026 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989187 T16026 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989188 T16026 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989188 T16026 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989189 T16026 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989189 T16026 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989190 T16026 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4989191 T16026 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4989208 T16026 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989210 T16026 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 4989214 T16026 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4989214 T16026 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4989215 T16026 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.7451171875, floorSegmentMB=0.6640625, forceMergeDeletesPctAllowed=29.335149011310897, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 4989216 T16026 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@37fa5b99 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d6d88ea)),segFN=segments_1,generation=1}
   [junit4]   2> 4989216 T16026 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4989217 T16026 oass.SolrIndexSearcher.<init> Opening Searcher@32b06c9b[collection1] main
   [junit4]   2> 4989217 T16026 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 4989225 T16026 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4989306 T16027 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32b06c9b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 4989313 T16026 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4989330 T16025 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 4989338 T16025 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 4989342 T16025 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 4989347 T16025 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 4989351 T16025 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:63664
   [junit4]   2> 4989354 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 4989354 T16025 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4989355 T16025 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr
   [junit4]   2> 4989356 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/'
   [junit4]   2> 4989404 T16025 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 4989562 T16025 oasc.CoreContainer.<init> New CoreContainer 1766085693
   [junit4]   2> 4989564 T16025 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/]
   [junit4]   2> 4989566 T16025 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 4989566 T16025 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 4989567 T16025 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 4989568 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4989568 T16025 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4989569 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4989569 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4989570 T16025 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4989571 T16025 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4989586 T16025 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4989587 T16025 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4989588 T16025 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 4989594 T16038 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1
   [junit4]   2> 4989595 T16038 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4989596 T16038 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4989597 T16038 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4989598 T16038 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4989688 T16038 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4989785 T16038 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4989786 T16038 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4989794 T16038 oass.IndexSchema.readSchema [collection1] Schema name=test-custom-field-sort
   [junit4]   2> 4989836 T16038 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4989840 T16038 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4989841 T16038 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4989842 T16038 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/control/data/
   [junit4]   2> 4989842 T16038 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5738ff7c
   [junit4]   2> 4989844 T16038 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/control/data
   [junit4]   2> 4989844 T16038 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/control/data/index/
   [junit4]   2> 4989844 T16038 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4989845 T16038 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/control/data/index
   [junit4]   2> 4989846 T16038 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24571304246398445]
   [junit4]   2> 4989847 T16038 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1f10229f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2311d9c2)),segFN=segments_1,generation=1}
   [junit4]   2> 4989847 T16038 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4989848 T16038 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 4989853 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989854 T16038 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989854 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4989855 T16038 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4989855 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989856 T16038 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989856 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4989857 T16038 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 4989857 T16038 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 4989857 T16038 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4989858 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4989859 T16038 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989859 T16038 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989859 T16038 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4989860 T16038 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 4989860 T16038 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 4989861 T16038 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4989862 T16038 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 4989863 T16038 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 4989864 T16038 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4989865 T16038 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 4989866 T16038 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4989867 T16038 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 4989867 T16038 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4989867 T16038 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 4989868 T16038 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4989869 T16038 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989870 T16038 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989871 T16038 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989871 T16038 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989872 T16038 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989873 T16038 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989874 T16038 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989875 T16038 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4989875 T16038 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4989876 T16038 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4989879 T16038 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989881 T16038 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 4989887 T16038 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4989887 T16038 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4989888 T16038 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.7451171875, floorSegmentMB=0.6640625, forceMergeDeletesPctAllowed=29.335149011310897, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 4989889 T16038 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@1f10229f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2311d9c2)),segFN=segments_1,generation=1}
   [junit4]   2> 4989890 T16038 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4989890 T16038 oass.SolrIndexSearcher.<init> Opening Searcher@3f2be016[collection1] main
   [junit4]   2> 4989891 T16038 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 4989926 T16038 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4989987 T16039 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f2be016[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 4989996 T16038 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4989996 T16025 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 4989997 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 4990002 T16025 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 4990007 T16025 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 4990010 T16025 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47131
   [junit4]   2> 4990014 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 4990014 T16025 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4990015 T16025 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr
   [junit4]   2> 4990015 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/'
   [junit4]   2> 4990064 T16025 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 4990163 T16025 oasc.CoreContainer.<init> New CoreContainer 1449787431
   [junit4]   2> 4990165 T16025 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/]
   [junit4]   2> 4990167 T16025 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 4990167 T16025 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 4990168 T16025 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 4990169 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4990169 T16025 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4990170 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4990171 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4990171 T16025 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4990172 T16025 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4990188 T16025 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4990189 T16025 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4990189 T16025 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 4990196 T16050 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1
   [junit4]   2> 4990196 T16050 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4990198 T16050 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4990199 T16050 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4990199 T16050 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4990289 T16050 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4990386 T16050 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4990387 T16050 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4990394 T16050 oass.IndexSchema.readSchema [collection1] Schema name=test-custom-field-sort
   [junit4]   2> 4990436 T16050 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4990439 T16050 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4990441 T16050 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4990442 T16050 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard0/data/
   [junit4]   2> 4990442 T16050 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5738ff7c
   [junit4]   2> 4990443 T16050 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard0/data
   [junit4]   2> 4990444 T16050 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard0/data/index/
   [junit4]   2> 4990444 T16050 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4990445 T16050 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard0/data/index
   [junit4]   2> 4990445 T16050 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24571304246398445]
   [junit4]   2> 4990447 T16050 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3ab4eb86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1abf9b2a)),segFN=segments_1,generation=1}
   [junit4]   2> 4990447 T16050 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4990448 T16050 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 4990453 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4990454 T16050 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4990454 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4990455 T16050 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4990455 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4990455 T16050 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4990456 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4990456 T16050 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 4990457 T16050 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 4990457 T16050 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4990457 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4990458 T16050 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4990459 T16050 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4990459 T16050 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4990459 T16050 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 4990460 T16050 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 4990461 T16050 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4990462 T16050 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 4990463 T16050 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 4990464 T16050 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4990464 T16050 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 4990465 T16050 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4990466 T16050 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 4990466 T16050 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4990467 T16050 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 4990468 T16050 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4990468 T16050 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990469 T16050 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990470 T16050 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990471 T16050 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990472 T16050 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990472 T16050 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990473 T16050 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990474 T16050 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4990475 T16050 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4990476 T16050 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4990493 T16050 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4990495 T16050 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 4990501 T16050 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4990501 T16050 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4990502 T16050 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.7451171875, floorSegmentMB=0.6640625, forceMergeDeletesPctAllowed=29.335149011310897, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 4990503 T16050 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@3ab4eb86 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1abf9b2a)),segFN=segments_1,generation=1}
   [junit4]   2> 4990503 T16050 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4990504 T16050 oass.SolrIndexSearcher.<init> Opening Searcher@51f0468a[collection1] main
   [junit4]   2> 4990504 T16050 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 4990541 T16050 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4990601 T16051 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51f0468a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 4990610 T16050 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4990610 T16025 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 4990611 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 4990616 T16025 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 4990621 T16025 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 4990624 T16025 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:62277
   [junit4]   2> 4990627 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 4990628 T16025 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4990628 T16025 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr
   [junit4]   2> 4990629 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/'
   [junit4]   2> 4990677 T16025 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 4990777 T16025 oasc.CoreContainer.<init> New CoreContainer 1296771769
   [junit4]   2> 4990778 T16025 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/]
   [junit4]   2> 4990780 T16025 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 4990781 T16025 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 4990782 T16025 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 4990782 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4990783 T16025 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4990783 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4990784 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4990785 T16025 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4990785 T16025 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4990801 T16025 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4990802 T16025 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4990802 T16025 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 4990809 T16062 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1
   [junit4]   2> 4990809 T16062 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4990811 T16062 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4990811 T16062 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4990812 T16062 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4990902 T16062 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4991046 T16062 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4991048 T16062 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4991055 T16062 oass.IndexSchema.readSchema [collection1] Schema name=test-custom-field-sort
   [junit4]   2> 4991097 T16062 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4991100 T16062 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4991102 T16062 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4991102 T16062 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard1/data/
   [junit4]   2> 4991103 T16062 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5738ff7c
   [junit4]   2> 4991104 T16062 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard1/data
   [junit4]   2> 4991104 T16062 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard1/data/index/
   [junit4]   2> 4991105 T16062 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4991105 T16062 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard1/data/index
   [junit4]   2> 4991106 T16062 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24571304246398445]
   [junit4]   2> 4991107 T16062 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@50a3e688 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f86d67)),segFN=segments_1,generation=1}
   [junit4]   2> 4991108 T16062 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4991108 T16062 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 4991114 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4991114 T16062 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4991115 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4991115 T16062 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4991116 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4991116 T16062 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4991116 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4991117 T16062 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 4991117 T16062 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 4991118 T16062 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4991118 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4991119 T16062 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4991119 T16062 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4991120 T16062 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4991120 T16062 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 4991121 T16062 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 4991122 T16062 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4991122 T16062 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 4991123 T16062 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 4991124 T16062 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4991125 T16062 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 4991126 T16062 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4991127 T16062 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 4991127 T16062 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4991127 T16062 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 4991128 T16062 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4991129 T16062 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991130 T16062 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991131 T16062 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991131 T16062 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991132 T16062 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991133 T16062 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991134 T16062 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991134 T16062 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991135 T16062 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4991136 T16062 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4991153 T16062 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4991155 T16062 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 4991161 T16062 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4991161 T16062 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4991162 T16062 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.7451171875, floorSegmentMB=0.6640625, forceMergeDeletesPctAllowed=29.335149011310897, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 4991163 T16062 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@50a3e688 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f86d67)),segFN=segments_1,generation=1}
   [junit4]   2> 4991164 T16062 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4991164 T16062 oass.SolrIndexSearcher.<init> Opening Searcher@58f4316a[collection1] main
   [junit4]   2> 4991165 T16062 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 4991198 T16062 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4991259 T16063 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@58f4316a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 4991267 T16062 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4991268 T16025 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 4991269 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 4991274 T16025 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 4991278 T16025 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 4991282 T16025 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:54201
   [junit4]   2> 4991285 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 4991285 T16025 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4991286 T16025 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr
   [junit4]   2> 4991287 T16025 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/'
   [junit4]   2> 4991336 T16025 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 4991435 T16025 oasc.CoreContainer.<init> New CoreContainer 352063868
   [junit4]   2> 4991437 T16025 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/]
   [junit4]   2> 4991439 T16025 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 4991439 T16025 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 4991440 T16025 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 4991441 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4991441 T16025 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4991442 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4991442 T16025 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4991443 T16025 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4991444 T16025 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4991458 T16025 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4991458 T16025 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4991459 T16025 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 4991465 T16074 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1
   [junit4]   2> 4991466 T16074 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 4991467 T16074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 4991468 T16074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 4991469 T16074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 4991558 T16074 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 4991655 T16074 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4991656 T16074 oass.IndexSchema.readSchema Reading Solr Schema from schema-custom-field.xml
   [junit4]   2> 4991663 T16074 oass.IndexSchema.readSchema [collection1] Schema name=test-custom-field-sort
   [junit4]   2> 4991705 T16074 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 4991709 T16074 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4991710 T16074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4991711 T16074 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard2/data/
   [junit4]   2> 4991712 T16074 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5738ff7c
   [junit4]   2> 4991713 T16074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard2/data
   [junit4]   2> 4991713 T16074 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard2/data/index/
   [junit4]   2> 4991714 T16074 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4991714 T16074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.handler.component.DistributedQueryComponentOptimizationTest-1393371200243/shard2/data/index
   [junit4]   2> 4991715 T16074 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24571304246398445]
   [junit4]   2> 4991716 T16074 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@39d641b3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d6aa46d)),segFN=segments_1,generation=1}
   [junit4]   2> 4991717 T16074 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4991717 T16074 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 4991722 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4991723 T16074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4991724 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4991724 T16074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 4991724 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4991725 T16074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4991725 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4991726 T16074 oasup.UniqFieldsUpdateProcessorFactory.init WARN Use of the 'fields' init param in UniqFieldsUpdateProcessorFactory is deprecated, please use 'fieldName' (or another FieldMutatingUpdateProcessorFactory selector option) instead
   [junit4]   2> 4991726 T16074 oasup.UniqFieldsUpdateProcessorFactory.init Replacing 'fields' init param with (individual) 'fieldName' params
   [junit4]   2> 4991727 T16074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 4991727 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4991728 T16074 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4991728 T16074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4991729 T16074 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4991729 T16074 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 4991729 T16074 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 4991730 T16074 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4991731 T16074 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 4991732 T16074 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 4991733 T16074 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 4991734 T16074 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 4991735 T16074 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 4991735 T16074 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 4991736 T16074 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 4991736 T16074 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 4991737 T16074 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4991738 T16074 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991739 T16074 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991739 T16074 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991740 T16074 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991741 T16074 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991742 T16074 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991742 T16074 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991743 T16074 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4991744 T16074 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 4991745 T16074 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 4991762 T16074 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4991764 T16074 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 4991770 T16074 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4991770 T16074 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4991771 T16074 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=55.7451171875, floorSegmentMB=0.6640625, forceMergeDeletesPctAllowed=29.335149011310897, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 4991772 T16074 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory@39d641b3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3d6aa46d)),segFN=segments_1,generation=1}
   [junit4]   2> 4991773 T16074 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4991773 T16074 oass.SolrIndexSearcher.<init> Opening Searcher@43771604[collection1] main
   [junit4]   2> 4991774 T16074 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 4991782 T16074 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 4991842 T16075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43771604[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 4991851 T16074 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4991852 T16025 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 4991852 T16025 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2695 name=collection1 org.apache.solr.core.SolrCore@7517a0df
   [junit4]   2> 4991987 T16031 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1461055602441584640)} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2696 name=collection1 org.apache.solr.core.SolrCore@47d5a92f
   [junit4]   2> 4992117 T16043 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1461055602577899520)} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2697 name=collection1 org.apache.solr.core.SolrCore@70a8d3e8
   [junit4]   2> 4992248 T16055 C2697 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1461055602715262976)} 0 1
   [junit4]   2> ASYNC  NEW_CORE C2698 name=collection1 org.apache.solr.core.SolrCore@71545e64
   [junit4]   2> 4992379 T16067 C2698 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1461055602852626432)} 0 1
   [junit4]   2> 4992385 T16032 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[1 (1461055602858917888)]} 0 1
   [junit4]   2> 4992389 T16044 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[1 (1461055602863112192)]} 0 1
   [junit4]   2> 4992393 T16033 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[2 (1461055602868355072)]} 0 0
   [junit4]   2> 4992397 T16056 C2697 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[2 (1461055602871500800)]} 0 1
   [junit4]   2> 4992400 T16034 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[3 (1461055602875695104)]} 0 0
   [junit4]   2> 4992405 T16068 C2698 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[3 (1461055602879889408)]} 0 1
   [junit4]   2> 4992408 T16035 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[4 (1461055602884083712)]} 0 0
   [junit4]   2> 4992412 T16045 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[4 (1461055602888278016)]} 0 0
   [junit4]   2> 4992416 T16036 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[5 (1461055602891423744)]} 0 1
   [junit4]   2> 4992419 T16057 C2697 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[5 (1461055602895618048)]} 0 0
   [junit4]   2> 4992423 T16037 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[6 (1461055602899812352)]} 0 0
   [junit4]   2> 4992427 T16069 C2698 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[6 (1461055602904006656)]} 0 0
   [junit4]   2> 4992431 T16093 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[7 (1461055602908200960)]} 0 0
   [junit4]   2> 4992435 T16046 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[7 (1461055602912395264)]} 0 0
   [junit4]   2> 4992439 T16096 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[8 (1461055602916589568)]} 0 0
   [junit4]   2> 4992443 T16058 C2697 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[8 (1461055602919735296)]} 0 1
   [junit4]   2> 4992446 T16099 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[9 (1461055602923929600)]} 0 0
   [junit4]   2> 4992450 T16070 C2698 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[9 (1461055602928123904)]} 0 0
   [junit4]   2> 4992454 T16102 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[10 (1461055602932318208)]} 0 0
   [junit4]   2> 4992458 T16047 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[10 (1461055602936512512)]} 0 0
   [junit4]   2> 4992462 T16105 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[11 (1461055602940706816)]} 0 0
   [junit4]   2> 4992466 T16059 C2697 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[11 (1461055602944901120)]} 0 1
   [junit4]   2> 4992470 T16108 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[12 (1461055602948046848)]} 0 1
   [junit4]   2> 4992473 T16071 C2698 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[12 (1461055602952241152)]} 0 0
   [junit4]   2> 4992477 T16111 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[13 (1461055602956435456)]} 0 0
   [junit4]   2> 4992481 T16048 C2696 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&version=2} {add=[13 (1461055602960629760)]} 0 0
   [junit4]   2> 4992485 T16114 C2695 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4992486 T16114 C2695 oasup.LogUpdateProcessor.finish [collection1] webapp=/v_fzy/e path=/update params={wt=javabin&waitSearcher=true&softCommit=false&commit=true&version=2} {} 0 1
   [junit4]   2> 4992487 T16114 C2695 oasc.SolrException.log ERROR java.lang.UnsupportedOperationException: this codec cannot write docvalues
   [junit4]   2> 		at org.apache.lucene.codecs.lucene3x.Lucene3xCodec$1.fieldsConsumer(Lucene3xCodec.java:73)
   [junit4]   2> 		at org.apache.lucene.index.DocValuesProcessor.flush(DocValuesProcessor.java:77)
   [junit4]   2> 		at org.apache.lucene.index.TwoStoredFieldsConsumers.flush(TwoStoredFieldsConsumers.java:42)
   [junit4]   2> 		at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:80)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:506)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:616)
   [junit4]   2> 		at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2864)
   [junit4]   2> 		at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
   [junit4]   2> 		at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
   [junit4]   2> 		at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:578)
   [junit4]   2> 		at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
   [junit4]   2> 		at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
   [junit4]   2> 		at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1457)
   [junit4]   2> 		at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1434)
   [junit4]   2> 		at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
   [junit4]   2> 		at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
   [junit4]   2> 		at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
   [junit4]   2> 		at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   [junit4]   2> 		at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	
   [junit4]   2> 4992489 T16114 C2695 oasc.SolrException.log ERROR null:java.lang.UnsupportedOperationException: this codec cannot write docvalues
   [junit4]   2> 		at org.apache.lucene.codecs.lucene3x.Lucene3xCodec$1.fieldsConsumer(Lucene3xCodec.java:73)
   [junit4]   2> 		at org.apache.lucene.index.DocValuesProcessor.flush(DocValuesProcessor.java:77)
   [junit4]   2> 		at org.apache.lucene.index.TwoStoredFieldsConsumers.flush(TwoStoredFieldsConsumers.java:42)
   [junit4]   2> 		at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:80)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:506)
   [junit4]   2> 		at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:616)
   [junit4]   2> 		at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2864)
   [junit4]   2> 		at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
   [junit4]   2> 		at org.apache.lucene.ind

[...truncated too long message...]

lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  60) Thread[id=743, name=HashSessionScavenger-17, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  61) Thread[id=744, name=qtp781588329-744 Acceptor0 SocketConnector@127.0.0.1:33558, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  62) Thread[id=627, name=qtp1527114287-627 Acceptor0 SocketConnector@127.0.0.1:13711, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  63) Thread[id=626, name=HashSessionScavenger-11, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 	  64) Thread[id=718, name=searcherExecutor-396-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  65) Thread[id=956, name=searcherExecutor-479-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  66) Thread[id=637, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[4D75887B35EB759B]-SendThread(localhost:11714), state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:164)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:245)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
   [junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:102)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
   [junit4]   2> 	  67) Thread[id=986, name=qtp1698469511-986 Acceptor0 SocketConnector@127.0.0.1:39533, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4]   2> 	        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4]   2> 	        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4]   2> 	        at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector.accept(SocketConnector.java:117)
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  68) Thread[id=719, name=Thread-268, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1018)
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=fi_FI, timezone=America/Caracas
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=70,free=145633128,total=321912832
   [junit4]   2> NOTE: All tests run in this JVM: [TermsComponentTest, JSONWriterTest, DeleteReplicaTest, SOLR749Test, SolrXmlInZkTest, TestReloadAndDeleteDocs, TestFieldTypeResource, RAMDirectoryFactoryTest, ParsingFieldUpdateProcessorsTest, SimpleFacetsTest, TestComponentsName, TestOmitPositions, TestIndexSearcher, TimeZoneUtilsTest, SolrCoreTest, ShowFileRequestHandlerTest, LeaderElectionIntegrationTest, TestCopyFieldCollectionResource, CursorPagingTest, TestNonDefinedSimilarityFactory, HdfsBasicDistributedZk2Test, WordBreakSolrSpellCheckerTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=4D75887B35EB759B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=America/Caracas -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4D75887B35EB759B]:0)
   [junit4] Completed on J1 in 7224.58s, 1 test, 2 errors <<< FAILURES!

[...truncated 213 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:451: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:490: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1278: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:911: There were test failures: 363 suites, 1565 tests, 1 suite-level error, 2 errors, 228 ignored (7 assumptions)

Total time: 151 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1916 - Still Failing

Posted by Robert Muir <rc...@gmail.com>.
Thanks Mark!

On Sun, Mar 2, 2014 at 11:45 AM, Mark Miller <ma...@gmail.com> wrote:
> I just tried zapping the workspace for that job.
>
> - Mark
>
> http://about.me/markrmiller
>
> On Mar 2, 2014, at 11:44 AM, Mark Miller <ma...@gmail.com> wrote:
>
>> Yeah, I was just trying to figure this out too - best I can tell, that check out is in a bad state.
>>
>> - Mark
>>
>> http://about.me/markrmiller
>>
>> On Mar 2, 2014, at 11:14 AM, Robert Muir <rc...@gmail.com> wrote:
>>
>>> I think this one needs a clean-jars or similar on the workspace?
>>>
>>>
>>> On Sun, Mar 2, 2014 at 11:11 AM, Apache Jenkins Server
>>> <je...@builds.apache.org> wrote:
>>>> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1916/
>>>>
>>>> All tests passed
>>>>
>>>> Build Log:
>>>> [...truncated 28344 lines...]
>>>> check-licenses:
>>>>    [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
>>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
>>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1
>>>>
>>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
>>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
>>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
>>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1
>>>>
>>>> [...truncated 3 lines...]
>>>> BUILD FAILED
>>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
>>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
>>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
>>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
>>>>
>>>> Total time: 111 minutes 2 seconds
>>>> Build step 'Invoke Ant' marked build as failure
>>>> Archiving artifacts
>>>> Recording test results
>>>> Email was triggered for: Failure
>>>> Sending email for trigger: Failure
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1916 - Still Failing

Posted by Mark Miller <ma...@gmail.com>.
I just tried zapping the workspace for that job.

- Mark

http://about.me/markrmiller

On Mar 2, 2014, at 11:44 AM, Mark Miller <ma...@gmail.com> wrote:

> Yeah, I was just trying to figure this out too - best I can tell, that check out is in a bad state.
> 
> - Mark
> 
> http://about.me/markrmiller
> 
> On Mar 2, 2014, at 11:14 AM, Robert Muir <rc...@gmail.com> wrote:
> 
>> I think this one needs a clean-jars or similar on the workspace?
>> 
>> 
>> On Sun, Mar 2, 2014 at 11:11 AM, Apache Jenkins Server
>> <je...@builds.apache.org> wrote:
>>> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1916/
>>> 
>>> All tests passed
>>> 
>>> Build Log:
>>> [...truncated 28344 lines...]
>>> check-licenses:
>>>    [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1
>>> 
>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
>>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
>>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1
>>> 
>>> [...truncated 3 lines...]
>>> BUILD FAILED
>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
>>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
>>> 
>>> Total time: 111 minutes 2 seconds
>>> Build step 'Invoke Ant' marked build as failure
>>> Archiving artifacts
>>> Recording test results
>>> Email was triggered for: Failure
>>> Sending email for trigger: Failure
>>> 
>>> 
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1916 - Still Failing

Posted by Mark Miller <ma...@gmail.com>.
Yeah, I was just trying to figure this out too - best I can tell, that check out is in a bad state.

- Mark

http://about.me/markrmiller

On Mar 2, 2014, at 11:14 AM, Robert Muir <rc...@gmail.com> wrote:

> I think this one needs a clean-jars or similar on the workspace?
> 
> 
> On Sun, Mar 2, 2014 at 11:11 AM, Apache Jenkins Server
> <je...@builds.apache.org> wrote:
>> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1916/
>> 
>> All tests passed
>> 
>> Build Log:
>> [...truncated 28344 lines...]
>> check-licenses:
>>     [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1
>> 
>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
>> [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
>> [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1
>> 
>> [...truncated 3 lines...]
>> BUILD FAILED
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
>> 
>> Total time: 111 minutes 2 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>> 
>> 
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1916 - Still Failing

Posted by Robert Muir <rc...@gmail.com>.
I think this one needs a clean-jars or similar on the workspace?


On Sun, Mar 2, 2014 at 11:11 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1916/
>
> All tests passed
>
> Build Log:
> [...truncated 28344 lines...]
> check-licenses:
>      [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1
>
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1
>
> [...truncated 3 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
>
> Total time: 111 minutes 2 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1916 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1916/

All tests passed

Build Log:
[...truncated 28344 lines...]
check-licenses:
     [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1

 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1

[...truncated 3 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.

Total time: 111 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1915 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1915/

All tests passed

Build Log:
[...truncated 28367 lines...]
check-licenses:
     [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1

 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1

[...truncated 3 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.

Total time: 103 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1914 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1914/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([9F02B25F17FA0C85:9B0A3DAC055FE3A4]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 10218 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 127058 T368 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 127059 T368 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1393611306913
   [junit4]   2> 127060 T368 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 127066 T368 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 127067 T368 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 127067 T369 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 127168 T368 oasc.ZkTestServer.run start zk server on port:14056
   [junit4]   2> 127169 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 127201 T375 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@129ace22 name:ZooKeeperConnection Watcher:127.0.0.1:14056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 127201 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 127202 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 127205 T377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@365d1fe2 name:ZooKeeperConnection Watcher:127.0.0.1:14056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 127206 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 127215 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 127218 T379 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@267e5d35 name:ZooKeeperConnection Watcher:127.0.0.1:14056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 127219 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 127219 T368 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 127223 T368 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 127233 T368 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 127236 T368 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 127238 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 127242 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 127244 T381 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21065fec name:ZooKeeperConnection Watcher:127.0.0.1:14056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 127245 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 127246 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 127249 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 127252 T381 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 127252 T375 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 127253 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 127255 T383 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a0a925d name:ZooKeeperConnection Watcher:127.0.0.1:14056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 127255 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 127257 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 127259 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 127271 T368 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:14056_solr
   [junit4]   2> 127272 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 127275 T368 oasc.Overseer.start Overseer (id=91331710610964484-127.0.0.1:14056_solr-n_0000000000) starting
   [junit4]   2> 127284 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 127288 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 127292 T368 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 127296 T385 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 127297 T385 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 127297 T384 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 128299 T383 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 128301 T384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 128302 T384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 128303 T384 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 128303 T384 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 128312 T383 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 128314 T381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 128314 T375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 128800 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 128828 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 128835 T383 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 128836 T368 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 128837 T384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 128840 T383 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 128840 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 128842 T384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 128843 T384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 128846 T383 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 128949 T375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 128951 T381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 130008 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 130036 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130039 T387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4befe1a1 name:ZooKeeperConnection Watcher:127.0.0.1:14056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130039 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130051 T368 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:14056_solr
   [junit4]   2> 130051 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 130054 T368 oasc.Overseer.start Overseer (id=91331710610964485-127.0.0.1:14056_solr-n_0000000001) starting
   [junit4]   2> 130071 T389 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 130072 T389 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 130072 T388 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 130075 T388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 130076 T388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 130076 T388 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 130080 T387 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 130081 T381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 130081 T375 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 130137 T387 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 130137 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 130139 T388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 130152 T368 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91331710610964483-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 
   [junit4]   2> 130161 T381 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 130163 T388 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:633)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:628)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:949)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:235)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:192)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	
   [junit4]   2> 130163 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 130174 T387 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 130177 T387 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 130177 T375 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 130181 T368 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:14056 14056
   [junit4]   2> 130255 T368 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=9F02B25F17FA0C85 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=da -Dtests.timezone=EAT -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   3.26s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9F02B25F17FA0C85:9B0A3DAC055FE3A4]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:426)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:156)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:154)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:590)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 130330 T368 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 130330 T368 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 130331 T390 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 130432 T368 oasc.ZkTestServer.run start zk server on port:41651
   [junit4]   2> 130433 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130439 T396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57fe2b6 name:ZooKeeperConnection Watcher:127.0.0.1:41651 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130440 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130493 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130501 T398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4985d2 name:ZooKeeperConnection Watcher:127.0.0.1:41651 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130501 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130502 T368 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 130506 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130515 T400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c325a3b name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130515 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130516 T368 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 130519 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130521 T402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45d7d580 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130521 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130523 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 130526 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 130531 T368 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41651_solr
   [junit4]   2> 130531 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 130534 T368 oasc.Overseer.start Overseer (id=91331710825070595-127.0.0.1:41651_solr-n_0000000000) starting
   [junit4]   2> 130537 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 130541 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 130545 T368 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 130548 T404 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 130549 T368 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 130549 T404 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 130549 T403 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 130553 T368 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 130563 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130566 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130569 T406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b1451ed name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130569 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130571 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130574 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 130577 T400 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 130577 T406 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 130578 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130580 T408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a71217a name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130580 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130582 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130585 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 130588 T408 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 130588 T406 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 130588 T400 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 130589 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130591 T410 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1209739c name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130591 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130593 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130596 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 130599 T406 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 130600 T408 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 130599 T410 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 130599 T400 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 130600 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130603 T412 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@183b5819 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130603 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130605 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130607 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 130611 T406 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 130611 T410 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 130611 T400 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 130611 T412 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 130611 T408 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 130612 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130614 T414 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bd33b15 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130614 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130616 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130619 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 130622 T410 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130622 T406 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130622 T408 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130622 T400 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130622 T412 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130622 T414 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 130623 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130625 T416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c40a6f8 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130625 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130627 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130630 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 130633 T400 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130633 T412 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130633 T408 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130633 T410 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130633 T406 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130634 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130634 T414 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130634 T416 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 130637 T418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1db9b30c name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130637 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130639 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130641 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 130645 T414 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T418 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T406 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130646 T400 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T416 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T408 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T412 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130645 T410 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 130646 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130648 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61a05acb name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130649 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130650 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130653 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 130657 T400 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T410 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T416 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130658 T408 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T414 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T418 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T406 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T412 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130657 T420 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 130658 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130660 T422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bf7a927 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130661 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130662 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130665 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 130669 T408 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T412 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T400 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130670 T422 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T410 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T414 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T406 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T418 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130669 T416 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130670 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130670 T420 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 130673 T424 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ccefbbd name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130673 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130675 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130678 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 130681 T416 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T412 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T424 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T422 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T406 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T414 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130681 T408 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130681 T418 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130683 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130682 T420 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T400 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130682 T410 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 130685 T426 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6466fd34 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130686 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130688 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130690 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 130694 T418 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T416 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T424 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130695 T410 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130695 T412 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T408 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130695 T400 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130695 T420 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130695 T422 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T414 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T426 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130694 T406 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 130696 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130698 T428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3182afc9 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130699 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130700 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130703 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 130707 T422 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130707 T424 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T410 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130707 T420 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T412 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130707 T428 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130707 T414 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130707 T400 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130709 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130708 T418 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T416 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T406 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T408 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130708 T426 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 130711 T430 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c2fe4e7 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130712 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130714 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130716 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 130720 T424 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130720 T430 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T412 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T400 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T422 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130720 T414 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T418 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130720 T408 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130720 T406 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130720 T416 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130722 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130721 T410 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T426 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T428 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130721 T420 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 130724 T432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fecb886 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130725 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130727 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130729 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 130733 T426 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130733 T412 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T424 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T418 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T428 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T410 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T432 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130733 T420 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130735 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130735 T430 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130735 T406 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T416 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T414 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T422 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T408 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130734 T400 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 130737 T434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1672f0a6 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130739 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130740 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130743 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 130747 T406 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130747 T426 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T420 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T410 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T432 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T414 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T408 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130747 T424 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130747 T434 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130749 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130749 T428 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130749 T400 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130749 T422 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T412 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T430 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T418 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130748 T416 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 130751 T436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@765746ca name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130753 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130754 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130757 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 130761 T414 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T422 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T430 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T406 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T420 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T434 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T416 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T428 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T400 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T424 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T408 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130761 T436 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130763 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130763 T432 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T410 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T412 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T418 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130762 T426 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 130765 T438 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64862576 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130767 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130768 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130771 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 130775 T432 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T434 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T422 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T430 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T424 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130777 T438 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T418 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T420 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T410 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T412 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T406 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130776 T426 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130778 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130777 T408 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130780 T440 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12892419 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130777 T414 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130777 T428 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130777 T436 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130777 T416 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130777 T400 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 130781 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130784 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130787 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 130791 T406 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T434 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T418 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T436 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T420 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T400 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T424 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130793 T408 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T428 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T412 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T410 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T426 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T432 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T416 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130791 T440 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130795 T442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e6e00 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130793 T422 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130793 T430 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T414 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130792 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130792 T438 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 130797 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130799 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130802 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 130806 T422 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T436 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T414 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T420 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T426 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T438 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T428 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T430 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T424 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T400 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130809 T432 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130809 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130809 T440 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T418 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T406 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T434 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T416 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T412 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T410 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130808 T442 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130807 T408 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 130811 T444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97d9475 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130813 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130815 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130818 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 130822 T420 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T426 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T440 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T434 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T418 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T438 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T430 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T406 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T412 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T442 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T410 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T432 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130823 T416 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130825 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130825 T444 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130825 T436 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T408 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T414 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T424 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T400 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T422 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130824 T428 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 130828 T446 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3076f54c name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130831 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130834 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130837 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 130842 T428 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T444 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T432 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T412 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T430 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T438 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T406 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130842 T424 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130842 T400 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130845 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130844 T446 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T434 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T410 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T416 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T420 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T414 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T418 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130844 T442 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T440 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T436 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T408 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T422 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130843 T426 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 130848 T448 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a40ce0a name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130850 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130852 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130855 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 130859 T446 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T410 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T420 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T416 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T440 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T418 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130859 T406 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130859 T448 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130859 T434 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130862 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130862 T428 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T430 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T400 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T424 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T422 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T436 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T408 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T438 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T444 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T414 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130861 T432 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T412 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T442 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130860 T426 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 130865 T450 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c4b88a5 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130868 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130869 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130872 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 130877 T406 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T416 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T408 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T422 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T432 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130879 T444 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130879 T420 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T446 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130879 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130877 T426 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T448 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T450 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T434 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130877 T418 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130879 T410 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130882 T452 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16d11a0e name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130879 T412 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130879 T430 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T440 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T436 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T414 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T442 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T400 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T428 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T424 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130878 T438 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 130882 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130887 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130890 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 130909 T408 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T424 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T452 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T428 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T400 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T412 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130910 T430 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T450 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T436 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T414 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T442 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T406 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130914 T454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e709107 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130910 T434 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T418 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130910 T422 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T416 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T426 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T444 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T432 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130909 T438 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130914 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130911 T446 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T448 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T420 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T410 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130911 T440 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 130918 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130921 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 130926 T408 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T426 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T406 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T434 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T420 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T440 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T452 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T400 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T430 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T428 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T422 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T418 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T416 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T432 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T438 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T444 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130926 T424 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130929 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130928 T410 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T414 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T454 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T442 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T446 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T448 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130928 T412 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T436 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130927 T450 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 130931 T456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72e0e90a name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130935 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130936 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130939 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 130943 T410 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T412 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130943 T420 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T422 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T424 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T434 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T414 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130946 T452 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T432 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T440 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T444 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T408 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T406 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T430 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T428 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T400 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T456 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T442 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T446 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T448 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130944 T450 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130949 T458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cd2aec1 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130947 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130946 T418 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130946 T426 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130946 T436 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130946 T416 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T454 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130945 T438 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 130951 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130954 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130957 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 130961 T408 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T424 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T458 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T420 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130961 T426 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T428 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130961 T444 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130964 T452 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130964 T430 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130964 T422 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T450 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130963 T446 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T416 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T432 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T438 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T448 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130963 T410 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T440 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T456 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T412 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T442 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T414 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T406 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T400 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T418 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130962 T434 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130966 T460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e09645c name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130964 T454 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130964 T436 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 130970 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130972 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130975 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 130980 T416 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T438 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T424 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T442 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T444 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T418 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T440 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T436 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T448 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130983 T400 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T406 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130983 T368 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 130980 T432 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T414 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T422 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T454 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T452 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T446 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130986 T462 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42ff95b0 name:ZooKeeperConnection Watcher:127.0.0.1:41651/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 130982 T458 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130982 T430 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T420 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T412 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T428 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T410 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T450 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T456 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T460 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130981 T408 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T434 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130980 T426 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 130986 T368 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 130991 T368 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130994 T368 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 130998 T460 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 130998 T440 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 130998 T412 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 130998 T448 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 131000 T430 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 131000 T422 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 131000 T434 oascc.ZkState

[...truncated too long message...]

nit4]   2> 152204 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 152208 T368 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:26420_solr
   [junit4]   2> 152208 T368 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 152210 T368 oasc.Overseer.start Overseer (id=91331712246415364-127.0.0.1:26420_solr-n_0000000000) starting
   [junit4]   2> 152212 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 152215 T368 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 152218 T368 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 152220 T898 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 152220 T898 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 152220 T897 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 152222 T897 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 152223 T897 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 152223 T897 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 152224 T897 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard10
   [junit4]   2> 152226 T896 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152227 T894 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 152227 T888 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 152723 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard10/election
   [junit4]   2> 152753 T368 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard10
   [junit4]   2> 152766 T896 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152766 T368 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 152767 T897 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 152775 T888 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 152775 T894 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 152778 T368 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:26420 26420
   [junit4]   2> 152843 T368 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 152846 T368 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 152846 T368 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 153322 T881 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 153322 T881 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 153323 T881 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 155847 T368 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=da, timezone=EAT
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=75672752,total=145752064
   [junit4]   2> NOTE: All tests run in this JVM: [AlternateDirectoryTest, TestRemoteStreaming, TestSystemIdResolver, TestCloudManagedSchemaAddField, TestJmxMonitoredMap, TestValueSourceCache, CircularListTest, ClusterStateUpdateTest, TestCollationField, QueryElevationComponentTest, SuggestComponentTest, RequestHandlersTest, TestSchemaResource, OverseerTest]
   [junit4] Completed on J0 in 28.84s, 8 tests, 1 error <<< FAILURES!

[...truncated 987 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:451: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:490: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1278: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:911: There were test failures: 363 suites, 1570 tests, 1 error, 36 ignored (8 assumptions)

Total time: 63 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1913 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1913/

All tests passed

Build Log:
[...truncated 28417 lines...]
check-licenses:
     [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1

 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1

[...truncated 3 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.

Total time: 108 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1912 - Still Failing

Posted by Robert Muir <rc...@gmail.com>.
I committed a fix.

On Wed, Feb 26, 2014 at 6:01 PM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1912/
>
> All tests passed
>
> Build Log:
> [...truncated 28267 lines...]
> check-licenses:
>      [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1
>
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
>  [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
>  [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1
>
> [...truncated 3 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
>
> Total time: 107 minutes 7 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1912 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1912/

All tests passed

Build Log:
[...truncated 28267 lines...]
check-licenses:
     [echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jcl-over-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jcl-over-slf4j-1.6.6.jar.sha1

 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/jul-to-slf4j-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/jul-to-slf4j-1.6.6.jar.sha1
 [licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/example/lib/ext/slf4j-api-1.6.6.jar
 [licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/licenses/slf4j-api-1.6.6.jar.sha1

[...truncated 3 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:471: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:64: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:254: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.

Total time: 107 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure